Observability

When one run out of two hundred goes wrong, reading logs is a poor way to find it. Leviath can export traces, metrics, and logs over OpenTelemetry, so your existing dashboards can answer "which run is stuck" and "is anything actually moving" without you going digging.

It is off by default. Turning it on is one config block, and nothing about how a run behaves changes either way.

Turning it on

toml

[observability]
enabled = true
exporter = "otlp"                     # "otlp" | "stdout" | "none"
endpoint = "http://localhost:4318"
service_name = "leviath"

The standard OpenTelemetry environment variables fill in anything the file leaves out. OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_SERVICE_NAME are both honoured, and an explicit config value beats the environment.

To see what would be exported without sending it anywhere, set exporter = "stdout". It narrates the same events as readable lines on stderr.

Warning

Export is OTLP over HTTP, on port 4318. Many collector examples use 4317, which is the gRPC port, and Leviath does not speak it. Pointing at 4317 fails silently from the outside.

Traces

Every run becomes one trace, shaped like the run itself:

flowchart TD
  RUN["agent.run<br/>(one per run)"]
  RUN --> S1["agent.stage: plan"]
  RUN --> S2["agent.stage: implement"]
  S1 --> I1["agent.inference"]
  S2 --> I2["agent.inference"]
  S2 --> T1["agent.tool_call"]
  S2 --> T2["agent.tool_call"]

Stage transitions, retries, and the final status all land as span attributes. That means a stuck or looping run is visible as a shape on a trace view, rather than something you have to infer from log lines.

Metrics

Per run, labelled by agent, stage, provider, and model:

Metric	Type	What it tells you
`leviath.agents.active`	gauge	How many runs are going right now
`leviath.tokens.total`	counter	Tokens consumed
`leviath.tool_calls.total`	counter	Tool calls made
`leviath.stage_duration`	histogram	How long stages take
`leviath.inference_latency`	histogram	How long model calls take
`leviath.runs.total`	counter	One per finished run, attributed by `leviath.status` and `leviath.empty_output`

Per daemon, sampled every 30 seconds:

Metric	What it tells you
`leviath.tool_lane.busy`	Tool batches running
`leviath.tool_lane.queued`	Tool batches waiting for a slot
`leviath.tool_lane.parked`	Batches waiting on something with no time limit, holding no slot
`leviath.scheduler.dead_cycles.total`	30-second intervals where the lane was full, work was queued, and nothing moved
`leviath.tool_lane.relief.total`	Times the daemon widened the lane to break a jam
`leviath.provider.circuit.open`	Providers Leviath has currently stopped sending work to
`leviath.provider.circuit.opened.total`	Times a provider was pulled, attributed by `leviath.provider` and `leviath.reason`

leviath.reason is one of credits-exhausted, auth-failed, forbidden, or unreachable.

The three worth alerting on

Most of the above is for dashboards. These three are for pages.

leviath.scheduler.dead_cycles.total separates a busy daemon from a stuck one. A dead cycle is a full 30-second interval where a lane was at capacity, work was queued behind it, and no run moved anywhere. A healthy daemon sits at zero. Anything sustained above zero means work is arriving that nothing is getting to.

leviath.provider.circuit.open goes back to zero on its own once a provider recovers, so a reading that stays non-zero means a person has to top up an account or fix a key. This is the one that catches a drained account, which otherwise hides: the runs it kills die before producing any per-run telemetry at all.

The leviath.empty_output attribute on leviath.runs.total is worth charting as a rate. A run reaching complete only means it got to the end of its pipeline. It does not mean anything came of it. Divide empty runs by total runs, and a jump usually means an agent started editing through the shell, where Leviath cannot see the writes. Runs by agents that never had a file-writing tool are excluded, so a fleet of routers and researchers will not drown the signal. See what counts as output.

Logs

Log records carry the run's trace ID. A collector that joins all three signals can therefore jump from a log line straight to the span that produced it.

Trying it locally

Any OTLP-over-HTTP collector works. With a local Jaeger all-in-one listening on 4318, enable the block above and run an agent. The run shows up as a trace named agent.run under whatever service_name you set.

The daemon owns the exporter. It starts with the daemon and flushes on shutdown, so short-lived CLI commands do not each pay the setup cost. See the daemon for where this sits in its lifecycle.

Where one run's cost went

The metrics above answer "is the fleet healthy". For "what did this one run spend, and on which stage", read its stage ledger instead: lev stages <run-id> at a terminal, or GET /api/agents/{id}/stages over HTTP. Both carry the per-stage token split, the cache read and write halves, and the largest each context region reached while that stage was active, which is the number to look at before trimming a layout.