Introduction

Harness gave you durable state: an append-only event log, compaction, resume, a sandbox. That is what keeps a session alive. It does not tell you which stage is burning money, which retrieval query returned garbage, or why one tenant's p95 latency doubled after Tuesday's deploy.

Observability for LLM apps is the instrumentation layer on top: traces that follow a request through classify → retrieve → tool → synthesize, cost fields tied to each provider call, and enough artifacts to replay a bad answer without guessing from chat text alone.

Orchestration already named the per-stage fields worth logging. This article is how you wire them into something you can search, graph, and alert on — Langfuse, OpenTelemetry exporters, or a Postgres table you own. The shape matters more than the vendor.

Why LLM observability is not regular APM

A REST handler that returns 200 usually did the same work every time. An LLM path can return 200 with a wrong refund explanation, a loop that ran twelve tool calls, or a $4 completion because someone pasted a 90k-token PDF into retrieval.

Three differences show up in every production incident:

1. The expensive unit is tokens, not CPU milliseconds. A "slow" request might be cheap; a fast one might be ruinous.
2. The failure is often semantic, not a stack trace. The HTTP status is fine; the retrieved chunks were wrong.
3. One user turn is a tree, not a span. Classifier call, embedding search, two tool rounds, synthesis retry — all one "request" in the product UI.

You still want latency histograms and error rates. You also need generation records with model id, token counts, finish reason, and the inputs that actually reached the model after projection and compaction.

Trace, span, generation

Tools like Langfuse popularized a hierarchy that maps cleanly onto orchestration and harness turns:

• Trace — one product-visible unit: a support reply, an agent turn, a pipeline run. Carries trace_id, session_id, user_id, environment, release version.
• Span — one step inside the trace: retrieval, a tool handler, a policy check. Has name, start/end, status, attributes.
• Generation (or observation type generation) — a provider LLM call. Holds model, parameters, input/output messages (or hashes), usage, latency, finish_reason.

A single trace often contains several generations. An agent loop with four model calls is one trace with four generation children, not four unrelated traces.

Keep IDs stable across the harness event log and the trace. When the harness writes seq 42 tool_result, the matching span should carry event_seq=42. Resume after a crash should not fork observability identity.

The minimum attribute set

If you only instrument six things, make them these:

Add retrieval-specific fields on the retrieve span: query text (or hash), index name, top-k chunk IDs, scores, reranker latency. Add tool spans: tool name, validated-args hash (not raw secrets), idempotency key, downstream HTTP status.

Orchestration's fallback reason belongs on the trace root when synthesis fails the quality gate — otherwise you only see "empty answer" in the UI.

Wiring a trace through a pipeline

Below is a skeleton that wraps the orchestration pipeline from part 7. No framework magic: start a trace, open spans, record generations where the HTTP client returns.

type Tracer struct {
  backend Backend // Langfuse, OTel, your DB
}

type Trace struct {
  ID        string
  SessionID string
  backend   Backend
}

type Span struct {
  TraceID string
  Name    string
  backend Backend
  start   time.Time
}

func (t *Tracer) StartTrace(sessionID string) *Trace {
  id := uuid.NewString()
  t.backend.CreateTrace(id, sessionID)
  return &Trace{ID: id, SessionID: sessionID, backend: t.backend}
}

func (tr *Trace) Span(name string) *Span {
  return &Span{TraceID: tr.ID, Name: name, backend: tr.backend, start: time.Now()}
}

func (s *Span) End(status string, attrs map[string]any) {
  s.backend.EndSpan(s.TraceID, s.Name, s.start, status, attrs)
}

func (s *Span) RecordGeneration(model string, usage Usage, finish string, ms int64) {
  s.backend.RecordGeneration(s.TraceID, s.Name, model, usage, finish, ms)
}

func RunPipeline(tr *Trace, state *State) error {
  s := tr.Span("classify")
  state.Intent = classifyIntent(state.UserInput)
  s.End("ok", map[string]any{"intent": state.Intent})

  if needsRetrieval(state.Intent) {
    r := tr.Span("retrieve")
    state.Evidence, state.ChunkIDs = retrieveEvidence(state.UserInput)
    r.End("ok", map[string]any{
      "chunk_ids": state.ChunkIDs,
      "k":         len(state.ChunkIDs),
    })
  }

  g := tr.Span("synthesize")
  resp, err := callChatCompletion(buildMessages(state))
  if err != nil {
    g.End("error", map[string]any{"err": err.Error()})
    return err
  }
  g.RecordGeneration(resp.Model, resp.Usage, resp.FinishReason, resp.LatencyMs)
  state.FinalAnswer = resp.Text
  g.End("ok", nil)
  return nil
}

The retrieve span ends before synthesis on purpose. When the answer hallucinates a policy clause, you want chunk IDs on a sibling span — not buried inside a generation blob.

Agent loops and harness turns

An agent turn from part 6 is a trace (or a child trace under a session root) with a repeating pattern:

trace: agent_turn_17
  span: project_messages
  generation: model (tool_calls=[grep, read_file])
  span: tool grep
  span: tool read_file
  generation: model (final text)

Each generation records usage separately so you can sum cost per turn and spot turns where the model called tools unnecessarily. The harness interrupted event should close the trace span with status=cancelled and still persist partial usage if the provider billed for tokens already streamed.

Compaction events belong in metadata, not as a replacement for history in traces. Store compaction_through_seq on the trace so you know the model saw a summary instead of turns 1–40 when debugging a regression.

Subagents from part 8 should be nested traces linked by parent_trace_id. The parent only needs the subagent's result span plus token totals — not thirty thousand tokens of grep output duplicated in two places.

Cost: where the bill actually comes from

Part 2 introduced usage.prompt_tokens and usage.completion_tokens. Observability is where those fields become accounting.

On every generation record:

• Model id (including pinned session model from the harness)
• Prompt, completion, and total tokens
• Cached prompt tokens when the provider exposes them (prefix cache hits from part 8)
• Estimated USD from a price table keyed by model — compute in your exporter, do not hand-wave in dashboards

Roll up at three levels:

1. Per trace — what this reply cost the business
2. Per session — what this coding agent hour cost
3. Per tenant / feature flag — who to throttle or upsell

Alert on rate of change, not just absolutes. A 40% jump in mean prompt tokens per trace after a prompt template change is a retrieval or history bug long before finance notices.

Watch completion tokens for runaway decode. finish_reason: "length" with max_tokens pinned high is an agent that never learned to stop — cheap to spot in aggregates, expensive to miss.

Latency: prefill, decode, and everything else

Provider latency on a generation span should split when you can:

• Time to first token (TTFT) — mostly prefill on long prompts; spikes when RAG dumps grow
• Total generation time — prefill plus decode; grows with output length
• Non-model time — retrieval, embedding, rerank, tool HTTP, sandbox execution

Put retrieval and tool latency on their own spans. Otherwise p95 "LLM latency" hides a 4s vector search.

For streaming UIs, record when the first token arrived on the generation span. Users complain about "slow" when TTFT is bad even if total time is fine.

Debugging workflows that actually work

Three questions cover most on-call pages:

Why was this answer wrong? Open the trace. Check retrieve span chunk IDs and scores — empty or low-score retrieval explains most groundedness failures. Compare the projected messages hash on the generation span to what you thought you sent. If they differ, the bug is assembly or compaction, not the model.

Why was this slow? Sort spans by duration inside the trace. If retrieve dominates, fix the index or shrink k. If TTFT dominates, shrink the prompt or fix prefix cache poisoning. If tool spans dominate, the model is doing too much work per turn.

Why was this expensive? Sum generation usage across the trace. Multi-call agent loops show up immediately as four stacked generations. Compare prompt tokens trace-over-trace for the same intent — a jump means history or RAG grew.

Keep a "golden bad trace" link in runbooks: one known failure with annotations. New engineers learn the UI faster from a concrete example than from a dashboard tour.

Redaction, sampling, and retention

Traces tempt you to store full prompts. That is also where PII, secrets, and scraped page content live.

Redact at capture time on tool outputs and environment reads — the same boundary as the harness sandbox. Store SHA-256 hashes of message payloads for dedup and "did we send the same thing twice" checks; store full text only in restricted buckets with shorter TTL.

Sampling strategies:

Always record 100% of traces that hit a fallback, quality-gate failure, or approval denial. Those are the ones you need for fixes and for eval datasets in the next article.

Logs, traces, and evals

These layers stack; they do not replace each other.

Logs alone rarely answer "which chunks were retrieved for ticket 8842." Traces alone rarely prove regression across a thousand labeled questions — that is Evals.

Export trace IDs into eval failure rows when a human marks an answer bad. You get a direct link from a failing test case to the exact retrieval and generation records.

Failure modes

• One span for the whole pipeline. You see total latency, not which stage broke.
• Logging final text only. Wrong answers with no chunk IDs are not debuggable.
• Missing session_id on traces. Multi-turn cost and compaction bugs look like one-off noise.
• Storing raw tool args with secrets. Traces become a credential leak.
• Ignoring cached token fields. Prefix cache wins disappear from cost charts.
• New trace per model call in an agent loop. Loops become impossible to reason about in aggregate.
• 100% full-prompt retention forever. Compliance incident waiting to happen.
• Dashboards without alerts on token drift. Finance finds the problem before engineering does.

Conclusion

Observability turns orchestration stages and harness events into something you can measure: traces for the path, generations for token economics, sibling spans for retrieval and tools. Wire IDs once, redact at the boundary, and keep enough metadata to replay bad answers without production SSH.

Next up is Evals — labeled datasets, scoring, and the loop that proves a change actually helped before you ship it to everyone.