Introduction

Observability tells you why one response happened: which chunks landed, which tool fired, how many tokens the generation burned. That is debugging. It is not proof.

A prompt tweak that fixes the ticket you had open can quietly tank recall on fifty other questions. A cheaper model that "looked fine" in three traces can miss the refuse path RAG depends on. Evals are how you catch that before everyone else does: a labeled set, a runner that walks the same path the product uses, scorers that turn outputs into numbers, and a baseline you compare against.

Traces still matter. Failed eval rows should carry trace_id so you can open the retrieve span instead of re-guessing from the final sentence.

An eval is a comparison

Four pieces, none optional for long:

1. Dataset — cases with an input and a label of what "good" means (relevant chunk IDs, a must-cite set, a tool sequence, a refuse flag).
2. Runner — executes the product path, or a frozen slice of it, and records the artifacts observability already named: chunk IDs, citations, tool names, final text.
3. Scorer — maps those artifacts to numbers or booleans. Code first; a judge model only where code is blind.
4. Baseline — the last version you would still ship. A score without a comparison is a vibe with extra steps.

Orchestration's quality gate is a cousin, not a substitute. A gate is one request, live: citations missing → fallback template. An eval is a set, offline or sampled: did this change help across the cases you care about? You want both. The gate stops a bad answer; the eval stops a bad deploy.

Score the failure you actually have

"Helpfulness, 1–5" is a weak default. It averages unlike things and hides the bug you shipped. Pick metrics from the layer that breaks.

Retrieval (from Embeddings & Retrieval): recall@k and whether the gold passage is in the packed set, not whether cosine "looked high." A fluent answer over the wrong neighbors is a retrieval miss, even if the judge loves the prose.

RAG: faithfulness to packed evidence, citation IDs that exist in what you sent, and the refuse path when retrieval is empty or irrelevant. Unverified [chunk_id] strings are cheap to fake — check them in code against the IDs on the retrieve span.

Tools and agents: did the handler that should have run actually run, with valid args, under the step budget? Task success is a fixture ("order 1842 is refundable, expect create_credit once"), not "the loop felt done." Illegal or extra tool calls are failures even when the user-visible sentence is polite.

Classifiers in the orchestration pipeline: accuracy against labeled intents. If classify is wrong, retrieve and tools never get a fair chance — score that stage alone so you do not blame synthesis.

Start with whatever a unit test could assert. Add a rubric later for the residue code cannot see (tone, paraphrase that is still faithful).

Gold from production traces

You do not invent a thousand questions on day one. You steal them from traffic you already store.

Always-keep traces from the observability article — fallbacks, quality-gate failures, approval denials — are the first cases. A human marks: relevant chunk IDs, whether the answer should have refused, which tool should have fired. Export trace_id onto the eval row so a regression opens the same retrieve span six weeks later.

Cover the failure modes you already wrote down, not a balanced academic set:

• Empty retrieval and "I don't know"
• Right neighbor, amputated chunk (the 14-day rule lived in the parent)
• Citation invented against packed IDs
• Agent that calls the write tool twice
• Compaction that dropped the constraint from turn one — session evals need a transcript, not a single query

Two runner modes, and they answer different questions:

If you only run end-to-end, a retrieval regression and a prompt regression look identical. If you only freeze chunks, you will ship an index that no longer finds the gold passage. Keep both; tag each case with which mode it belongs to.

Version the dataset like code. When the policy doc changes, the gold labels change — a case that expected "14 days" is wrong after legal ships "30 days." Pin corpus version next to the case, the same way you pin model id on a harness session.

Fifty cases that hit real failures beat two thousand paraphrases of the happy path. Grow the set from new production misses, not from synthetic variety for its own sake.

Code scorers before judges

A scorer that parses citations and computes recall does not need another model, does not drift when the judge vendor ships a new snapshot, and is cheap enough to run on every PR.

type Case struct {
  ID               string
  Query            string
  RelevantChunkIDs []string
  ExpectRefuse     bool
}

type Output struct {
  ChunkIDs  []string
  Citations []string // parsed [id] markers, already extracted
  Refused   bool
}

type Score struct {
  CaseID     string
  RecallAtK  float64
  CiteValid  float64
  RefuseOK   bool
}

func recallAtK(got, relevant []string, k int) float64 {
  if len(relevant) == 0 {
    return 1
  }
  seen := map[string]bool{}
  for i, id := range got {
    if i >= k {
      break
    }
    seen[id] = true
  }
  hits := 0
  for _, id := range relevant {
    if seen[id] {
      hits++
    }
  }
  return float64(hits) / float64(len(relevant))
}

func citeValid(cites, packed []string) float64 {
  if len(cites) == 0 {
    return 0
  }
  allowed := map[string]bool{}
  for _, id := range packed {
    allowed[id] = true
  }
  ok := 0
  for _, id := range cites {
    if allowed[id] {
      ok++
    }
  }
  return float64(ok) / float64(len(cites))
}

type Report struct {
  MeanRecall float64
  MeanCite   float64
  N          int
  Failures   []string
}

func RunEval(cases []Case, k int, run func(Case) Output) Report {
  var rec, cite float64
  var fails []string
  for _, c := range cases {
    out := run(c)
    r := recallAtK(out.ChunkIDs, c.RelevantChunkIDs, k)
    v := citeValid(out.Citations, out.ChunkIDs)
    refuseOK := c.ExpectRefuse == out.Refused
    rec += r
    cite += v
    if r < 1 || !refuseOK || (!c.ExpectRefuse && v < 1) {
      fails = append(fails, c.ID)
    }
  }
  n := float64(len(cases))
  return Report{
    MeanRecall: rec / n,
    MeanCite:   cite / n,
    N:          len(cases),
    Failures:   fails,
  }
}

run is your pipeline, not a mocked paragraph. For generation-only cases it replays packed chunks from the fixture; for end-to-end it calls retrieve for real. The report's Failures list is the part you read. Means without IDs are how a 2% dip on the one case that handles refunds gets lost in a 0.01 mean bump.

Citation validity of 0 when the model refused is expected — the refuse check owns that case, not citeValid. Mixing those two into one "quality" number is how a model that never cites and never refuses looks "average."

Judges, when code cannot see it

Faithfulness — "does this sentence follow from the evidence block?" — is the usual gap. An LLM-as-judge is another completion with a rubric, the packed evidence, and the candidate answer. It is not a personality test.

A usable rubric is short and operational:

• Supported: every factual claim appears in Evidence (paraphrase allowed).
• Unsupported: a number, date, policy clause, or name that Evidence does not contain.
• Contradiction: Evidence says 14 days, the answer says 30.
• Refuse: Evidence is empty or irrelevant and the answer still invents.

Ask for a structured verdict (label, span of the bad claim, evidence_id) and parse it like any other schema. Free-form "7/10, pretty good" is how judges launder vibes back into your dashboard.

Pairwise ("is A better than B?") is often stabler than absolute scores for prompt bake-offs. Swap order and run twice; positional bias is real. Do not use the same model family as both candidate and judge when you can avoid it — it grades its own style.

Humans still label the gold set and audit a sample of judge disagreements. The judge is a multiplier on labels you already trust, not a replacement for them.

Noise, pinning, and the baseline

Part 2 already warned that temperature: 0 is not bitwise replay. Provider replicas, batching, and MoE routing move tokens. For evals that means:

• Pin model id (and judge id) on the run, the way the harness pins the session model.
• Record sampling params. A CI eval at temperature 0.7 is measuring noise.
• Treat exact string match as a bonus. Prefer structured fields and code scorers that do not care about comma placement.
• If you need a tighter interval, run the same case n times and score the rate — expensive, so reserve it for the flaky slice, not the whole set.

The number you ship against is the baseline report from the last known-good commit, same dataset version. "Faithfulness 0.81" means nothing without "was 0.84 on eval-set@v12 last Tuesday." Gate the deploy on deltas you picked in advance (recall@k must not drop, refuse cases must stay green). A mean that wiggles inside the noise band is not a win.

Run the suite in CI on prompt, chunking, index, and tool-handler changes. Offline evals are the unit tests of this stack. Online sampling — score a slice of production traces after the fact — is the canary. Neither replaces the other: CI never sees the weird tenant; production sampling never has gold chunk IDs unless you labeled them.

Offline, online, and the live gate

Three loops, three jobs:

Offline eval — frozen dataset, runner, report vs baseline. You decide to merge.

Online eval — sample production, attach scores later (code scorers on citations; periodic human or judge on a subsample). You decide to roll back. This is also how the dataset grows: disagreements become new cases.

Live quality gate — orchestration, one request, no gold label. Check parse, citation IDs ⊂ packed IDs, refuse regex, schema. Fail → fallback, and keep the trace at 100% sample rate so it can enter the gold set.

If the live gate and the offline scorer disagree on the same rule, the scorer is wrong or the gate never shipped. Keep them the same function.

Which scorer for which question

Vendors (Langfuse datasets, Promptfoo, whatever you already use for traces) are storage and a UI on this loop. They do not pick the metric. If the product is RAG, your first scorer is still recall and citation checks, whether the row lives in Postgres or someone else's eval tab.

Failure modes

• Demo path, not product path. Eval calls the model with hand-waved chunks; production runs hybrid retrieval. You measured a prompt that production never sends.
• Happy-path-only set. No empty retrieval, no write-tool case, no compacted session. Means go up while the incidents stay.
• One blended score. Refund-tool failures drown in Q&A faithfulness. Split reports by intent / stage.
• Exact match on prose. Paraphrase is the point of an LLM; match IDs, schema, and claims, not the whole string.
• Judge without a rubric. You paid for a second model's opinion of fluency.
• Same model as candidate and judge. It likes itself.
• Iterating until 100%. You fitted the prompt to 50 cases. Hold out a slice, or the next production miss is guaranteed.
• Unversioned gold. Policy changed; the eval still wants the old number.
• No trace_id on fail rows. You cannot open the retrieve span, so you rewrite the prompt again.
• Declaring victory on 20 cases and a 0.02 mean. That is noise. Look at the failure list.

Conclusion

Evals close the loop Essentials sketched. The model stack turns tokens into text; the app stack owns context, retrieval, tools, sessions, and traces. None of that tells you whether a change helped until a labeled set and a scorer say so — against a baseline, with failures you can open.

The rest of the series is what you measure. Measure the layer that actually broke.