Reference · Building Reliable AI Agents

Evals cheat sheet

The compressed version of all five lessons. Print it, or keep it open while you work in the lab.

Vocabulary

TermMeaning
taskOne test case: defined inputs plus success criteria.
trialOne attempt at a task. Run several, because outputs vary.
graderLogic that scores one aspect of a trial. A task can have several.
transcriptThe full record of a trial: outputs, tool calls, intermediate results. Also called a trace.
outcomeThe final state of the world after a trial (the refund row exists), not what the agent says.
reference solutionA known-good run that passes every grader. Proves a task is solvable.
spanOne timed step in a trace (chat, execute_tool, invoke_agent). Spans nest.
root causeThe first span whose output was wrong given correct inputs.
paired differencePer task: candidate score − baseline score.
gateWritten rules turning a comparison into SHIP / BLOCK / INCONCLUSIVE.

Source: Anthropic, Demystifying evals for AI agents (2026).

Why the same input gives different outputs

SourceFixable?What to do
Sampling (temperature > 0)PartlyLower temperature, but never rely on it.
Server batching changes numerics, even at temperature 0Not by youRun many trials and grade rates. 1,000 runs → 80 unique
One request at a time on your own machineOften repeatableMeasured on Ollama: 30 runs at temperature 0 → 1 unique output. Repeatable here isn't repeatable elsewhere: other hardware, Ollama versions or parallel load can differ.
Model updates (ollama pull can move a tag to new weights)Yes: pin itRecord model and its digest (from /api/tags) in every result, and refuse comparisons across digests.
seedBest effortOllama accepts it but documents no guarantee.

Grader ladder

RungExampleUse when
Exact matchout === goldenOutput is a closed enum or ID, never free text.
Normalisedtrim, lowercase, sort keysRarely. Usually a sign you want properties.
Propertiesparses · field = value · type · regex never matches a UUIDStructured output, and the default unit test.
Outcomerefund row exists in DBAgents that act on the world. Prefer it over checking the path.
Model graderLLM judge + binary rubricOpen-ended quality. Calibrate against human labels.
Humanexpert reviewBuilding labels and calibrating judges.

Rule: use the cheapest rung that still judges what the user cares about. Be strict about what matters and indifferent to what doesn't.

Formulas

MetricFormulaQuestion it answers
pass rate ppasses / nHow often does one trial pass?
pass@k1 − (1 − p)^kDoes at least one of k succeed? (candidates, retries a human picks from)
pass^kp^kDo all k succeed? (consistency, customer-facing)

Anchors: 0.75³ ≈ 0.42 · 0.9⁵ ≈ 0.59 · 0.95¹⁰ ≈ 0.60 · 0.99¹⁰ ≈ 0.90.

Capability eval: starts low, a hill to climb. Regression eval: sits near 100% and guards what already works. Reliable capability tasks graduate into the regression suite.

Reference model: local client

import OpenAI from "openai"                          // used as an HTTP client: Ollama speaks this format
const client = new OpenAI({ baseURL: "http://localhost:11434/v1/", apiKey: "ollama" })
// Ollama: no tool_choice, no n, no logprobs (as of Oct 2026)

Reference model: trials + properties

let passes = 0
for (let i = 0; i < n; i++) {
  const res  = await client.chat.completions.create({ model, messages, temperature, seed })
  const text = res.choices[0].message.content ?? ""
  const json = text.match(/\{[\s\S]*\}/)?.[0]        // tolerate fences and prose
  const out  = json ? JSON.parse(json) : null
  if (out?.intent === "refund" && out?.order_id === "4417") passes++
  log({ text })                                       // keep the transcript
}
const p = passes / n                                 // a rate, compared to a threshold

The full version is in labs/01-variance/src/. The agent version, with tools, tracing and the gate, is labs/02-bench/.

Dataset (Lesson 2)

{ "id": "refund-outside-window", "suite": "regression" | "capability",
  "tags": ["edge", "no-act"], "source": "seed" | "observed: <run> <task>#<k>",
  "ticket": "…",
  "expect": { "refunds": [],                 // the OUTCOME, checked first
              "must_call": ["lookupOrder"],  // path checks only for real rules
              "reply_must": ["30"] },        // loose regex, never exact wording
  "reference": { "calls": [ … ], "reply": "…" } }   // npm run validate replays it
Test every task forWhy
Two experts would agree on pass/failOtherwise you're measuring the ambiguity
A reference solution passes itA 0% task is usually a broken task
It grades the outcomePath checks punish valid approaches
The suite covers "act" and "don't act"One-sided evals create one-sided optimisation

Sources: 20–50 tasks from real failures to start (Anthropic). Error analysis: open coding (one note on the first failure per trace) → axial coding (group + count) → stop at saturation (Husain & Shankar). Synthetic: dimensions first, 20 tuples by hand, then expand.

Model judges: binary · one dimension per judge · an UNKNOWN way out · calibrated on your labels. Report TPR = failures caught ÷ all failures and TNR = good passed ÷ all good, never raw agreement.

Tracing (Lesson 3)

Span nameKey attributes
invoke_agent {agent}gen_ai.agent.name, ticket, final reply
chat {model}gen_ai.request.model, .temperature, gen_ai.usage.*_tokens, gen_ai.response.finish_reasons, output messages
execute_tool {tool}gen_ai.tool.name, gen_ai.tool.call.id, .arguments, .result

OpenTelemetry GenAI semantic conventions, status Development (Oct 2026).

First wrong step is…Seen as
a model decisionchat output: wrong or skipped tool, wrong args
a tool bugexecute_tool result wrong given right args
a misread resultright result, next chat contradicts it
an eval bugevery span right, grader still ✗

npm run failures → npm run trace -- <file> → fix as a new version → re-run the task with K=5 and the suite with K=3.

Comparing versions (Lesson 4)

QuantityFormulaNote
SE of a pass rate√(p(1−p)/n)15 tasks at 90% → ±15 pp at 95%
task scoremean of its k trialstrials of a task aren't independent
paired SEsd(dᵢ) / √n, dᵢ = candᵢ − baseᵢMiller Eq. 7
95% CImean(d) ± t·SEt = 2.14 at n = 15, → 1.96 for large n
VerdictRule (gate.json)
NOT COMPARABLEdataset hash, model, model digest or temperature differ, or synthetic data
BLOCKregression suite < 95%, or the whole CI < 0
INCONCLUSIVECI reaches below −5 pp (fails CI by default)
SHIPotherwise

CI without a GPU: run the eval locally, commit candidate/result.json with the PR. CI runs npm run verify (agent hash and dataset hash must match the PR's code) and then npm run compare.

Publishing checklist (Final)

Go deeper