Tracing and debugging agent failures

The eval says 9 trials failed. That's the alarm going off. The trace is the security footage.

By the end you'll be able to read an agent's trace, point at the span where it first went wrong, and prove a fix with a re-run. Then you'll turn a pile of failures into a short, counted list of failure types, so you fix the biggest one first. You'll work on bookstore-v2, a "small improvement" a teammate shipped.

Trace definition from the Evals FAQ (Husain & Shankar). Anthropic calls the same thing a transcript.

Why you have to read traces

A pass rate tells you that something broke. Only the run itself tells you why. Anthropic is blunt: "You won't know if your graders are working well unless you read the transcripts and grades from many trials", and "failures should seem fair: it's clear what the agent got wrong and why"^[1]. Reading traces also checks your eval. Sometimes the agent was right and the task was wrong.

Mental model: a trace is a stack trace for decisions. A stack trace shows which function called which. A trace shows which model call asked for which tool, what the tool returned, and what the model did next, all with timing. The model breaks down in one important way: the "code" that made each decision is a sampled model. Re-running won't reliably reproduce the bug, so the trace has to record inputs and outputs in full, or you're debugging from memory.

Using the standard names costs nothing and keeps your traces portable. Any OpenTelemetry backend (Langfuse, Phoenix, Jaeger) will understand them later. The conventions are still marked Development, so expect attribute names to shift^[2].

Worked example: tracing a tool call

From labs/02-bench/src/agent.ts. The whole tracer (src/trace.ts) is about 40 lines, so you can read every byte it records:

const ts = tracer.start(`execute_tool ${c.function.name}`, {     1
  "gen_ai.operation.name": "execute_tool",
  "gen_ai.tool.name": c.function.name,
  "gen_ai.tool.call.id": c.id,                                  2
  "gen_ai.tool.call.arguments": c.function.arguments,           3
}, root)                                                        4
const result = runTool(world, c.function.name, args)
const failed = "error" in result
tracer.end(ts, { "gen_ai.tool.call.result": JSON.stringify(result) },
           failed ? "error" : "ok")                             5

1. The span is named after the tool, so a waterfall reads like a story: chat → execute_tool refundOrder → chat.
2. The id the model gave the call. It links this span to the model output that requested it.
3. The raw arguments string, before parsing. If a small model sends broken JSON, that's exactly what you need to see.
4. The parent is the agent's root span, which makes the trace a tree.
5. Record the result and mark tool errors. Failed spans show up red in npm run trace.

Walk backwards from the symptom

The grader tells you the symptom. To find the cause, start at the symptom and walk backwards until you reach the first span whose output was already wrong, given correct inputs. Hamel and Shreya's heuristic for big traces: "focus on the first upstream failure", because errors compound and everything downstream is usually a consequence^[3].

At each span, ask one question: were the inputs right, and was the output right given those inputs? That splits agent failures into a few places to look:

Find the root cause

Notice that no span is red. Nothing errored: every tool call succeeded. Most agent bugs look like this. That's why you need graders on the outcome and traces to explain them, not just error logs.

From one failure to a failure taxonomy

Fixing failures one at a time is whack-a-mole. Do error analysis instead: the open and axial coding from Lesson 2, now with a trace for every note. npm run failures lists every failing trial and writes a CSV with empty note and category columns:

1. Open coding: for each failing trial, run npm run trace and write one sentence about the first thing that went wrong.
2. Axial coding: group the notes into 3–6 categories and count each one^[3].
3. Fix the biggest category first, as a new agent version. Then re-run.

Code these notes

Prove the fix

A fix is a hypothesis until a re-run says otherwise. Keep v2 frozen and add a v3 to src/agents/index.ts, so the history stays comparable. Then run two checks:

AGENT=v3 TASKS=outside K=5 npm run eval    # 1. the task that failed now passes, repeatedly
AGENT=v3 K=3 npm run eval                  # 2. nothing else broke (you'll gate this properly in Lesson 4)

Five trials, not one, because one passing roll proves nothing (Lesson 1). The task stays in the regression suite forever, so the bug can't quietly come back.

Practice: from memory

No scrolling up.

Fill the blanks: name the spans

const root = tracer.start(` ${agent.name}`)
const span = tracer.start(` ${MODEL}`, { "gen_ai.request.model": MODEL }, root)
const ts   = tracer.start(`execute_tool ${name}`, { "gen_ai.tool.": name }, root)
tracer.end(ts, { "gen_ai.tool.call.": JSON.stringify(result) })

cd labs/02-bench
AGENT=v2 K=3 npm run eval
npm run failures
npm run trace -- results/traces/<run>/<task>__t1.jsonl

Read or watch next

Primary source: the trace and error-analysis questions in the Evals FAQ by Hamel Husain & Shreya Shankar. Start with "Why is error analysis so important?" and "How do you review a trace that is really large?".

• AI Evals: Everything You Need to Know (Evals FAQ)📄 Article · skim by question · Hamel Husain & Shreya ShankarOpen and axial coding, "focus on the first upstream failure", and logging the whole workflow from trigger to business outcome.
• Why AI evals are the hottest new skill for product builders▶ Video · watch 09:56–23:54 · Lenny's Podcast with Hamel Husain & Shreya ShankarWatch someone actually read real traces, write notes, and turn them into categories.
• OpenTelemetry GenAI semantic conventions🛠 Spec repo · browse docs/gen-ai/ · OpenTelemetryThe span names and gen_ai.* attributes this lab uses. Read gen-ai-spans.md (execute_tool) and gen-ai-agent-spans.md (invoke_agent).
• Demystifying evals for AI agents📄 Article · ~25 min · Anthropic EngineeringRe-read the transcripts advice: reading them is how you check that the eval measures what matters.

💬 Stuck on a trace? Paste the npm run trace output into the terminal and we'll walk it backwards together.