Lesson 2 · Milestone: Dataset

Designing an evaluation dataset

Lesson 1 measured one prompt. Your agent will see thousands of tickets. A dataset is how you choose the few dozen that speak for all the rest.

By the end you'll be able to write tasks that are graded on outcomes, proven solvable by a reference solution, and balanced between "act" and "don't act", and to check whether a model-graded rubric agrees with you. From here on the course uses a real tool-calling agent: the bookstore agent in labs/02-bench/, with lookupOrder, refundOrder and searchCatalog.

taskOne test case: an input, a starting world, and success criteria.
outcomeThe state of the world when the run ends, such as "a refund row exists". Not what the agent says.
reference solutionA known-good run that passes every grader. It proves the task can be solved.
rubricWritten criteria a model (or human) grader applies when code can't judge.

Definitions from Anthropic's Demystifying evals for AI agents (2026), the course's primary source.

Where tasks come from

Don't start by brainstorming clever test cases. Start from evidence. Anthropic's advice is concrete: "20-50 simple tasks drawn from real failures is a great start", and later "converting user-reported failures into test cases ensures your suite reflects actual usage"[1]. Before you have users, your own runs are the failures: read the transcripts and write down what went wrong.

SOURCES Real failuressupport queue, bugs Your own tracesnpm run failures Synthetic tuplesintent × condition Open codinga note per trace Axial codinggroup + count Tasks.jsonl stop when new traces stop revealing new failure types "theoretical saturation"
Error analysis, after Hamel Husain & Shreya Shankar's Evals FAQ.

Open coding means writing a free-text note on each trace: what went wrong, in your own words. Note the first failure you see, "because upstream errors can cause downstream issues". Axial coding means grouping those notes into a small failure taxonomy and counting each group. You stop when new traces stop revealing new failure types, which the FAQ calls theoretical saturation[2]. For synthetic tasks, the FAQ says to define dimensions first (intent × condition × persona) and to write about 20 tuples by hand before you let a model expand them. Otherwise generated tickets all sound the same[2].

Mental model: a dataset is a map of your product's risk, not a random sample of its traffic. You over-sample the expensive mistakes (a wrong refund costs money) and the tricky edges, because that's where versions differ. The model breaks down if you take it too far: a suite with only edge cases misses a regression on the boring, common path. Keep some plain happy-path tasks too.

Anatomy of a task

Here is one line from labs/02-bench/dataset/tasks.jsonl, spread out so it can be annotated:

{ "id": "refund-outside-window",
  "suite": "regression", "tags": ["edge", "no-act", "policy"],      1
  "source": "seed",                                                 2
  "ticket": "I want a refund for order 3301, I didn't like the ending.",
  "expect": {
    "refunds": [],                                                  3
    "must_call": ["lookupOrder"],                                   4
    "reply_must": ["30"] },                                         5
  "reference": {                                                    6
    "calls": [{ "name": "lookupOrder", "args": { "order_id": "3301" } }],
    "reply": "Order 3301 was delivered 75 days ago, which is outside our 30-day refund window..." } }
  1. Suite and tags. Regression tasks should pass nearly always. Capability tasks leave room to climb. Tags are your coverage dimensions, so you can report a pass rate per tag.
  2. Provenance. When a task comes from a failure you saw, record which run and trial. A year from now, that line explains why the task exists.
  3. The outcome. Exactly which orders end up refunded. Here the right answer is none. The starting world (order 3301, delivered 75 days ago) comes from fixtures that are rebuilt fresh for every trial, with a frozen clock so "75 days" never drifts.
  4. A path check, used sparingly. Anthropic warns that grading the exact path punishes valid approaches[1]. "Looked the order up before deciding" is a business rule, not a style preference, so it earns a check.
  5. A reply property. A loose regex on what the customer reads: the reply must mention the 30-day rule. It doesn't check exact wording.
  6. The reference solution. npm run validate replays these calls in a fresh store and runs the real graders. If the reference fails, the task is broken, not the agent.

Watch one task go through the harness, and see what goes wrong if you only grade the words. Press ▶, use the chapters, or turn on 🔇 narration.

Four tests for every task

  1. Two experts would agree. "A good task is one where two domain experts would independently reach the same pass/fail verdict"[1]. "There's a typo on page 3, kind of annoying" fails this test: is that a refund request or not? Rewrite the ticket, or decide the policy, before you grade it.
  2. It's solvable, and you've proven it. Every task carries a reference solution that passes its graders. If a task sits at 0% across many trials, Anthropic says that "is most often a signal of a broken task, not an incapable agent"[1].
  3. It grades the outcome. Check the world first. Add path checks only for real rules, and reply checks only for facts the customer must hear.
  4. The suite tests both directions. "Test both the cases where a behavior should occur and where it shouldn't. One-sided evals create one-sided optimization"[1]. A suite that only asks for refunds rewards an agent that refunds everything. A suite that only says no rewards an agent that never refunds, and that's the side the seed set leans to.
THE 15 SEED TASKS IN labs/02-bench · from npm run validate expects NO refund12 expects a refund3 ← too few edge7 normal5 adversarial · negative21
The seed set leans hard toward "don't act". Fixing that imbalance is part of this lesson's challenge.

Sort the candidates

When code can't judge: a rubric

Some qualities aren't a regex. "Does the reply only claim what the tools actually did?" needs judgement. Anthropic's rule is "deterministic graders where possible, LLM graders where necessary", with three habits: grade one dimension per judge, give the judge a way out ("Unknown"), and calibrate it against humans[1]. Hamel and Shreya add: make it binary. Pass/fail "force[s] clearer thinking and more consistent labeling" than a 1–5 scale[2]. The lab's judge (JUDGE=1) follows all four:

You grade ONE thing: is the support reply FAITHFUL to the tool results?
PASS: every action or fact the reply states is supported by the tool results below.
FAIL: the reply claims an action that didn't happen, states a fact the tools
      contradict, or invents a fact the tools never returned.
UNKNOWN: you can't tell from the information given.
Ignore tone, length and politeness. Answer with exactly one word.

Trust, but measure. Label some replies yourself, then compare. Don't report raw agreement: if only 5% of replies are bad, a judge that always says PASS agrees 95% of the time and catches nothing[3]. Report two rates instead:

JUDGE SAYS FAILPASS you: badyou: good caughttrue positive missedfalse negative false alarmfalse positive passedtrue negative TPR = caught ÷ all bad TNR = passed ÷ all good "positive" = a failure found, so TPR is the judge's catch rate
TPR and TNR, as recommended in Hamel Husain's LLM-as-a-Judge guide.

Ten labels won't give you a precise number. The guide says that below about 60 examples confidence intervals are often too wide to be useful[3]. But ten will show you how a judge fails, which is the first thing to learn.

Practice: from memory

No scrolling up.

One task fails all 30 trials on two different models, while similar tasks pass. What should you suspect first?

For "refund order 3301" (outside the 30-day window), which check matters most?

Every task in your suite asks for a refund. What's the biggest risk?

Your judge agrees with your labels 95% of the time. Failures are rare. What do you report?

During open coding, a trace has three things wrong. What do you write down?

Fill the blanks: a new task

The book really did arrive damaged, but order 3301 was delivered 75 days ago. This task isn't in the seed set, so you can't copy it.

{ "id": "refund-damaged-outside-window", "suite": "",
  "ticket": "Order 3301 arrived damaged back in July. Refund please.",
  "expect": { "refunds": [],
              "must_call": [""] },
  "reference": { "calls": [{ "name": "lookupOrder", "args": { "order_id": "" } }],
                 "reply": "Order 3301 was delivered 75 days ago, outside our 30-day window, so I can't refund it." } }

Hints: it's a rule the agent must always follow, and a good reason doesn't override the 30-day window. Leave a blank empty if the right answer is "nothing".

🏆 Milestone challenge: Dataset

Turn the 15 seed tasks into your dataset. The lab is labs/02-bench/ (see its README).

cd labs/02-bench && npm install
npm run validate
AGENT=v1 K=3 npm run eval                        # Ollama, qwen2.5:7b, about a minute on a GPU
npm run failures                                 # failing trials + an error-analysis CSV
  1. Error analysis. Read every failing trial and at least 10 passing ones. Open-code them in the CSV's note column. If v1 fails too rarely on your model to give you material, also run AGENT=v2 or a small Ollama model: failures from any real run count.
  2. Grow the dataset to 20+ tasks in dataset/tasks.jsonl. Turn observed failures into tasks, and set "source" to the run and trial they came from. Fix the act / don't-act imbalance.
  3. Calibrate the judge. Run once with JUDGE=1. Label 10 replies faithful or unfaithful before you look at the judge's verdicts, then compare.

Send me back: your new tasks (the diff), the output of npm run validate, the run summary, and your 10-label table with TPR, TNR and two sentences on whether you'd trust this judge in CI.

Pass criteria (check yourself before sending):

When I've reviewed it, I'll write a learning record and unlock Dataset. Your tasks.jsonl becomes the test set of your benchmark.

Read or watch next

Primary source: Demystifying evals for AI agents, Anthropic. Read the section "Going from zero to one: a roadmap to great evals for agents". Its first two steps are this lesson.

💬 Unsure whether a task is ambiguous, or how to word a rubric? Ask me in the terminal. Paste the task and I'll pressure-test it with you.