Why LLM outputs are hard to test

Your agent passed yesterday and failed today, and nothing changed. That isn't a flaky test. It's the system showing you what it is.

By the end you'll be able to replace an exact-match test with property checks over many trials, and to say which of two numbers, pass@k or pass^k, your product actually needs. This is the first brick of your benchmark repo. Every later eval reuses the harness you run today.

These are Anthropic's definitions, from Demystifying evals for AI agents (2026), the primary source for this course.

A function vs. a loaded die

Normal tests work because the code under test is a function: same input, same output, so assertEquals is a complete specification. An LLM call is not a function. The model produces a probability distribution over next tokens and samples from it, token after token. The same input gives you a draw from a distribution of possible outputs.

Mental model: an LLM call is a loaded die you can't see. A test that checks a single roll tells you almost nothing. A good eval rolls the die many times and checks that the die is acceptable, meaning enough rolls land on answers you'd accept. The model breaks down in one way: the die isn't fixed. A new model version, a backend change or your own prompt edit re-weights it, which is exactly why you need regression evals.

Watch the die roll. Press ▶, use the chapter buttons, or turn on 🔇 narration. Around 20 seconds, and you control the pace.

"Just set temperature to 0"

Temperature 0 means "always pick the most likely token", so in theory the die has one face. In practice, Thinking Machines Lab sampled Qwen3-235B 1,000 times at temperature 0 with the prompt "Tell me about Richard Feynman" and got 80 unique completions. All 1,000 agreed for 102 tokens, then split at token 103^[1]:

The cause isn't the usual "floating point + GPU concurrency" story. It's batching: the server groups your request with whoever else is calling, and the numerical result of a forward pass depends on the batch size. Server load is invisible to you, so from your side the output varies^[1]. A seed doesn't close the gap either: Ollama accepts it but documents no guarantee^[2].

On your own machine the picture is calmer. Ollama serving you alone runs one request at a time, so the batch never changes. In a test run on 2026-10-19, qwen2.5:3b and qwen2.5:7b each gave 1 unique output in 30 trials at temperature 0. Don't read that as determinism. It holds for this machine, this Ollama version and these weights. Other hardware, a new version or parallel requests can change it, and ollama pull can quietly move a tag like qwen2.5:7b to new weights. That last one you can pin: record the model's digest, a hash of the exact weights.

So determinism isn't a setting you flip. Design tests that are correct despite variation, then record what you can (model, digest, temperature, seed) so a run is reproducible statistically.

Lab: roll the die

Here is the first step of our running example, a bookstore support agent. It triages a ticket into JSON: "Hi, order #4417 arrived with the cover torn off. I'd like my money back please." → {"intent":"refund","order_id":"4417"}.

Try this sequence: (1) run 20 trials at temperature 0 with exact match. (2) Switch to 1 and run again. (3) Flip the grader to properties without re-running. Click the red chips.

What you just saw: at temperature 1, exact match flags valid answers as failures (false alarms), so the team learns to ignore red builds. The property grader passes the harmless variations and still catches the three real bugs. A good grader is strict about what matters and indifferent to what doesn't.

Choosing a success criterion

A success criterion answers: what would have to be true for this output to be acceptable to the next step, or to the user? Graders form a ladder, from cheap and brittle to flexible and costly:

Rule: use the cheapest rung that still judges what the user cares about. For agents, Anthropic recommends grading the outcome (is the reservation actually in the database?) rather than the exact path or wording, because rigid checks on steps punish valid approaches you didn't anticipate^[4]. We'll climb to outcomes and model graders in Lesson 2.

Worked example: a test that survives variation

async function evalTriage(n = 20) {
  let passes = 0
  for (let i = 0; i < n; i++) {                  1
    const { text } = await chat(msgs, { temperature: 0.7 })
    const out = JSON.parse(extractJson(text))     2
    if (out.intent === "refund" &&                3
        out.order_id === "4417") passes++
  }
  const p = passes / n                            4
  expect(p).toBeGreaterThanOrEqual(0.95)          5
}

1. Many trials, because one roll is an anecdote.
2. Tolerate harmless formatting (fences, prose) before judging.
3. Check only what the next step depends on: the intent and the id, typed correctly.
4. The result is a rate, not a boolean.
5. The threshold is a product decision. "You don't necessarily need a 100% pass rate."^[5]

One trial is an anecdote: pass@k vs pass^k

Once your result is a per-trial pass rate p, there are two different questions you can ask about k trials^[4]:

• pass@k: does at least one of k attempts succeed? 1 − (1 − p)^k. It rises with k. Use it when one success is enough, such as generating candidates a human picks from.
• pass^k: do all k attempts succeed? p^k. It falls with k. Use it when users need consistency. A customer-facing support agent is a pass^k product.

At p = 0.75 and k = 3, pass^k is about 0.42, the example from Anthropic's guide. A "75% agent" disappoints a user who tries it three times more often than it pleases them. Drag p up to 0.95 and watch how much consistency each percentage point buys.

Capability vs. regression. A capability eval asks what the agent can do. It should start with a low pass rate, which gives you a hill to climb. A regression eval asks whether the agent still does what it used to, and should sit near 100%. Capability tasks that become reliable "graduate" into the regression suite^[4]. Because both are rates, "did it regress?" means "did the pass rate drop by more than noise?" Measuring that noise is Lesson 4.

Practice: from memory

No scrolling up. Retrieval is the point.

Place it on the chart

Fill the blanks

const p       = passes / 
const passAll = p **              // pass^k
const passAny = 1 - (1 - p) **    // pass@k
if (typeof out.order_id !== '') fail("order_id not a string")

cd labs/01-variance && npm install
ollama pull qwen2.5:3b && ollama pull qwen2.5:7b
TEMP=0   npm run trials
TEMP=0.8 npm run trials
MODEL=qwen2.5:7b TEMP=0   npm run trials
MODEL=qwen2.5:7b TEMP=0.8 npm run trials

Read or watch next

Primary source: Demystifying evals for AI agents, Anthropic. Read the definitions and the pass@k / pass^k section now. The rest will map onto Lessons 2–4.

• Demystifying evals for AI agents📄 Article · ~25 min · Anthropic Engineering, Jan 2026The vocabulary this course uses (task, trial, grader, transcript, outcome), plus grader trade-offs and "start with 20–50 tasks from real failures".
• Defeating Nondeterminism in LLM Inference📄 Article · ~20 min · Thinking Machines Lab, 2025The Feynman experiment above, and why batching (not "GPU randomness") is the culprit. Skim the kernel details. The intro and experiment are what matter here.
• Your AI Product Needs Evals📄 Article · ~30 min · Hamel Husain, 2024Read "Level 1: Unit Tests". It's the property-assertion idea from this lesson, applied to a real product.
• Why AI evals are the hottest new skill for product builders▶ Video · watch 04:57–23:54 · Lenny's Podcast with Hamel Husain & Shreya ShankarWhat evals are, then a demo of reading real traces. This previews Lessons 2 and 3.

💬 Anything unclear, or want to go deeper on one part (batching, sampling, the pass^k math)? Ask me in the terminal. I'm your teacher for this course, and questions are how the lessons get sharper.