Ship an evaluation pipeline

98% against 47% needs no statistics. 91% against 88% starts an argument in every PR. A gate turns that argument into an exit code.

By the end you'll be able to compare two agent versions on the same tasks with a paired confidence interval, write the ship/block rules down, and enforce them on every pull request with GitHub Actions. You can already measure variance (Lesson 1), build a dataset (2) and explain failures (3). This lesson automates the decision.

A raw drop is not a verdict

Your pass rate is an estimate. Evan Miller (Anthropic) argues that the real object of interest isn't the score on these tasks, but the score on the larger universe of tasks they were drawn from. So every score needs an error bar^[1]. For pass/fail scores, the standard error is^[2]:

SE = √( p · (1 − p) / n )        95% interval ≈ p ± 1.96 · SE

Plug in a typical small suite: 15 tasks at a 90% pass rate gives SE ≈ 0.077, so the interval is about ±15 points. A 5-point drop is invisible at that size. With 60 tasks the margin shrinks to about ±7.6, and with 200 to about ±4.2. Shrinking error bars costs tasks, and each halving of the margin needs about four times as many.

A real run, 2026-10-19: the seed suite on qwen2.5:7b, 5 trials per task. v1 passed 64%, v2 passed 44%: a 20-point drop, with six policy tasks falling from 100% to 20–40%. The paired 95% interval was −46 to +6 points. Twenty points, and 15 tasks still can't rule out "no change". That's why Lesson 2's challenge grows the suite.

Mental model: two error bars, not two numbers. When you compare versions, picture each score as a blurry blob. "Better" means the blobs are clearly apart. The model breaks down in a useful way: when both versions run the same tasks, the blobs move together, because a hard task is hard for both. Comparing per task, as below, is far sharper than comparing the two blobs.

Try it. This simulates a candidate that's truly worse by the amount you choose. Watch how the verdict depends on the number of tasks, and press Rerun a few times at small n.

Simulated data: each task gets a random difficulty, and both versions are sampled k times per task. The dashed red line is the true effect, which you never get to see in real life.

Average the trials, then pair the tasks

Two corrections make the comparison honest and sharp, both from Miller's paper^[2]:

1. Average each task's k trials first. Three trials of one task aren't three independent data points, because they share the task's difficulty. Treating them as independent makes intervals look too narrow. (The paper found clustered errors over three times larger than naive ones on popular evals^[1].) More trials per task do help, by reducing the noise within each task.
2. Compare paired differences. Both versions answer the same tasks, so subtract per task and analyse the differences. This "removes shared question-difficulty variance". Use it whenever you compare two models or versions^[1].

Worked example: the paired comparison in compare.ts

const pairs = ids.map((id) => ({ b: B.get(id).pass_rate,          1
                                 c: C.get(id).pass_rate }))
const d = pairs.map((p) => p.c - p.b)                              2
const half = crit95(d.length) * sd(d) / Math.sqrt(d.length)       3
const [lo, hi] = [mean(d) - half, mean(d) + half]
if (hi < 0) verdict = "BLOCK"                                      4
else if (lo < -gate.min_detectable_effect) verdict = "INCONCLUSIVE"
else verdict = "SHIP"

1. Each task's score is already the mean of its k trials (correction 1). Tasks are matched by id, so only tasks present in both runs count.
2. One difference per task (correction 2). A task both versions always pass contributes 0. That's the cancellation at work.
3. The paired standard error, Miller's Eq. 7: the standard deviation of the differences over √n. With fewer than ~30 tasks the multiplier is a t value (2.14 at 15 tasks), not 1.96. Small suites get wider, honest intervals.
4. The rules, in order. The full version also enforces a regression-suite floor and refuses runs that aren't comparable.

Write the rules down

A gate is only as good as rules everyone can read before the numbers arrive. Otherwise people decide after they see the numbers. The lab keeps them in gate.json:

• BLOCK if the whole interval is below zero: the candidate is worse beyond plausible luck. Also BLOCK if the regression suite falls under its floor.
• INCONCLUSIVE if the interval reaches below −min_detectable_effect: you can't rule out a drop you care about. The fix is more tasks or trials, not a shrug. It fails CI by default (inconclusive_blocks: true), because "we couldn't tell" isn't "it's fine".
• SHIP otherwise.
• Notice the asymmetry: an interval of −3 to −1 blocks, even though −3 is inside the 5-point tolerance. These rules block any drop you can prove, however small, and use the tolerance only when you can't prove anything. That's a deliberate, cautious choice. If you'd rather ship small proven drops (a non-inferiority rule: SHIP whenever the lower bound is above −5), say so in gate.json's justification.
• NOT COMPARABLE (exit 3) when the dataset hash, model, model digest or temperature differ. A different model changes the die itself (Lesson 1), so the numbers can't be paired.

Call the verdict

The pipeline

Hamel and Shreya draw the line clearly: "CI evals protect against known regressions before deployment", while production monitoring finds new failures. Keep the CI set small and purpose-built, and favour deterministic checks over LLM judges there, because CI runs constantly^[3]. Here's the whole loop, from pull request to verdict:

Where the model runs

Your model runs on your GPU, and GitHub's hosted runners have none. Running a 7B model on their CPUs would take most of an hour per PR. So the work splits in two. You run the eval locally and commit the result file with the PR. CI checks that the file really came from the code in that PR, then applies the gate. No API key, no cost, about a minute.

AGENT=v3 K=3 OUT=candidate/result.json npm run eval   # on your machine, then commit the file

The check is npm run verify. Every result file carries the hash of the agent code that produced it and the hash of the dataset it ran on. verify recomputes both from the PR's code. If you changed one word of the prompt after running the eval, the hashes differ and CI fails with "code changed since this run: re-run it". That's the same idea as a lockfile: the file says exactly what it was built from, and CI refuses a mismatch.

The honest limit: the hashes prove which code a result came from, not that nobody edited the numbers. For a solo portfolio repo, that's a fair trade, and you'll say so in Lesson 5's limitations. A team would add a self-hosted GPU runner, but GitHub advises against those on public repos, because a fork's PR could run code on your machine.

Worked example: ci/evals.yml

on:
  pull_request:                                   1
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-node@v7
        with: { node-version: 22 }
      - run: npm ci                               2
      - run: npm run validate                     3
      - run: npm run verify -- baseline/result.json candidate/result.json   4
      - run: npm run compare -- baseline/result.json candidate/result.json  5
      - uses: actions/upload-artifact@v7
        if: always()                              6
        with: { name: eval-report, path: results/report.md }

1. Run on every pull request, so a regression is caught before merge, not after.
2. npm ci installs exactly what's in the lockfile. Reproducibility starts with pinned dependencies.
3. A broken task fails fast.
4. Both result files must match this commit's agent code and dataset. A stale or synthetic result fails here, before any verdict.
5. compare exits non-zero on BLOCK, INCONCLUSIVE or NOT COMPARABLE, which fails the step, the job and the check.
6. Upload report.md even when the gate fails. That's exactly when you need it. The traces stay on your machine for npm run trace (Lesson 3).

The baseline is a file in git

The baseline is just the result file of the shipped version, committed as baseline/result.json. Every result file records what's needed to refuse a bad comparison: agent hash, dataset hash, model, the model's digest (the hash of the exact weights), Ollama version, temperature, seed and k. The digest matters because ollama pull can quietly move a tag like qwen2.5:7b to new weights, and that changes the die (Lesson 1). Change the baseline on purpose, in its own PR, when you ship a new version or switch models. The diff then shows exactly when and why the bar moved.

Practice: from memory

No scrolling up.

Fill the blanks

on:
  :
...
      - run: npm run  -- baseline/result.json candidate/result.json
...
      - uses: actions/upload-artifact@v7
        if: 

// paired standard error over n tasks
const half = crit95(n) * sd(d) / Math.sqrt()

# 1. your repo, seeded from the lab
cp -r labs/02-bench ~/code/agent-bench && cd ~/code/agent-bench && git init
mkdir -p .github/workflows && cp ci/evals.yml .github/workflows/
# 2. the baseline: v1, committed
AGENT=v1 K=3 OUT=baseline/result.json npm run eval
cp baseline/result.json candidate/result.json      # main = "candidate is the baseline", so main passes
# 3. push to GitHub, protect main (require the "evals" check)

Read or watch next

Primary source: A statistical approach to model evaluations, Anthropic (Nov 2024). Its five recommendations are the theory behind compare.ts.

• A statistical approach to model evaluations📄 Article · ~10 min · Anthropic Research, 2024The short version: standard errors, clustering, resampling, paired differences and power analysis.
• Adding Error Bars to Evals📄 Paper · ~40 min · Evan Miller (Anthropic), arXiv 2411.00640The formulas: Eq. 2 (Bernoulli SE), Eq. 7 (paired SE) and Eq. 9 (how many tasks you need to detect a given drop).
• Evals FAQ: CI/CD vs. production monitoring📄 Article · ~5 min for that question · Hamel Husain & Shreya ShankarWhat belongs in CI evals (small, purpose-built, deterministic) and what belongs in monitoring.
• Workflow syntax for GitHub Actions📄 Docs · reference · GitHubFor the on: pull_request filters (paths: to run evals only when the agent or dataset changes) and job settings.

💬 Unsure what minimum detectable effect fits your product, or how many tasks you'd need for it? Ask me. We can work through Miller's Eq. 9 with your own numbers.