Final · Milestone: Benchmark

Publish the benchmark

You have every piece. A benchmark is the version of them that a stranger can re-run, and then believe.

By the end you'll be able to publish a repo that makes one clear claim about two agent versions, backs it with numbers anyone can reproduce for free on Ollama, and says honestly what it can't show. This is the portfolio piece from your mission. A reviewer will judge it in two minutes on the README and in twenty on whether it runs.

claimThe one sentence your benchmark supports, such as "v3 restores v1's policy compliance at equal cost".

reproducibleSomeone else, on another day, gets results within your error bars using one command.

threat to validityA specific reason your numbers might not mean what they seem to.

What makes a benchmark believable

Each column is something you already built: result-file config (L2/L4), paired CIs and the gate (L4), judge calibration (L2) and the taxonomy (L3).

Miller's reporting advice applies directly: when you compare models, report "mean differences, standard errors, confidence intervals, and correlations"^[1]. Report the results that went against you too. A benchmark where the new version wins everything reads like marketing.

The repo

agent-bench/
├── README.md               ← the claim, the results table, how to reproduce, limitations
├── ANALYSIS.md             ← the written analysis (or a section of the README)
├── dataset/tasks.jsonl     ← your ≥20 tasks from Lesson 2, with sources
├── src/                    ← agent versions v1 · v2 · v3, harness, graders, tracer, gate
├── results/                ← committed result files for the runs you report (traces optional)
│   └── v1_ollama_qwen2.5-7b_….json  v2_…json  v3_…json
├── baseline/result.json    ← what CI compares against
├── candidate/result.json   ← the latest PR's result, checked by npm run verify
├── gate.json               ← the rules, justified in the README
├── .github/workflows/evals.yml
└── package-lock.json       ← pinned dependencies

Worked example: the README's results section

## Results                                                        1
22 tasks × 5 trials · temperature 0.3 · dataset `3f9c…` · 2026-11-02

| model (digest)            | v1            | v2            | v3            |   2
|---------------------------|---------------|---------------|---------------|
| qwen2.5:7b  `845dbda0`    | 0.78 ± 0.11   | 0.55 ± 0.14   | 0.80 ± 0.10   |
| llama3.1:8b `…`           | 0.70 ± 0.13   | 0.49 ± 0.15   | 0.74 ± 0.12   |

v3 − v1 (paired, qwen2.5:7b): +2 pp, 95% CI −4 to +8 → no detectable change   3
v2 − v1 (paired, qwen2.5:7b): −23 pp, 95% CI −38 to −8 → regression (blocked in CI, #4)

Biggest failure type in v2: refunds without a policy check (14 of 21)  4
Cost per full run: $0, about 4 min on an 8 GB GPU (Ollama 0.40.6)       5

The numbers above are a template, not results. Yours come from your runs.

Say what was run before showing any number: n tasks, k trials, temperature, dataset hash, date.
Every score has its error bar (the 95% interval over tasks, using the t value for small n, as in Lesson 4), for every version. A second model row is optional, and it shows whether your result depends on the model.
The comparisons that matter, as paired differences with intervals, and their plain-language meaning. "No detectable change" is a result. Say it.
Connect the numbers to the failure taxonomy (Lesson 3). That turns a scoreboard into an explanation.
Time and hardware, so a reader knows what reproducing it will take.

Write the analysis: claim → evidence → caveat

Every paragraph of ANALYSIS.md follows the same shape: one claim, the numbers that support it (with intervals), and the caveat that limits it. Then a section of threats to validity. The ones that apply to almost every small agent benchmark:

Small n. 20–30 tasks can't detect small differences (Lesson 4). Say what size of change you could have detected.
Author bias. The same person wrote the agent, the tasks and the rubric, and may have tuned v3 to the test set. Mitigation: tasks from observed failures, and a held-out set you didn't look at while fixing.
Drift. ollama pull can move a tag to new weights. You recorded the model digest and Ollama version in every result file, so say whether they stayed the same across the runs you compare.
Self-reported results. CI checks which code produced each committed result (Lesson 4), not that nobody edited the numbers. Say so, and point readers at the one command that re-runs any row.
Judge reliability. If a model judge contributes to the score, report its TPR and TNR and how many labels they come from (Lesson 2).
Synthetic environment. A six-order in-memory store isn't production. Say what it does and doesn't stand in for.

Supported, or not?

Practice: from memory

What lets a stranger verify your table without trusting you?

v3 − v1 is +1 pp with a 95% CI of −3 to +5. What do you write?

Which is a threat to validity you created yourself in Lesson 3?

Why does your results table give each model's digest, not just its name?

Fill the blanks: "How to reproduce"

npm 
npm run 
AGENT=v3 K=5 OUT=results/v3.json npm run 
npm run  -- results/v1.json results/v3.json

🏆 Final challenge: Benchmark

Publish agent-bench (or your own name for it) as a public GitHub repo, and send me the link. I'll clone it and run it.

Pass criteria:

Reproducible: a fresh clone runs npm ci && npm run validate clean, and the README's commands reproduce the results table on Ollama, within its error bars. Committed result files carry their config, hashes and model digest.
Comparable: at least two agent versions (v1 and your v3, ideally v2 too) on the same 20+ task dataset, with paired differences and 95% CIs. A second local model (for example llama3.1:8b) is a bonus, not a requirement.
Gated: the evals workflow runs on pull requests, with links to one blocked run and one passing (or honestly inconclusive) run.
Honest: the analysis states one main claim with its numbers, links it to the failure taxonomy, and lists at least three concrete threats to validity for this benchmark.

When I've reviewed it, I'll write the final learning record and mark Benchmark mastered. That completes the mission.

Read or watch next

Primary source: Adding Error Bars to Evals by Evan Miller. Read the recommendations on what to report. It's the bar your results section should clear.

Adding Error Bars to Evals📄 Paper · ~40 min · Evan Miller (Anthropic), 2024How to report comparisons: mean differences, standard errors, confidence intervals and correlations.
Demystifying evals for AI agents📄 Article · ~25 min · Anthropic EngineeringRe-read "Maintain and use the eval long-term" (saturation, graduating tasks into regression suites) before writing your Limitations section.
Evalite🛠 Tool · Matt Pocock · TypeScript, Vitest-basedA possible next step: port your harness to a TS-native eval runner with a trace UI. You'll understand what it does under the hood, because you built it.

💬 Want a review before you publish? Send me the README draft, or ask me to role-play a skeptical hiring manager reading it.