Course

Building Reliable AI Agents

Evals, test datasets, graders, tracing and regressions, in TypeScript, on local models with Ollama.

Final challenge

Compare two versions of your agent on a reproducible test set, and publish the benchmark, the automated tests and an analysis of the results as a public repo. That unlocks Benchmark. The brief and pass criteria are in Final: publish the benchmark.

Reference

Evals cheat sheet: vocabulary, grader ladder, pass@k / pass^k, task schema, span names, paired CIs, gate rules

Labs live in labs/: 01-variance for Lesson 1, and 02-bench (the agent, dataset, tracing and gate) for Lessons 2 to 5. Each lesson's challenge is real work you send back to your teacher in the terminal. Setup: README.md.