Evals and Quality Gates
If you cannot measure it, you cannot ship it.
This is the most under-taught and highest-leverage skill in the field. Without evals, every prompt tweak is a coin flip and every model upgrade is a gamble. With them, you can change things fast because regressions fail the build instead of the customer.
What you learn
- Golden datasets: building them, keeping them honest, avoiding leakage
- CI eval gates: regressions fail the build, not the customer
- Online metrics and drift: watching quality after launch
- LLM-as-judge, its legitimate uses and its traps (position bias, self-preference, cost)
- When a cheap deterministic check beats an expensive judge
Tools you will actually touch
The assignment pack
Your team wants to swap the underlying model to cut cost by 60 percent. Nobody can say whether quality will hold. Build the eval suite that answers that question with evidence, and wire it so no future change ships without passing it.
- A golden set with documented provenance and no answer leakage
- Automated eval run producing per-dimension scores, not one blended number
- CI gate that fails the build on a seeded regression
- A written model-swap recommendation backed by the numbers
A handful of spot-check prompts run manually. No gate, no baseline.
Real golden set, automated scoring, CI gate proven to catch a seeded regression.
Judge bias is measured and mitigated, online metrics complement offline ones, and the recommendation states its own confidence limits.
The tutorial pack
Build an eval gate that actually blocks
- 1Collect 50 real queries and label the correct outcomes
- 2Split into a development set and a held-out set you do not tune against
- 3Write deterministic checks first, judges only where necessary
- 4Run the suite against your current system and record the baseline
- 5Seed a regression on purpose and confirm the gate fails the build
- 6Wire the gate into CI so it runs on every PR
Write a golden set where the expected answer leaks into the question. Watch your scores hit 95 percent and mean nothing. Then rebuild the set honestly and watch the real number appear.
- Baseline scores per dimension, with the held-out set reported separately
- Gate correctly fails on the seeded regression and passes on a no-op change
- Judge agreement with human labels on a sample
You must be able to answer
AI use is mandatory on every assignment. The integrity mechanism is defend-your-work, so these are the questions that decide whether the work counts.
- “How do you know your golden set is not leaking?”
- “Where is an LLM judge the wrong tool, and what did you use instead?”
- “Your eval score went up but users complained more. How do you investigate?”
- “What does your gate NOT catch?”
Want to be walked through this phase?
The packs are delivered with live teaching, reviewed assignments, and a defence.
Train with me