GuideCraft · Intermediate

Experiments and evals

Prove an AI change helps before and after it ships: cheap checks first, a golden dataset with a holdout, judges calibrated against people, small-sample statistics, CI gates, A/B tests, agent evals and today's tools.

Checked 13 Oct 202624 min read2 prompts

On this page

Why evals
The metric ladder
Your first eval this week
Members only: Build a golden dataset
Members only: Run offline experiments
Members only: LLM-as-judge done safely
Members only: Statistics at small sample sizes
Members only: Gates in CI
Members only: Online evaluation and A/B tests
Members only: Agent and trajectory evals
Members only: Tools compared
Members only: Mistakes to avoid
Members only: Exercises

An eval is a test for software whose output varies. It has three parts: a set of cases, a way to score each output, and a number you compare across versions of your prompt, model or pipeline. Without evals, every prompt edit, model upgrade and new tool is a guess, and your customers do the grading.

Why evals#

Four things go wrong without them:

The model changes under you. Vendors retire models on fixed dates and change defaults between versions. A Claude request that omits effort now runs one level lower on Claude Opus 5.5 than on Claude Opus 5, because the default moved from high to medium. Microsoft warns that Microsoft Copilot (formerly Microsoft 365 Copilot) moves to newer models automatically, so agent behavior shifts over time. A fixed set of cases tells you what changed.
Fixes break other cases. A prompt edit that fixes the case in front of you often breaks two you are not looking at. Only a fixed set of cases shows the trade.
Cost decisions need evidence. A cheaper model or a lower reasoning setting that scores the same on your cases saves money every day. One that scores 3 points lower may still be the right call, but you should know the number.
Reading outputs does not scale. OpenAI's eval guide lists "vibe-based evals" ("it seems like it's working") as an anti-pattern. You can read 20 outputs carefully. You cannot read 2,000 after every change.

The eval loop has seven steps:

Read real outputs and write down how they fail. 50 to 100 is enough to start. This is error analysis, and it decides what you measure.
Turn each failure mode into a check, the cheapest one that catches it (the metric ladder below).
Collect cases into a golden dataset with a holdout you rarely touch.
Run experiments: compare variants on the same cases, with statistics that respect small samples.
Gate releases in CI so a regression cannot merge silently.
Ship gradually and score production traffic with the same checks.
Feed new failures back into the dataset, and go around again.

Anthropic's engineers put the starting bar low: "20-50 simple tasks drawn from real failures is a great start." They also warn that "You won't know if your graders are working well unless you read the transcripts." Expect your definition of good to change as you grade. Researchers who studied this call it criteria drift: people need criteria to grade outputs, and grading outputs is how they discover the criteria (Shankar et al., 2024).

The metric ladder#

Climb only as far as the quality you need to measure requires. Each rung is slower, costlier and noisier than the one below it.

Rung	What it checks	Examples	Cost and noise
1. Deterministic checks	What code can decide	JSON parses and matches the schema; label equals the expected label; required fields present; no banned phrases; SQL runs; unit tests pass; the right tool called with the right arguments; length, latency and cost limits	Near zero cost, no variance. Run on every commit
2. Reference and retrieval metrics	Overlap with a known answer or source	Field-level precision and recall; retrieval recall@k (the right passage is in the top k results); context recall; numeric answers within a tolerance	Cheap. Needs labelled references
3. LLM judge	Qualities code cannot decide	Faithfulness to sources; tone; policy compliance; whether a reply resolves the issue	Model cost per case. Noisy, and only as good as its calibration
4. People and production	What users experience	Expert review of a sample; resolution rate; escalations; edits before sending; complaints	Slowest. The ground truth the other rungs approximate

Five rules keep the ladder honest:

Exhaust rung 1 first. Anthropic's grading guidance ranks code-based grading as the fastest and most reliable, and says to test an LLM grader for reliability before you scale it.
One criterion per check. A judge asked "is this good?" returns a number you cannot act on.
Prefer pass or fail with a written rule to a 1 to 10 scale. Hamel Husain writes that "tracking a bunch of scores on a 1-5 scale is often a sign of a bad eval process": nobody can say what separates a 3 from a 4. Binary verdicts are also easier to calibrate against people.
Treat text-overlap scores as weak signals. OpenAI lists relying only on academic metrics such as BLEU as an anti-pattern for open-ended text.
Tie every check to a failure mode you saw in real outputs. A check that never fails, on any version, measures nothing.

Your first eval this week#

You need no tools for the first round, only a spreadsheet:

Pick one task and pull 30 real inputs from it. Remove personal data first.
Write the expected output or a pass rule for each case, in columns: id, input, expected, notes.
Add three deterministic checks you can compute with a formula or 20 lines of code, such as "label matches", "reply under 120 words" and "no promised dates".
Run your current prompt and one variant on all 30. Count the passes for each, then read every case where the two disagree.

That is an eval. While you read outputs, write a one-line note on each failure, then turn the notes into failure modes with this prompt:

Prompt

Turn reviewer notes into failure modes

Level 3Any assistant

Below are notes a reviewer wrote while reading [NUMBER] outputs from our [TASK, FOR EXAMPLE SUPPORT REPLY DRAFTS]. Each note starts with the case ID.

<notes>
[ONE NOTE PER LINE, FOR EXAMPLE: sr-0042 promised a refund date the policy does not give]
</notes>

Group the notes into failure modes.
- Name each failure mode in five words or fewer and define it in one sentence a new reviewer could apply.
- List the case IDs in each mode and count them. A case can belong to more than one mode.
- Put notes that fit no mode under "Unclear" instead of forcing them in.
- Sort the modes by count, highest first.

Then suggest the cheapest check that could catch each mode: a code check (say exactly what it tests), a comparison with a reference answer, or a one-sentence judge rubric. Prefer code checks.

Return a table with columns: failure mode, definition, count, case IDs, suggested check.

Warning

Outputs from production often contain customer details. Paste only your notes and redacted cases into a consumer chat app, never raw transcripts. For raw transcripts, use an enterprise plan that does not train on your data.

Build a golden dataset

The rest of this guide turns that spreadsheet into a system you can trust: a golden dataset with provenance and a holdout, experiments with paired statistics, an LLM judge calibrated against human labels, a CI gate that runs on every pull request, production scoring with ...

Members only below this line

Get the rest of this guide free

What you get on this page:

1 prompt

Also in the members part

Run offline experiments
LLM-as-judge done safely
Statistics at small sample sizes
Gates in CI
Online evaluation and A/B tests
Agent and trajectory evals
Tools compared
Mistakes to avoid
Exercises

Sources

Checked against these on 13 Oct 2026.

Role playbookPlatform engineers, DevOps and SREA role that works this way Role playbookSoftware developersA role that works this way Role playbookCTOs and engineering leadersA role that works this way