Experiments and Prompt Optimization
Choose by experiment, not by taste.
Every team eventually has three plausible fixes and no way to read a diff and know which is better, by how much, or at what cost. Evals stop a regression; experiments pick a winner. This phase teaches the procedure that survives review: one change at a time, a frozen holdout, an interval on every claim, cost next to quality, and a gate that defends the winner after it ships.
What you learn
- Prompting from easy to hard: a seven-level ladder from a clear ask to optimizing with data
- Experiment design: one variable, train, dev and a frozen holdout, paired comparisons, bootstrap intervals, sample sizes
- Running the same experiment in promptfoo, Langfuse, Braintrust, Arize Phoenix and Inspect AI
- Automatic prompt optimization with DSPy (MIPROv2, GEPA) and vendor prompt improvers, and when it pays
- Online experiments: shadow tests, A/B tests with guardrail metrics, ramp-ups, and model selection that feeds routing
Tools you will actually touch
The assignment pack
A furniture retailer triages 40,000 WhatsApp messages a month with a prompt that returns valid JSON every time and still routes most messages to the wrong queue. A support lead, an engineer and a vendor each propose a fix. Decide by experiment which one replaces it, prove the gain on data nobody tuned against, and make sure the next tidy-up cannot quietly undo it.
- A versioned dataset with provenance on every item and a holdout frozen by a digest that a careful edit cannot get past
- A prompt ladder with one change per rung, run in pinned promptfoo with deterministic assertions and per-token costs
- An optimizer that draws examples from train, scores on dev and never reads the holdout
- A winner picked on dev by a written rule and confirmed on the holdout with paired win rates and a bootstrap interval above zero
- A CI gate that fails a tidy-looking regression, a quality regression and a cost regression, and passes the winner and a whitespace change
Prompts compared by eye in a playground, the dev score reported as the result, or a gate that passes a cheaper prompt with the injection defence deleted.
Frozen holdout, one change per variant, a selection rule written before looking, intervals on every comparison, cost and latency logged, a gate that holds in both directions.
A sample-size argument for the holdout, a cost-aware stopping rule for the search, a second harness reproducing the holdout number, and a shadow-and-ramp plan with guardrail metrics.
The tutorial pack
Run the experiment, then let a tidy-up try to sneak past
- 1Probe a deterministic model and see exactly which prompt features it rewards
- 2Build a stratified dataset with provenance, then freeze the holdout and try to move it on a copy
- 3Write a prompt ladder with one change per rung
- 4Let a greedy optimizer pick few-shot examples on dev, and read every step it took
- 5Run the ladder in pinned promptfoo and cross-check it against the optimizer, item for item
- 6Pick the winner by rule, read the holdout once, and report win rates with a bootstrap interval
- 7Gate the winner and attack the gate with quality, cost and whitespace changes
A colleague deletes the trust section to save 67 tokens a call. The prompt gets 16 percent cheaper and looks like housekeeping. Watch the gate fail it on the injection slice, then ask what would have happened if the dataset had no injection items.
- Winner against baseline and runner-up on the holdout, with paired intervals and seeds
- The dev-to-holdout gap of the searched prompt, explained item by item
- Cost per item for every variant, plus what the search itself cost
- Gate exit codes on the winner, the tidy-up, the planted regressions and a whitespace control
You must be able to answer
AI use is mandatory on every assignment. The integrity mechanism is defend-your-work, so these are the questions that decide whether the work counts.
- “Your searched prompt scored higher on dev than on the holdout. Which items caused the gap, and what would you change in the search?”
- “A vendor optimizer returns a prompt that scores 0.98 on its own dashboard. What do you do with it before a customer sees it?”
- “The offline experiment says the new prompt wins, but the A/B test shows re-queues flat and latency up. Which do you believe, and what do you ship?”
Want to be walked through this phase?
The packs are delivered with live teaching, reviewed assignments, and a defence.
Train with me