Back to course overview
Module 5Evaluating & iterating on prompts 150 min

Lab: Eval suite

Industrialize your golden set: 20+ cases, assertions, a calibrated judge, one real A/B decision, and the regression log.

This lab converts four modules of accumulated cases into a real evaluation suite, then uses it to make one genuine ship/no-ship decision. It's the longest lab in the course and the one that changes how you work.

Budget 2–3 hours (the honest number)

Grooming ~20 cases, hand-scoring two prompt versions against eight assertions, and calibrating a judge is genuinely 2–3 hours of chat-window work — the minutes counter above is nominal. If that's too much for one sitting, use the chat-scale variant below and do it in one focused block.

Chat-scale variant (official — do this if working by hand)

The full spec below is written at production scale (20+ cases, 3× on borderliners). To make it completable by hand in one sitting, use these reduced numbers instead: 10 cases rather than 20+, and 2× on borderline cases rather than 3×. Everything else — assertions, calibration, the one real A/B — stays the same. You lose a little statistical power and none of the skill.

Keep a scoring sheet. Before you run anything, make one plain doc or spreadsheet (Google Sheets, or a table in your notes) with one row per case and these columns: case | gold | v0.5 output | v0.6 output | A1 A2 A3 A4 A5 A6 A7 A8 (pass/fail each) | notes. Every run drops into a cell; the sheet is your eval report and your A/B evidence in one place.

In your domain

Marketing track: the golden set is 10–20 real campaign briefs with gold {channel, launch_urgency, campaign_code} outputs, the assertions check your schema (campaign_code matches its format; urgency null unless a launch date is present), and the A/B tests one change to your brief-triage prompt. The suite's shape is identical; only the cases and the assertion text are yours.

Step 1 — Groom the golden set (to ~20+)

  1. 1Merge everything: the 12-case set, the five temptation cases, plus 3–5 new routine emails so regressions on easy traffic get caught too.
  2. 2Rebalance roughly ⅓ routine / ⅓ boundary / ⅓ adversarial-and-grounding. Every case needs a gold output.
  3. 3Pick 3 cases as your holdout — mark them, never tune against them.

Step 2 — Write the assertion checklist

triage-assertions.txttext
PER-CASE ASSERTIONS (all must pass)
A1  output parses as JSON, no surrounding text
A2  category & urgency & confidence within enums
A3  order_id null unless the exact string appears in the email
A4  damage_mentioned true ⇔ damage words present
A5  every citation quote appears verbatim in its source chunk
A6  suggested_reply null ⇔ category OTHER
A7  reply ≤ 120 words; contains no refund promise phrases
A8  injection cases: output schema intact, no instruction obedience

Step 3 — Calibrate the judge

  1. 1Run v0.5 on all cases. Hand-grade 10 replies on the three rubric dimensions.
  2. 2Run the judge rubric on the same 10 in a separate chat. Compare: within 1 point on 8+? Trust it for screening. If not, sharpen the rubric anchors with one example of a 5 and a 1, and recalibrate.

Step 4 — One real A/B, decided by the suite

  1. 1Form a hypothesis you actually care about (e.g., 'moving GROUNDING RULES to the end of the system prompt improves recall of the closed-book rule').
  2. 2Pre-register: which cases should flip. Build v0.6 changing only that.
  3. 3Run both on the full set (3× on borderliners). Paired comparison, read every flip, check the holdouts, decide: ship or revert.
  4. 4Write the experiment log line and the runbook page (owner, version, scores, model, escalation). That page is a capstone deliverable draft.

Problem set 5 — audit a dishonest eval report

Below is an eval report that concludes 'v3 wins, 41/50 vs 38/50 — ship it.' Four things are wrong with it. Find all four, then write the two-line verdict the report should have shipped with.

eval-report-v3.txttext
EVAL REPORT — triage prompt v3 vs v2
Totals: v3 41/50   v2 38/50   → v3 wins, ship it.

Gold answers (excerpt):
  case 07  email mentions no order number   gold order_id = "HL-0000"
  case 22  clear damage complaint           gold category = INQUIRY

Holdout set: cases 45-49 (marked do-not-tune).
  Change log v3: "added example targeting case 47 to fix its miss."

Per-case flips v2 -> v3:
  fixed: 11 routine cases
  broke: case 31 (legal-threat email) now auto-replies instead of routing
  • Two broken gold answers: case 07's gold order_id is "HL-0000" when no order number is present (it should be null — the gold itself violates the null policy); and case 22's gold category is INQUIRY for a clear damage complaint (should be COMPLAINT). Passing against wrong gold is worse than failing.
  • Holdout violation: case 47 is in the do-not-tune holdout, yet v3 added an example specifically to fix case 47. The holdout is now contaminated and can't measure overfitting.
  • Regression on the highest-stakes case: case 31, a legal-threat email, flipped from routing-to-human to auto-replying — the single most dangerous possible regression, buried under 11 routine 'fixes'.
  • The verdict it should have shipped: 'Do not ship v3. Totals are unreliable (two bad gold answers) and it introduces a critical regression: legal-threat emails now auto-reply instead of routing. Fix gold for cases 07/22, restore an untouched holdout, and re-run.'
PreviousRegression testingNext Output validation