Eval suites for conversations

You've been building the eval suite all course without ceremony — the 10-question base, the 15-attack bank, the drill transcripts. This lesson organizes it into the machine that makes change safe. Conversation evals differ from the single-shot evals of Prompt Engineering in one important way: the unit of quality is the trajectory, not the reply — a perfect answer in turn 3 doesn't redeem an interrogation in turns 1-2 or a botched ending in turn 5.

• The golden set (target ~60 cases at launch, grown from real traffic after): single-turn checks (policy Q&A with expected citations — cheap, catch most regressions), scripted multi-turn flows (the customer's side pre-written, agent responses evaluated at each beat: clarify → confirm → resolve → release), the attack bank (all 15, plus every real incident forever after — incidents are eval cases with a budget), and the handoff drills (does the right trigger fire at the right turn, and does the package pass the 8-second test).
• The judge — an LLM grading transcripts against your rubric, one dimension at a time: grounded? (claims vs. provided context), on-persona? (against the spec, with the counter-examples as few-shots), scope-clean? correct-trigger? Each dimension pass/fail with a quoted justification — never a single 1-10 vibe score. And the judge gets calibrated: you hand-grade 20 transcripts first, measure agreement, and fix the rubric where you and it disagree. An uncalibrated judge is a random-number generator with good grammar (LLMOps teaches the full discipline; this is the working version).
• The regression gate: no prompt edit, knowledge update, or model swap ships without the full suite green — the same rule your automation and finance courses drilled, now guarding your brand's public voice. The suite runs cheap (it's an hour of model calls); the incident it prevents is not.

Grow the suite from production, deliberately

Launch-day suites are guesses; month-two suites are evidence. The pipeline: every honest-miss, every bad-CSAT transcript, every handoff the weekly hour flags as wrong → candidate eval case → dedupe → add with expected behavior written down. Twenty cases a month is a healthy pace. Within a quarter, your suite is your customers' actual hard cases — which is the moment eval scores start predicting CSAT instead of just correlating with hope.