Lab: The eval suite

Assembly day. Everything you've logged since Module 1 becomes one suite with one score — and then you'll break the agent on purpose to watch the gate work. (Spreadsheet + chat window remains a fully legitimate harness; the discipline, not the tooling, is the certification.)

1. 1Assemble the golden set (~40 cases minimum): the 10-question base (with expected citations noted), 6 scripted multi-turn flows (write the customer side for: happy return, clarify-fork, angry escalation, out-of-scope redirect, honest miss, three-loop), all 15 attacks, the 4 handoff drills, and 6 canary questions pinned to every number in the corpus (return days, credit window, prices). Each case: input(s), expected behavior in one sentence, and which rubric dimensions apply.
2. 2Write the judge rubric — four dimensions (grounded / on-persona / scope-clean / right-trigger), pass-fail each, justification-quote required. Then calibrate: hand-grade 12 transcripts yourself first, run the judge on the same 12, and reconcile every disagreement by fixing either the rubric wording or your own inconsistency (both happen). Target: ≥10/12 agreement before trusting it at scale.
3. 3Run the baseline: full suite against current Harbor Helper. Record the scorecard — overall pass rate, per-dimension rates, and the failure list. Expect 80-90%; a perfect score at this stage means your suite is soft (return to the red-team escalation prompt).
4. 4Prove the gate: make a plausible-looking bad change — 'improve' the persona section to be more enthusiastic and helpful (the classic well-meaning edit) — and re-run. Watch what fails: typically persona counter-examples, a compensation attack (enthusiasm leaks generosity), and a refusal that turned apologetic-vague. Revert, re-run, green. That loop — change, gate, catch, revert — is the muscle memory this lab exists to install.
5. 5Wire the growth pipeline on paper: where new cases come from (miss log, CSAT flags, weekly hour), who adds them, monthly cadence, and the metric definitions from the scorecard lesson written as signable one-liners. This document + the suite is what Module 6's launch gets judged against.