Lab: Eval suite

Your Module 1 baseline was hand-graded. This lab makes it a running system: a reusable harness with assertion and judge checks, producing a scorecard on demand. This suite is the spine of every remaining module.

Step 1 — The harness

1. 1Implement evalharness.py from the lesson: run_evals(feature, golden_rows, checks, n_runs=1) returning the scorecard dict, with the per-check breakdown and the N-run averaging knob. n_runs defaults to 1 — a single pass, correct for a deterministic feature — and when you point it at your live model you pass n_runs=3 (or more) so each case's passed becomes a per-case pass rate, the thing you'll gate on rather than a lucky single pass.
2. 2Keep it feature-agnostic — it takes a callable and checks, nothing feature-specific. Prove it by pointing it at your feature via the run_feature entry point from Module 1.

Step 2 — Write the checks

1. 1Assertions first (aim for 5+): parses as expected type, required fields present, enums legal, output within a latency budget, no forbidden phrases. Each is a (row, output) -> CheckResult.
2. 2One judge check: implement the faithfulness (or tone) rubric from the lesson as a check that calls a model and parses its JSON score, passing at ≥4.
3. 3Calibrate the judge: hand-grade 15 outputs, run the judge on them, compute agreement. Record the number in BASELINE.md. If under 80%, sharpen anchors and redo — do not proceed with an uncalibrated judge.

Step 3 — Score and compare

1. 1Run the full suite on golden/v1. Read the per-check breakdown — which check drags the score?
2. 2Make ONE improvement to your feature targeting the weakest check (a prompt tweak, a validator). Re-run. Did the target check improve without regressing others?
3. 3Compare to your Module 1 hand-graded baseline: does the automated pass rate roughly match your manual one? A big gap means a check is wrong (too strict or too lax) — fix the check, not the feature.

Step 4 — Lock it in

Commit evalharness.py, your checks, and golden/v1.jsonl together. From now on, no change to the feature ships without a scorecard. You've just replaced 'I think it's better' with a number — the single most important habit in the course, and the thing that makes every later module possible.