Golden sets & judges

The machinery, at the altitude a PM operates it:

• The golden set is a product artifact — curate it like the roadmap. It's the concrete answer to 'what must this feature handle?': the common cases in proportion, every caveat from your prototype test, each error-map quadrant represented, the edge cases user research surfaced, and — permanently — every real incident after launch. The PM's curation duties: does the set still look like production traffic (quarterly check against real distributions)? does anything in it embarrass the quality contract (a case no dimension covers = a contract gap)? and is it growing from reality (Module 4's loop) rather than from engineering's imagination?
• LLM judges: the calibration ritual is your warrant to trust the numbers. A judge (a model grading outputs against your rubrics) makes eval cheap enough to run on every change — if calibrated: humans grade a sample, judge grades the same sample, agreement gets measured, rubric gets fixed until agreement is high, and the ritual repeats on a schedule (judges drift when models or rubrics change). Name the bar: below roughly 85% raw agreement between the judge and human graders on the same sample, treat every dashboard number the judge produces as directional, not decisive — recalibrate the judge before you trust it. The PM doesn't run the calibration; the PM demands its receipts — 'what's our judge-human agreement, measured when?' is a question you're now licensed to ask in any eval review, and the answer 'we haven't checked lately' means every dashboard number has an asterisk.
• Human panels stay for what judges can't hold: the caring-vs-surveilling dimension, brand feel, the monthly 'read 20 real transcripts' session that no metric replaces (every operator course in this academy converges on the transcript hour; the PM version reads for product truth — are users getting what we promised? — not just correctness).

Reading the dashboard like a PM (four reflexes)

• Ask for the distribution, not the average — 94% overall with one dimension at 78% in the expensive quadrant is a launch-blocker wearing a passing grade.
• Ask what the set is — a score means nothing without knowing what it's computed over; 'we're at 96%' on a set that under-represents the hard cases is the pilot-to-scale trap in eval costume.
• Watch trends across versions, not points — the question is 'did the last change move quality, where, and what regressed?'; single scores are weather, deltas are climate.
• Chase the failures, not the score — the 20 failing cases are the roadmap: read them monthly with engineering, sort into fix-with-data / fix-with-scope / accept-and-design-forgiveness (a coarse first cut; Module 4 refines this into four cheapest-first layers), and notice that this sorting is a product decision each time.