Eval frameworks

You've been evolving one prompt across four modules, scoring changes on a fixed test set. That habit has a name — evaluation — and this module turns it from a lab ritual into an engineering system. The core claim: a prompt without an eval is an opinion.

The golden set

Your 17 cases with gold answers are a golden set — the single most valuable artifact in any prompt project. Growing and grooming it is the real work:

• Size: 20–50 cases catches most regressions for a single-task prompt; you don't need thousands.
• Composition: ~⅓ routine cases (regression canaries), ~⅓ boundary cases (the decisions you argued about), ~⅓ failure archive (every real-world miss gets immortalized here — the eval set is where production bugs go to die).
• Gold answers are decisions. Writing the correct output for a hard case forces the policy question a vague prompt was hiding. Disagreement about gold answers is requirements discovery — welcome it.

Two kinds of checks

• Assertions — binary, mechanical, cheap. JSON parses; enums legal; order_id null when absent; quotes verify verbatim; reply ≤120 words; no competitor names. Runs in code, instant, objective. Cover everything assertable first.
• Graded dimensions — qualitative, rubric-scored. Is the reply's tone right? Is the summary faithful? Score 1–5 against a written rubric. Humans can grade; models can too — which is exactly what the next section builds.

LLM-as-judge, done honestly

You are grading a customer-service reply draft. Score each dimension 1-5:

TONE (Harbor Lane voice: warm, direct, no corporate filler)
  5 = indistinguishable from our best agent   3 = generic-professional
  1 = cold, stiff, or saccharine
FAITHFULNESS (only facts from the email + cited policy)
  5 = every statement traceable   3 = one unsupported nicety
  1 = invents policy or order facts
ACTIONABILITY (customer knows exactly what happens next)
  5 = names the exact next step AND a timeframe   3 = vague next step, no timeframe
  1 = no next step stated

Return JSON: {"tone": n, "faithfulness": n, "actionability": n,
"worst_sentence": string, "one_fix": string}

• A judge needs a rubric with anchored levels — 'score the tone' without anchors just measures the judge's mood.
• Calibrate once: grade 10 outputs yourself, run the judge on the same 10, compare. Agreement within a point on ≥8 → trust it for screening. Systematic drift → fix the rubric anchors.
• Judges screen, humans decide. Use the judge to rank and flag at scale; spot-check its extremes; never let it be the last word on a shipping decision.