Offline evals

Across three courses you built golden sets and scored them, each time slightly differently. LLMOps demands one thing more: a reusable eval harness — infrastructure, not a one-off script — that any version of your feature plugs into and any teammate can run. This lesson makes the eval a first-class system.

The two check types (unified at last)

• Assertions — deterministic, code-checked, free: output parses, enum is legal, required field present, no forbidden phrase, latency under budget. These are your regression bedrock; they never flake. Cover everything assertable first.
• Graded checks — qualitative, rubric-scored: is the tone right, the answer faithful, the classification correct on a judgment call. Scored by a human or (Module 2's next lesson) an LLM judge. Necessary for the things assertions can't reach.
• The design rule: push checks toward assertions wherever possible. 'The reply is polite' is a fuzzy graded check; 'the reply is ≤120 words and contains no refund-promise phrase' is two crisp assertions that capture most of what you meant. Every check you can make deterministic is a check that can't flake in CI.

from dataclasses import dataclass

@dataclass
class CheckResult:
    name: str; passed: bool; detail: str = ""

def run_evals(feature, golden_rows, checks, n_runs=1) -> dict:
    """feature: callable input->output. checks: list of (name, fn).
    n_runs: run EACH row this many times and average — gate on the RATE,
    not a lucky single pass. Default 1 (fine for a deterministic feature);
    pass n_runs>=3 for a live, non-deterministic feature.
    Returns a scorecard the whole course reuses."""
    records, all_runs = [], []
    for row in golden_rows:
        runs = [[c(row, feature(row["input"])) for c in checks]  # CheckResults
                for _ in range(n_runs)]                            # per run
        all_runs.append(runs)
        run_passed = [all(r.passed for r in rr) for rr in runs]
        records.append({"id": row["id"], "output": None,
                        "checks": [r.__dict__ for r in runs[-1]],  # last run
                        # bool at n_runs==1, else the per-case pass RATE:
                        "passed": run_passed[0] if n_runs == 1
                                  else sum(run_passed) / n_runs})
    pass_rate = sum(r["passed"] for r in records) / len(records)
    # per-check pass rate, averaged over every (row, run) — the M6 gate reads
    # this to gate each check independently, not just the overall rate.
    per_check = {}
    for name in [c["name"] for c in records[0]["checks"]] if records else []:
        hits = sum(cr.name == name and cr.passed
                   for runs in all_runs for rr in runs for cr in rr)
        per_check[name] = hits / (len(records) * n_runs)
    return {"n": len(records), "pass_rate": pass_rate,
            "per_check": per_check, "records": records}

Notice the shape: the harness knows nothing about your feature — it takes a callable and a list of checks. That decoupling is the whole point. You'll add judge-checks, run it in CI, and point it at any of your prior projects without rewriting it. This is the artifact the rest of the course builds on.

Scoring you can trust

• Run each case N times (3–5) and report pass rate, not pass/fail — the non-determinism you measured in Module 1 means a single run is an anecdote. That's the n_runs knob on run_evals: it defaults to 1 (a single pass, fine for a deterministic feature — the stub scores the same at any n_runs), and you pass n_runs=3+ for a live model so passed becomes a per-case pass rate you can gate on.
• Report per-check, not just overall. 'Pass rate 82%' hides which check is failing; a breakdown ('faithfulness 95%, tone 70%') points at the fix.
• Separate a holdout you never tune against — the overfitting defense you've used since Prompt Engineering.
• Version the eval set itself. golden/v1.jsonl → v2 as it grows; a score is only comparable against the same set.