Lab: CI pipeline

The automation capstone: your eval suite becomes a merge gate, and your release becomes a canary with an auto-rollback. After this lab, a regression cannot silently reach production — the system stops it.

Step 1 — The CI gate

Both scripts speak one frozen report.json schema — the same scorecard your harness already returns, dumped to disk — so run_evals.py writes it and check_gate.py can rely on it: {n, pass_rate, per_check: {name: rate}, records: [{id, output, passed, checks: [{name, passed, detail}]}]}. Freeze that shape once and every consumer (gate, dashboard, CI artifact) reads the same keys.

import json, argparse
from evalharness import run_evals
from feature import run_feature
from checks import CHECKS          # your assertion + judge checks

if __name__ == "__main__":
    ap = argparse.ArgumentParser()
    ap.add_argument("--golden", required=True)
    ap.add_argument("--out", default="report.json")
    args = ap.parse_args()
    rows = [json.loads(l) for l in open(args.golden)]
    report = run_evals(run_feature, rows, CHECKS)   # {n, pass_rate, per_check, records}
    json.dump(report, open(args.out, "w"), indent=2)
    print(f"n={report['n']} pass_rate={report['pass_rate']:.2%} -> {args.out}")

import json, sys

CRITICAL = {"no_pii_leak", "valid_json", "no_injection_obedience"}  # hard-fail checks
PRODUCTION_BASELINE = 0.90    # last-known-good overall pass rate
MARGIN = 0.03                 # allowed slack for non-determinism

def main(path):
    report = json.load(open(path))
    per_check = report["per_check"]
    failures = []
    for name in CRITICAL:                              # any critical < 100% fails
        if per_check.get(name, 0.0) < 1.0:
            failures.append(f"CRITICAL {name} at {per_check.get(name, 0.0):.0%}")
    if report["pass_rate"] < PRODUCTION_BASELINE - MARGIN:
        failures.append(f"pass_rate {report['pass_rate']:.2%} "
                        f"< {PRODUCTION_BASELINE - MARGIN:.2%}")
    print(f"n={report['n']} pass_rate={report['pass_rate']:.2%}")
    for name, rate in sorted(per_check.items()):
        print(f"  {name:24s} {rate:.0%}")
    if failures:
        print("GATE FAILED:", "; ".join(failures))
        sys.exit(1)
    print("GATE PASSED")

if __name__ == "__main__":
    main(sys.argv[1])

1. 1Implement run_evals.py and check_gate.py as above: the first writes the frozen report.json, the second reads report["per_check"][name] for the critical checks and report["pass_rate"] for the overall bar, and calls sys.exit(1) on any failure.
2. 2Wire the workflow (the lesson's evals.yml shape, or a local make eval-gate if you're not on CI) so it runs run_evals.py then check_gate.py on change and uploads the scorecard — the same two script names the YAML already references.
3. 3Tier it: a fast smoke-eval (critical checks + 10 cases) for speed, the full suite as the merge gate.

Step 2 — Prove the gate bites

1. 1Open a change that introduces a regression (weaken a rule so a category of cases fails). Run the gate. It must go red and block — with a report naming which checks failed.
2. 2Open a change that's a genuine improvement. Gate goes green. Confirm the gate distinguishes the two — a gate that passes everything is theater.
3. 3Add one adversarial case as a hard-fail check; prove that a single injection-obedience failure fails the whole build regardless of the average.

Step 3 — The canary script

1. 1Write canary.py: route a configurable % of a replayed traffic stream to a new version, run the sampled judge on both slices, and print the comparison (quality, cost, latency, rejection rate) canary vs. control.
2. 2Add the auto-rollback: if the canary's quality proxy drops below control by more than a threshold, the script reverts the pointer to last-known-good and prints a 'ROLLED BACK' alert.
3. 3Run it two ways: promoting a good version through 5% → 50% → 100%, and catching a bad version at 5% and auto-reverting.

Step 4 — The release runbook

Write the one-page release procedure: open PR → CI eval gate must pass → merge → canary at 5% → check metrics → expand or auto-rollback → promote → old version stays as last-known-good. This runbook plus your gate and canary scripts is a real deployment pipeline for AI — the thing most teams shipping AI features conspicuously lack.