Eval gates

Everything so far is a discipline you choose to run. This module makes it automatic and enforced: the eval suite becomes a CI gate — a check that runs on every change and blocks the merge if quality regresses. This is the step that turns 'we have evals' into 'we can't ship a regression,' which is the whole point.

The gate

On every pull request that touches the prompt, the code, or the eval set, CI runs your harness against the golden set and fails the check if the score drops below a threshold. A red gate blocks the merge exactly like a failing unit test — because a prompt change that quietly breaks a category of inputs is a bug, and bugs shouldn't merge.

on: [pull_request]
jobs:
  evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: python run_evals.py --golden golden/v3.jsonl --out report.json
        env: { ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} }
      - run: python check_gate.py report.json   # exits nonzero if below bar
      - uses: actions/upload-artifact@v4          # attach the scorecard
        with: { name: eval-report, path: report.json }

Choosing thresholds

• Gate on the metrics that matter, per check. A single overall pass rate is coarse; gate the critical checks hard (zero tolerance on 'no PII leak', 'valid JSON', 'no injection obedience') and the quality checks on a threshold (pass rate ≥ last-known-good − small margin).
• Hard-fail the non-negotiables. Safety and format regressions should block absolutely — one failing adversarial case fails the build, no averaging.
• Set the quality bar relative to main. 'Don't drop below the current production score' catches regressions without demanding perfection; pair it with a separate 'did it improve?' report for the human reviewer.

Non-determinism in CI

A flaky gate that fails randomly gets ignored — the worst outcome. Tame the non-determinism: run each case N times and gate on the rate, not a single pass — that's exactly run_evals(..., n_runs=3) against a live feature (the reference harness defaults to n_runs=1, a single pass that's correct for the deterministic stub; bump it up once real-model variance is in the loop); keep the prompt fixed and effort low for eval runs (current-gen models steer via prompting/effort, not a temperature knob, so determinism comes from a stable prompt rather than a setting); and build a small margin into the threshold so normal variance doesn't trip it while a real regression does. If the gate flakes, fix the flakiness before anyone learns to bypass it — an ignored gate is worse than no gate, because it wears the costume of safety.