Regression testing

A/B testing answers 'is the new version better?' Regression testing answers the quieter, more important question: 'did anything that used to work stop working?' Prompt systems are unusually prone to silent regressions, because the parts interact — a new example subtly outweighs an old rule, three added lines shift attention off the grounding block. Distance in the file is no defense.

The regime (you already own all the pieces)

• Prompts live in version control — files with version numbers and change logs, like your triage-system-v0.x series. Not in a chat window, not in someone's Notion.
• The eval suite is the test suite. No prompt change ships without a full golden-set run. Green → ship with the log line. Red → the diff tells you exactly which change to interrogate.
• Assertions are your unit tests; judge dimensions are your quality bar; held-out cases are your honesty check.
• A one-page runbook per production prompt: owner, version, eval score, model + settings it's pinned to, and the escalation path when it misbehaves. (This becomes your capstone documentation.)

The regression nobody's edit caused: model drift

Your prompt can regress while the file is untouched — the model behind it got updated. Providers improve models continuously; behavior shifts, usually for the better, occasionally not for your specific prompt. The defenses are procedural: pin model versions in production where the API allows; re-run the eval suite on a schedule — set yourself a weekly calendar reminder to re-run your suite (engineering teams automate this with a scheduled job) — not just on edits; and when a new model version arrives, eval it like any other A/B — sometimes the 'upgrade' loses on your task, and now you have the receipts to wait.

Close the loop from production

1. 1Log inputs + outputs (redacted per your Module 4 Foundations rules) with the prompt version that produced them.
2. 2Route human corrections — the edited reply, the overridden category — back to review weekly. Each correction is a candidate golden-set case.
3. 3Track the assertion metrics on live traffic (parse rate, null discipline, quote verification). A dipping live metric is your earliest drift alarm — earlier than complaints.