Regression testing

Prompt regression testing had one moving part — the prompt. RAG has five: the corpus, the chunker, the embedding model, the retrieval config, and the generation prompt. Any of them can change independently, and each can silently regress the others' work. The defense is the same discipline, multiplied: version everything, gate every change through the eval suite.

The gates

• Index rebuilds (corpus or chunker changed): run retrieval metrics on the new index before the atomic swap — the manifest's eval block is this gate's receipt. A rebuild that drops recall@5 by six points never goes live; you read the diff instead (usually: a big document was restructured and now chunks differently).
• Prompt changes: the full two-layer suite, exactly your Prompt Engineering protocol — paired comparison, read the flips, check the holdouts.
• Config changes (k, thresholds, reranker on/off): retrieval metrics + abstention rates. Threshold changes especially — a FLOOR nudged for one bad case can silently mute a fifth of your traffic.
• Model version changes — all three models: generation, embedding, reranker. Embedding is the treacherous one (Module 4's trap: full re-embed required, both sides, gated like any rebuild). Pin versions where the API allows; eval before adopting any 'upgrade'.

Drift you didn't cause

Two kinds arrive uninvited. Corpus drift: documents accumulate, topics shift, and retrieval quality erodes for reasons no diff explains — the weekly scheduled eval run catches the trend while it's still gentle. Query drift: users start asking about things your corpus (and your golden set) don't cover — a new product line launches and your eval says everything's fine because nobody told the eval. The fix is the production feedback loop: sample real queries weekly, check them against the archetype mix, and let reality refresh the set. Your golden set should smell like last month's traffic, not last year's launch.