Test datasets
Building the RAG golden set: question archetypes, synthetic generation from your own corpus, gold answers with source references, and keeping it honest.
A RAG golden set is richer than a prompt golden set: each case carries a question, the gold answer, the gold source chunks, and an answerability flag. That structure is what lets you score both layers. Building ~50 good cases sounds like a slog; your own pipeline makes it half-automatic.
The archetype checklist (coverage beats volume)
- Direct lookups (~15): answer sits in one chunk. Your regression canaries.
- Phrasing-gap cases (~8): user vocabulary ≠ document vocabulary — the reason embeddings exist; keep testing them.
- Multi-chunk synthesis (~6): answer requires two sources. Tests assembly and over-synthesis guards at once.
- Exception-clause cases (~5): the rule and its exception — the boundary-wound detector.
- Unanswerables (~8): near-misses, not absurdities. 'Do you price match?' (plausible, undocumented) tests abstention; 'what's the moon made of?' tests nothing.
- Identifier queries (~4) and conversational follow-ups (~4): your hybrid search and query rewriting, respectively, each need standing coverage.
Synthetic generation — with the trap named
Here are three chunks from my help-center corpus: [paste]. For each, generate: (1) two questions a real customer would ask that this chunk answers — in customer language, not document language; (2) one question a customer would plausibly ask that this chunk does NOT answer but seems adjacent to. Mark which is which.
Synthetic questions bootstrap the set fast — but generated-from-the-chunk questions are systematically easier than real ones (they inherit the document's vocabulary, which is exactly the hard part in real traffic). The customer-language instruction fights this; production questions replace synthetics over time. Never let a synthetic-only set make you complacent.
Gold answers and upkeep
- Write gold answers from the source chunks, citing them — you're producing the reference the judge compares against, so it must itself be exemplary: grounded, complete, appropriately hedged.
- Gold sources are chunk IDs, which means corpus changes can invalidate cases. Tie the golden set to the corpus snapshot in the index manifest; when documents change, a small script flags cases whose gold chunks were touched. Stale gold labels are how eval suites quietly rot.
- Same governance as always: holdouts never tuned against; production misses feed the set weekly; every case's judgment call logged in the relevance policy.
If every case passes, your set is a mirror, not a test. The archetypes force fear in: unanswerables you'd rather not measure, synthesis questions that flirt with over-reach, follow-ups that stress the rewriter. Comfort is the smell of an eval set gone decorative.