RAG eval metrics

You've been evaluating retrieval since Module 3. Now the whole pipeline needs a scorecard, and the cardinal rule is: evaluate the layers separately, then jointly — because a bad end-to-end answer with great retrieval means a generation problem, and a bad answer with bad retrieval means the generation layer never had a chance. One combined score hides which.

Layer 1 — Retrieval (owned)

Recall@k, precision@k, MRR against the labeled set — unchanged, still first, still the leading indicator. When end-to-end quality drops, these numbers tell you in thirty seconds whether to look above or below the retrieval line.

One reading subtlety when a question has more relevant chunks than k: recall@k is capped by construction. recall_at_k divides hits by the count of relevant chunks, so a question with 3 relevant chunks evaluated at k=1 can score at most 0.33 no matter how good retrieval is — you physically can't fetch three chunks in one slot. A low per-question recall@1 there is expected, not a bug; compare it against that ceiling (relevant-count-over-k), and lean on recall@5 or recall@10 for those multi-chunk questions.

Layer 2 — Generation (new)

• Faithfulness / groundedness — is every claim entailed by the retrieved sources? Measured by your citation-verification pass rate (mechanical) plus the groundedness judge (semantic). The RAG-defining metric: it separates 'answered from the docs' from 'answered plausibly'.
• Answer relevance — does it actually answer the question asked? Faithful-but-beside-the-point is a real failure class ('what's the restocking fee?' answered with the full returns process, fee unstated). Judge-scored with a rubric.
• Abstention quality — two error rates that trade against each other: answered-when-it-shouldn't (false confidence) and abstained-when-it-could (false modesty). You need unanswerable questions in the eval set to measure either; most teams forget and ship a system that never says no.
• Citation quality — verification pass rate + citation coverage (fact-bearing sentences carrying a citation).

The joint read

                     retrieval GOOD          retrieval BAD
answer GOOD     |  system working        |  model knew it anyway —
                |                        |  DANGER: works until it doesn't
answer BAD      |  generation problem:   |  retrieval problem:
                |  prompt/grounding/     |  chunking, embedding,
                |  synthesis             |  ranking — fix upstream

The top-right cell is the one that bites teams: the model answers correctly from general knowledge while retrieval fails — your eval looks fine until a question arrives that only the corpus can answer. It's detectable only because you measure layers separately: good answer + bad retrieval = flashing red, even though users are happy today.