Recall and precision

You've now 'improved' retrieval once and verified it with ten hand-checked questions. This module gives that instinct real instruments. The prompt-engineering course taught you a prompt without an eval is an opinion — the RAG corollary: retrieval without metrics is a demo.

The labeled set: judgments before numbers

Every retrieval metric needs a relevance-labeled set: questions, each mapped to the chunk(s) that genuinely answer it. Yours will hold 25–40 questions (your lab's 10 are the seed). Labeling forces the same healthy arguments gold answers did in Prompt Engineering — 'is the general returns chunk relevant to a holiday returns question, or only the specific one?' Decide, write it down; those decisions are your relevance policy.

The three numbers

• Recall@k — of the relevant chunks, what fraction appeared in the top k? The RAG metric, because generation cannot cite what retrieval never fetched. Recall@5 = 0.80 means that, on average, a fifth of the relevant material is missing from the top 5 (for a question with a single relevant chunk, that averages out to one such question in five being unanswerable before the model even runs).
• Precision@k — of the k retrieved, what fraction are relevant? Matters because irrelevant chunks aren't free: they burn context tokens and actively distract the model (you measured attention dilution in PE Module 1).
• MRR (mean reciprocal rank) — how high does the first relevant chunk sit? (1st → 1.0, 3rd → 0.33, averaged.) Your tie-breaker when recall is equal: rank 1 beats rank 4 for models and for citation UX.

def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    hits = sum(1 for c in retrieved[:k] if c in relevant)
    return hits / len(relevant)

def mrr(retrieved: list[str], relevant: set[str]) -> float:
    for rank, c in enumerate(retrieved, 1):
        if c in relevant:
            return 1 / rank
    return 0.0
# Average each over the labeled set. Twenty lines total, career-long habit.

Using them: the k trade-off

Raising k always helps recall and always hurts precision — the whole game is choosing the operating point. Practical method: plot recall@k for k = 1…10 on your set. It climbs steeply then plateaus; set k just past the knee, then spend your remaining effort making the ranking better (next two lessons) instead of the window bigger. And diagnose before treating: recall failures at k=10 are a chunking or embedding problem; good recall@10 but bad recall@3 is a ranking problem. Different diseases, different medicine.