Lab: Retrieval tuning

This is the course's pivotal lab: HarborDocs gets a measurement harness, and from here forward no retrieval change ships without numbers. You'll also discover which upgrades your corpus actually needs — possibly fewer than the lessons offered, which is itself the lesson.

Step 1 — The labeled set (the real work)

1. 1Grow your 10 questions to 30: ~15 routine, ~8 phrasing-mismatch cases (user words ≠ document words), ~4 exact-identifier queries (product names, 'HL-####' style), ~3 questions whose answer spans two chunks.
2. 2Label relevant chunk_ids for each (1–3 per question). Log every judgment call in RELEVANCE-POLICY.md — three lines of policy now saves thirty arguments later.
3. 3Hold out 5 questions. Same rule as always: never tune against them.

Step 2 — Harness + baseline

1. 1Write evaluate.py: run all 25 tuning questions through retrieval, print recall@k for k∈{1,3,5,10}, precision@3, MRR, plus the per-question misses.
2. 2Baseline your Module 2 index. Find the recall knee and set your k. Commit the numbers to LAB-NOTES.md.
3. 3Read the misses list. Categorize each failure: phrasing gap, identifier, ranking error, chunk boundary, or 'label was wrong'. This taxonomy chooses your next move.

Step 3 — Upgrades, earned one at a time

1. 1If identifier queries failed: add BM25 (pip install rank-bm25, ~10 lines) + RRF fusion. Re-run. Record the delta — expect identifier queries fixed, prose queries unchanged.
2. 2If ranking errors dominate: add a reranker over the top-25 (the LLM-judge below, or a hosted rerank API). Re-run. Expect MRR and precision@3 to move; recall@10 to stay put.
3. 3If boundary failures appeared: fix the chunker, re-index, re-run — cheaper than any model change and often bigger.
4. 4After each change: one experiment-log line (change, hypothesis, delta, keep/revert). You know this discipline; now it has retrieval metrics in it.

The LLM-judge reranker is the reranker you can build with tools you already have — no new dependency. Score each candidate for how directly it answers the query, then re-sort by that score. It reuses the same Claude call you'll formalize in Module 4:

import json
import re
import anthropic

JUDGE = """You score how directly a passage ANSWERS a question — not how
related it is. Output only a single integer 0-10, nothing else.
0 = irrelevant; 5 = related but doesn't answer; 10 = directly answers."""


def _score(client, question: str, passage: str) -> int:
    msg = client.messages.create(
        model="claude-sonnet-5",   # small/fast model is fine; id is swappable
        max_tokens=8,
        system=JUDGE,
        messages=[{"role": "user",
                   "content": f"Question: {question}\n\nPassage: {passage}"}],
    )
    text = msg.content[0].text.strip()
    m = re.search(r"\d+", text)
    return int(m.group()) if m else 0


def llm_rerank(question: str, candidates: list[dict], keep: int = 5) -> list[dict]:
    """candidates: [{'chunk_id':..., 'text':...}, ...] from stage-1 retrieval.
    Returns the `keep` highest-scoring, re-sorted by judged relevance."""
    client = anthropic.Anthropic()
    scored = []
    for c in candidates:
        c = {**c, "judge_score": _score(client, question, c["text"])}
        scored.append(c)
    scored.sort(key=lambda c: c["judge_score"], reverse=True)
    return scored[:keep]

Step 4 — The holdout verdict

Run your final configuration on the 5 holdouts. Within a few points of the tuning set → trust the pipeline. A big gap → you've been overfitting your 25 questions; diversify the set and retune. Freeze the winning config as retrieval-v1 — Module 4 wires it into a real application.