Reranking

Everything so far scores query and chunk separately — each embedded alone, compared by geometry. That's what makes it fast, and what caps its judgment: the vector for 'Customers may not cancel after shipment' sits close to a cancellation question whether or not it answers it. A reranker removes that cap by reading the query and the chunk together, attending across both texts, and scoring actual relevance — the relatedness-vs-relevance gap from Module 1, finally closed by a model built for it.

The two-stage architecture

1. 1Stage 1 — retrieve wide and cheap: hybrid search returns the top 25–50 candidates. Optimize this stage for recall: the right chunk merely has to be somewhere in the net.
2. 2Stage 2 — rerank narrow and smart: a cross-encoder (hosted rerank APIs from Voyage/Cohere, or open-source models) scores each candidate against the query; keep the top 3–5. Optimize this stage for precision at the top.
3. 3The economics work because the expensive model only ever sees 50 pairs, not 50,000 chunks — you get near-exhaustive judgment at candidate-list prices.

What reranking is worth — and when to skip it

• Expect the biggest wins on MRR and precision@3 — the reranker's whole job is putting the actually-relevant chunk first. Recall@50 doesn't change; it can't find what stage 1 missed.
• It fixes near-duplicate confusion — your corpus's three overlapping returns articles rank by which one answers this phrasing, not which is generically closest.
• Skip it when stage-1 precision is already high (small clean corpus — HarborDocs without the reranker may already be at ceiling), or when its added latency breaks your budget (Module 7). Add it when the labeled set shows ranking errors, not by default.
• LLM-as-reranker: a general model with a rubric prompt — 'score 0–10 how directly this passage answers the question' — works as a slower, costlier, more steerable substitute. You built exactly this judge pattern in Prompt Engineering Module 5; here it moonlights inside the pipeline.