Evaluation

Your system is frozen; now certify it. The eval report (deliverable 2) is the document that separates 'I built a RAG app' from 'I built a RAG app and can prove what it does' — in this course and in every job interview where this project comes up.

The report, section by section

1. 1System summary (½ page): corpus, architecture choices with their measured justifications, current versions from the manifest.
2. 2Golden set description: size, archetype mix, how many cases came from real users vs. synthetic, holdout policy, relevance-policy highlights. Reviewers read this first — a weak set invalidates everything after.
3. 3Retrieval layer: recall@k curve with your chosen k marked, MRR, per-archetype breakdown. Name the weakest archetype and whether you fixed or accepted it.
4. 4Generation layer: verification pass rate, judge-scored groundedness and relevance (with the calibration evidence: your 10 hand grades vs. the judge's), abstention correctness on unanswerables.
5. 5The diagnostic grid with counts, and one sentence on every top-right case (model-knew-it-anyway) — you know why those matter.
6. 6Confidence calibration: correctness rate per HIGH/MEDIUM/LOW bucket. This table is the single most impressive artifact to people who evaluate AI systems for a living.
7. 7Operations: p50/p95 latency, cost per query, cache hit rate.
8. 8Known limits, plainly: the archetypes it's weak on, the documents it can't read, the questions it wrongly refuses. Verbatim honesty — the strongest trust signal you own.

The holdout ceremony

Last measurement before writing: run the holdouts you never tuned against. Report both numbers — tuning set and holdout — side by side. Close together: your scores generalize, say so. A gap: report it and interpret it honestly (overfit tuning, or holdouts that drifted from the tuning distribution?). A reviewer who sees an honest gap analysis trusts every other number in the report; one who sees only suspiciously perfect scores trusts none.