Lab: Eval framework
Build the 50-case golden set and the two-layer eval runner, wire the index gate, and produce HarborDocs' first full scorecard.
The measurement capstone: HarborDocs gets its full evaluation system, and you get the scorecard you'll defend in Module 8.
Step 1 — The golden set (~50 cases)
- 1Expand your 30 labeled questions to ~50 using the archetype checklist — synthetic generation for speed, customer-language rule enforced, every archetype represented including 8 unanswerables.
- 2Write gold answers with gold chunk IDs for every answerable case. Log judgment calls in the relevance policy.
- 3Mark 7 holdouts across archetypes. Tie the set to the current corpus hash.
Step 2 — The two-layer runner
This runner is the most expensive loop in the course: for every case it runs the full pipeline (generation) plus judge calls (groundedness + relevance). Estimate first — roughly (cases × calls-per-case) generation calls per pass, so ~50 cases at 3 calls each is ~150 calls a run, and you'll run it many times while tuning. Do the first pass on 5 questions at k=10 to confirm the runner and judge work end to end, read the output, then run all ~50. It's easy to leave the full suite looping and quietly burn a small budget on a bug you'd have caught on five.
- 1Extend
evaluate.pyto run the FULL pipeline per case and score both layers: retrieval (recall@k, MRR vs. gold chunks) and generation (citation verification rate, judge-scored groundedness and relevance on a 1–5 rubric, abstention correctness vs. the answerability flag). - 2Calibrate the judge: hand-grade 10 answers, compare, tighten rubric anchors until agreement is within a point on 8+.
- 3Output one scorecard block per run — the diagnostic grid counts included (how many cases landed in each quadrant).
Step 3 — Wire the gates
- 1Make
build_index.pyrun retrieval metrics automatically post-build and write them into the manifest; refuse the swap (exit nonzero) below your floor. - 2Prove the gate works: sabotage the chunker (set MAX_TOKENS=80), rebuild, watch the gate reject it. Restore.
Step 4 — The scorecard, read honestly
- 1Run the full suite. Read the grid: any top-right cases (good answer, bad retrieval)? Those are your hidden time bombs — fix retrieval for them even though the answers look fine.
- 2Check abstention: what fraction of the 8 unanswerables were correctly declined? What did the false-confidence cases claim, and which defense layer should have caught them?
- 3Run the holdouts. Write the one-paragraph honest summary at the top of LAB-NOTES: current scores, weakest archetype, next improvement. This paragraph seeds your capstone eval report.
In the workbook: two scorecards for the same pipeline a month apart — end-to-end scores flat, but the grid shifted (more top-right, fewer top-left) and abstention drifted lenient. Diagnose what's actually degrading, what masked it, and which gate was missing.