Implementation
Build week: port the HarborDocs skeleton to your corpus, work through the real-document pain, and keep the eval harness running from day one.
Build week. You're not starting from zero — you're porting a working pipeline to a harder corpus. Budget four to six hours across the week, and expect the distribution to surprise you: mostly ingestion, barely any prompt work. That's not the course failing; that's RAG being what it is.
The build order (eval harness first, not last)
- 1Golden set v0 on day one: your 15 real questions, labeled for relevance the moment the first index builds. Every subsequent decision gets measured, not vibed — the single biggest difference between a one-week capstone that works and one that demos.
- 2Ingestion for your formats: extraction per type, cleaning, your chunking plan, idempotency with the ingestion log. Quarantine failures visibly. This is the half of the week that earns the certificate.
- 3Index with manifest + gate, baseline retrieval, run the metrics. Fix the worst archetype before adding anything (usually: one document type chunks badly).
- 4Upgrades only as metrics demand: hybrid if identifier queries fail, reranker if ranking errors dominate — each with its before/after log line.
- 5The trust layer, non-negotiable: verification gate, calibrated FLOOR, confidence routing. Grow the golden set to 30+ with unanswerables and exception cases as you learn where your corpus bites.
- 6Production posture: streaming, length cap, cost-per-query computed, the dashboard script pointed at your traces.
When the week fights back
- Extraction is worse than expected → shrink the corpus to formats you can extract well and say so in the limits section. A great pipeline over 80 documents beats a broken one over 400.
- Retrieval metrics are mediocre and nothing helps → interrogate the labeled set first (labels wrong? questions unanswerable?), then chunk boundaries, then — last — models. The debugging order is the course in one line.
- Answers are grounded but useless → your users' questions need synthesis your chunks are too small for; try small-to-big retrieval before rechunking everything.
Here are my capstone's current scorecard, my top-5 failing eval cases with traces, and my remaining time budget (2 hours): [paste]. Rank the possible improvements by expected metric impact per hour, and tell me which failing cases I should explicitly ACCEPT and document as known limits instead of fixing.
Knowing what not to fix is a grading criterion. The rubric rewards a documented limit over a half-fixed feature, every time.
End of week: freeze index version, prompt version, config, golden set, scorecard. The demo runs on the frozen system. Live-tweaking during demo week is how working capstones die on stage.