Back to course overview
Module 8Capstone project 25 min

Implementation

Build week: port the HarborDocs skeleton to your corpus, work through the real-document pain, and keep the eval harness running from day one.

Build week. You're not starting from zero — you're porting a working pipeline to a harder corpus. Budget four to six hours across the week, and expect the distribution to surprise you: mostly ingestion, barely any prompt work. That's not the course failing; that's RAG being what it is.

The build order (eval harness first, not last)

  1. 1Golden set v0 on day one: your 15 real questions, labeled for relevance the moment the first index builds. Every subsequent decision gets measured, not vibed — the single biggest difference between a one-week capstone that works and one that demos.
  2. 2Ingestion for your formats: extraction per type, cleaning, your chunking plan, idempotency with the ingestion log. Quarantine failures visibly. This is the half of the week that earns the certificate.
  3. 3Index with manifest + gate, baseline retrieval, run the metrics. Fix the worst archetype before adding anything (usually: one document type chunks badly).
  4. 4Upgrades only as metrics demand: hybrid if identifier queries fail, reranker if ranking errors dominate — each with its before/after log line.
  5. 5The trust layer, non-negotiable: verification gate, calibrated FLOOR, confidence routing. Grow the golden set to 30+ with unanswerables and exception cases as you learn where your corpus bites.
  6. 6Production posture: streaming, length cap, cost-per-query computed, the dashboard script pointed at your traces.

When the week fights back

  • Extraction is worse than expected → shrink the corpus to formats you can extract well and say so in the limits section. A great pipeline over 80 documents beats a broken one over 400.
  • Retrieval metrics are mediocre and nothing helps → interrogate the labeled set first (labels wrong? questions unanswerable?), then chunk boundaries, then — last — models. The debugging order is the course in one line.
  • Answers are grounded but useless → your users' questions need synthesis your chunks are too small for; try small-to-big retrieval before rechunking everything.
Prompt to try

Here are my capstone's current scorecard, my top-5 failing eval cases with traces, and my remaining time budget (2 hours): [paste]. Rank the possible improvements by expected metric impact per hour, and tell me which failing cases I should explicitly ACCEPT and document as known limits instead of fixing.

Knowing what not to fix is a grading criterion. The rubric rewards a documented limit over a half-fixed feature, every time.

Freeze, again

End of week: freeze index version, prompt version, config, golden set, scorecard. The demo runs on the frozen system. Live-tweaking during demo week is how working capstones die on stage.