Build
Assemble the stack in the right order — evals first, then tracing, versioning, cost, CI — reusing every artifact from the course.
Build week: assemble the full stack around your feature. You've built every component in the labs; now integrate them into one operating loop. Budget a focused day (or two evenings) — you're integrating existing lab artifacts, not building each from scratch — and follow the order, because it's the order that makes each piece testable as you add it.
The build order (each layer enables the next)
- 1Evals first (M2). A ~50-case versioned golden set, the reusable harness, assertion + calibrated-judge checks, a holdout. Nothing else can be trusted until this exists — it's how you'll verify every layer.
- 2Tracing (M3). Wrap the feature so every call emits a complete trace; build the metrics dashboard and a sampled online-eval. Now you can see the feature.
- 3Versioning (M4). Move the config into versioned objects with a production pointer and last-known-good; stamp traces with the version. Now you can change it safely.
- 4Cost levers (M5). Measure cost per request; apply prompt-thrift, caching, and routing — each gated by the eval harness so quality holds. Now it's affordable.
- 5CI gate + canary (M6). Wire the eval gate to block regressions and script the canary rollout with auto-rollback. Now it can't regress silently.
- 6Close the loop. Wire the flywheel: a failure surfaced in traces → added to the golden set → guarded by the gate. Demonstrate one full turn.
The traps of build week
- Polishing the feature instead of building the loop. Your capstone is graded on the operational stack, not on making the underlying AI a point better. A mediocre feature with an excellent loop beats a great feature you can't operate.
- A golden set of only easy cases. The loop's whole value is catching regressions and drift; a set with no hard, adversarial, or failure-archive cases can't catch anything. Put in the cases you fear.
- Skipping calibration. An uncalibrated judge makes every downstream number fiction. Do the 15-label comparison; record the agreement.
Here is my current LLMOps stack — harness, sample traces, version config, cost table, CI gate: [paste/describe]. Act as a reviewer doing a readiness check. For each loop station (evaluate, deploy, observe, improve), tell me whether it's actually functioning or just present, and give me the single highest-value gap to close with my remaining time.
'Present but not functioning' is the most common capstone flaw — a dashboard that renders but tracks the wrong metric, a gate that runs but never fails. This prompt hunts exactly that.
End of week, freeze everything and run one full loop turn on camera/notes: introduce a regression → the gate catches it → fix it → a new eval case guards it → canary promotes the fix. That single demonstrated turn is worth more than any static artifact — it proves the loop is alive.