Back to course overview
Module 6Capstone 13 min

Capstone: Evaluate

The evaluation that survives review: the full suite on the shipping artifact, error analysis with named causes, the sealed held-out slice, and the side-effect audit.

Adaptation projects are judged by their evaluation, because everything else can look good while the model is worse. The capstone's eval phase, structured as review will read it:

  1. 1Evaluate what ships, not what trained: the merged/quantized/serving-form artifact, full suite — task metrics by class, format adherence, escape behavior, the adversarial slice, and the general-capability slice (the forgetting audit; state what you checked and what you accept). Three columns minimum: ceiling / adapted / reference (teacher or requirement), with the pre-registered target from your brief marked met or missed.
  2. 2Error analysis with named causes: every remaining failure clustered — data-fixable / capacity / eval-bug — with counts, two worked examples per cluster, and what the next dataset version would change. This section is where reviewers decide whether you understand your model or merely trained it.
  3. 3Break the seal last: the held-out slice, run once, reported unedited beside the main results, with a one-paragraph reconciliation if they diverge. The seal discipline is the capstone's honesty proof — and the habit that will someday save you from shipping a contaminated 'improvement' with your name on it.
  4. 4The side-effect audit: what got worse? (Something always does — a class traded off, verbosity shifted, an escape rate moved.) Found and named beats discovered by a reviewer; a submission reporting zero regressions triggers deeper review, not higher marks — the same rule every audit-shaped course in this academy runs, because it's how real review works.
Claims match evidence, exactly

The report's every quantitative claim must trace to a table in the submission, computed on the artifact named. 'Matches teacher quality' when the table shows a 2.8-point gap is a failed review even if 2.8 points was within your stated tolerance — the passing sentence is 'within the 3-point tolerance agreed in the brief (gap: 2.8)' (your brief may set a different tolerance; this example used 3 points). Precision about your own results is the professional habit this course exists to certify; models are probabilistic, claims about them shouldn't be.