Capstone: Evaluate
The evaluation that survives review: the full suite on the shipping artifact, error analysis with named causes, the sealed held-out slice, and the side-effect audit.
Adaptation projects are judged by their evaluation, because everything else can look good while the model is worse. The capstone's eval phase, structured as review will read it:
- 1Evaluate what ships, not what trained: the merged/quantized/serving-form artifact, full suite — task metrics by class, format adherence, escape behavior, the adversarial slice, and the general-capability slice (the forgetting audit; state what you checked and what you accept). Three columns minimum: ceiling / adapted / reference (teacher or requirement), with the pre-registered target from your brief marked met or missed.
- 2Error analysis with named causes: every remaining failure clustered — data-fixable / capacity / eval-bug — with counts, two worked examples per cluster, and what the next dataset version would change. This section is where reviewers decide whether you understand your model or merely trained it.
- 3Break the seal last: the held-out slice, run once, reported unedited beside the main results, with a one-paragraph reconciliation if they diverge. The seal discipline is the capstone's honesty proof — and the habit that will someday save you from shipping a contaminated 'improvement' with your name on it.
- 4The side-effect audit: what got worse? (Something always does — a class traded off, verbosity shifted, an escape rate moved.) Found and named beats discovered by a reviewer; a submission reporting zero regressions triggers deeper review, not higher marks — the same rule every audit-shaped course in this academy runs, because it's how real review works.
The report's every quantitative claim must trace to a table in the submission, computed on the artifact named. 'Matches teacher quality' when the table shows a 2.8-point gap is a failed review even if 2.8 points was within your stated tolerance — the passing sentence is 'within the 3-point tolerance agreed in the brief (gap: 2.8)' (your brief may set a different tolerance; this example used 3 points). Precision about your own results is the professional habit this course exists to certify; models are probabilistic, claims about them shouldn't be.