Lab: The decision gate
Run three scenarios through the framework, then earn your own project's gate: build the eval set, establish the prompting ceiling, and write the adaptation brief.
Two halves: judgment reps on scenarios (cheap to be wrong about), then the real gate for the course project. By the end you'll have the three artifacts every later module consumes: the eval set, the baseline scores, and the adaptation brief.
The labs make frontier-model API calls end to end: teacher labeling (~1,200 emails) plus the synthetic generation in Module 2, the DPO judging pass in Module 4 (~400 prompts × 3 samples), and the cascade traffic in Module 5. Budget an illustrative ~$10–40 of API spend across the whole course — plus optional GPU rental (order of ~$5–15 per run, per Module 3's callout) if you take the local path instead of hosted. All figures illustrative; prices move, so confirm current rates before you start.
- 1Scenario verdicts (write ~150 words each): (a) a legal-tech startup wants to fine-tune on their contract library 'so the model knows our contracts' — wrong tool, knowledge not behavior: RAG, and say what you'd build instead; (b) a support platform's frontier-model reply-drafter is excellent but costs $40k/month at current volume, task narrow and stable — the distillation case: say what evidence you'd demand first (eval set, volume trend, task stability) and what tolerance you'd negotiate; (c) a brand team can't get any prompt to hold their voice across 10k product descriptions monthly — plausible SFT+DPO case if the prompting ceiling is proven: design the proof (voice rubric, judge, best-prompt baseline).
- 2Build the project eval set (or adapt your RAG-course assets): 150 labeled triage cases — the category/urgency/extraction schema, covering every class, ambiguous cases with adjudicated labels, adversarial and injection-bearing inputs, and 30 held out untouched until final evaluation (the discipline that makes your last number honest). Label quality is destiny from here on: hand-verify every case, because Module 2 will generate 1,000 more from whatever standard you set today.
- 3Establish the prompting ceiling: your genuine best prompt on the target student model (the 8B, raw) — few-shot, schema-enforced, iterated at least three rounds. Score it. Then score the teacher (frontier model) with the same harness — this is the eval harness you built in Building RAG; reuse it (a minimal version is sketched below). Typical shape: teacher ~97%, raw student with best prompt ~78-85%. That spread is your project: the gap is real (gate passed), and the ceiling row becomes the baseline every later module must beat.
- 4Write the adaptation brief (one page, the artifact a reviewer approves): the task and schema, the measured gap, the target ('student within 2 points of teacher on the eval set'), the technique plan (distillation SFT, preference pass if format adherence lags), the cost model (current frontier bill vs. projected student serving + training amortized), and the exit criteria — what result would make you stop and keep prompting instead. Adaptation projects without exit criteria don't exit.
# Minimal scoring-harness stub (reuse the eval harness from Building RAG).
# Loop the eval set, call the model, compare to gold, compute accuracy.
def score(model_fn, eval_set):
"""model_fn(prompt) -> parsed prediction; eval_set is [(prompt, gold), ...].
Returns overall accuracy; extend with by-class and schema-adherence counts."""
correct = 0
for prompt, gold in eval_set:
pred = model_fn(prompt)
if pred == gold:
correct += 1
return correct / len(eval_set)Five more scenarios of escalating subtlety — including the trap case (a team that wants to fine-tune to fix what is actually a retrieval-quality problem; the tell is in their error analysis) and the reverse trap (a team prompt-engineering heroically past the point where a $200 LoRA run was the honest answer). Sorting these fast is the skill that makes you valuable in roadmap meetings.