Lab: Few-shot design
Mine your gauntlet misses for boundary examples, add structured reasoning, and measure the before/after on a fixed test set.
This lab upgrades the triage assistant with Module 3's tools — and introduces the habit that defines the rest of the course: never change a prompt without measuring on a fixed set of cases.
This is the first lab with real hand-work: you'll write ~12 gold outputs and do roughly 36 chat runs (12 cases × three prompt variants), scoring each by eye. That's an hour and a half to two hours, not eighteen minutes — and it splits cleanly across two sittings (Steps 1–2 first, Steps 3–4 later) if you'd rather. The minutes counter above is nominal; trust this estimate.
Marketing track: your 12-case test set is 12 real (redacted) campaign briefs instead of emails, and your 'gold output' is the correct {channel, launch_urgency, campaign_code} for each. The mining move is identical — run zero-shot, look only at the misses, add one example per systematic miss. Only the field names change.
Step 1 — Build a 12-case test set
- 1Collect your gauntlet emails from Module 2, the forwarded-vase case, the injection email, plus enough new realistic ones to reach 12.
- 2For each, write the correct output JSON yourself (the 'gold' answer). Arguing with yourself about the right label for a hard case is the design work — capture the rule you settle on.
- 3Save as
triage-testset-v1.txt: 12 inputs, 12 gold outputs. This file outlives every prompt version you'll ever write.
Step 2 — Baseline, then add examples
- 1Run v0.3 (no examples) on all 12. Score: category exact-match /12, plus your Module 2 contract metrics. This is your baseline.
- 2Look at the misses only. Write ONE example targeting each systematic miss (max 3 examples), real-shaped, with
<why>lines, labels balanced. - 3Run again. Score again. Keep an example only if the score moved — an example that didn't help is pure token rent.
Step 3 — Add reasoning, measure again
- 1Add the capped
reasoningfield (first in the schema). Re-run the 12. - 2Compare: did reasoning fix any remaining miss? Did it break any pass (watch for the model talking itself out of easy calls)?
- 3Read the reasoning on the two hardest cases — is the right rule being cited, or the right answer for a wrong reason? Note it; wrong-reason passes fail later on paraphrases.
Step 4 — Ship v0.4
Keep whichever combination scored best, log the change ('v0.4: +2 boundary examples, +reasoning field — 9/12 → 12/12 on testset-v1'), and notice what just happened: you didn't argue about which prompt felt better. You measured. Module 5 industrializes exactly this loop.
Problem set 3 — repair a poisoned example set
Below is the example block from a support-ticket priority classifier (labels: LOW / MEDIUM / HIGH). It's broken in three ways at once. First predict the three failure modes it will cause; then rebuild it in four examples or fewer, defending each slot you keep.
RULE: A ticket is HIGH only if it reports an outage or data loss.
Example 1: "The site is completely down." -> HIGH
Example 2: "I can't log in at all, nothing loads." -> HIGH
Example 3: "Getting 500 errors on every page." -> HIGH
Example 4: "Checkout is throwing errors sometimes." -> HIGH
Example 5: "How do I change my billing email?" -> HIGH- All five are HIGH — the frequency signal teaches 'when unsure, say HIGH', so MEDIUM and LOW tickets get over-escalated.
- Worst offender is last (Example 5: a routine billing question labeled HIGH) — and the last example pulls hardest, so this one poisons the most.
- Example 5 contradicts the rule outright — a billing question is neither outage nor data loss, yet it's tagged HIGH. The example silently overrides the rule (Module 1 precedence).
- Your rebuild: keep 1–2 genuine HIGH cases, add a clear MEDIUM and a clear LOW, balance the labels, and put the most representative case last. Four slots, each defensible.