Dataset design

In fine-tuning, the dataset plays the role the prompt played before: it is the specification of behavior, executable and exact. Every property of your tuned model — the good, the weird, the embarrassing — traces to the data. Design principles that separate datasets from data piles:

• Coverage is a curriculum, not a census. The dataset needs every input family the model will face in proportion to how much learning each requires — not in proportion to natural frequency. Natural traffic is 70% easy cases the student already handles; a dataset mirroring it spends 70% of training reinforcing the known. Overweight the hard: ambiguous categories, minority classes, adversarial inputs, the [unclear] escape cases. (Your eval set stays natural-ish; your training set is a gym.)
• Output consistency is non-negotiable. Every example's output must follow the exact schema, the same conventions, the same escape-hatch style — because the model learns the distribution of your outputs, including your inconsistencies. One conflicting labeling convention in 5% of examples measurably degrades format adherence. This is why Module 2's lab audits before it scales.
• Size: smaller than the folklore says. For narrow-task LoRA on a capable base: hundreds of excellent examples beat thousands of mediocre ones; 500-2,000 is the working range for tasks like triage; past a few thousand, returns flatten fast while label-noise damage compounds. The binding constraint is almost never volume — it's verified quality, which is a human-hours budget. Spend it on auditing, not collecting.
• Splits with discipline: train / validation (steers training-time decisions) / test (the held-out 30 from Module 1, plus more — touched once, at the end). Leakage between them — near-duplicate emails landing on both sides — inflates every number you'll report; deduplication across splits is part of the split, not a nicety.

The hygiene trio (each one a project-killer if skipped)

• Contamination — your eval cases (or near-duplicates) inside the training set = a model that memorized the test. Run similarity checks between train and eval before any training run, and re-run them every time either side grows. A suspiciously great eval score should trigger this check first.
• Rights & terms — training data must be data you may train on: customer emails need the right basis under your privacy commitments; teacher outputs need the teacher's ToS to permit distillation (checked in Module 1); anything scraped needs the conversation with counsel you'd rather have now. The model is the data in compressed form; you can't un-train what you shouldn't have included.
• PII — training data gets memorized, occasionally verbatim, and a tuned model can be coaxed to emit fragments of it. Scrub names, addresses, account identifiers to realistic placeholders before training — the redaction-before-storage discipline from the operational courses, applied one stage earlier, where it's irreversible if skipped.