Cleaning, formatting & the training file

Between 'we have examples' and 'we can train' sits unglamorous work that decides more outcomes than hyperparameters ever will:

• The training file format — SFT data ships as conversation records (system / user / assistant messages, one JSON object per line). Two rules carry you: the system prompt in training should match the one you'll serve with (train with one, deploy with another, and you've measured a model you're not shipping), and the chat template must match the base model's — every model family wraps conversations in its own special tokens, tooling usually handles it, and 'usually' is why you verify by decoding one formatted example and reading it token-by-token once. Silent template mismatch is the classic mystery-bad-run.
• Cleaning is a pipeline, not a weekend — written as code, rerunnable, because you'll regenerate the dataset at least three times as the project matures: exact and near-duplicate removal (within splits and across them), length outlier handling, schema validation on every output (a training example with malformed JSON output is an instruction to produce malformed JSON), placeholder-PII substitution, and the class-balance report that tells you whether the curriculum you designed is the curriculum you built.
• The audit sample is the step that isn't optional. Automation catches structure; only reading catches wrongness. Sample 50-100 examples stratified by class, read them against the labeling guide, and score labeler agreement (yourself-vs-guide counts if you're solo — re-label 30 blind and measure your own consistency). Under ~90% agreement means the guide is ambiguous — fix the guide, re-generate, re-audit. This loop, run honestly, is the single highest-leverage hour in the whole course.

Generating data with a teacher (the distillation prelude)

When the data source is a frontier model rather than history: the teacher prompt = your labeling guide + schema + escapes, run over real (scrubbed) inputs at scale. Three disciplines make teacher data trustworthy: sample diversity in, quality out (feed it the curriculum's input mix, not just easy traffic), audit the teacher like a labeler (the same 50-100 sample read — frontier models are excellent and not perfect, and their systematic errors become your model's beliefs), and keep the teacher's uncertainty (cases the teacher flags as ambiguous are gold: adjudicate them by hand and overweight them; they're exactly where the student needs the most explicit signal).