Beyond imitation: why preferences

SFT has a structural limit: it teaches the model to imitate your examples, weighted equally — it cannot express 'this output is fine, that one is better, and here's the direction of better.' Some quality dimensions live exactly in that gap: concision without terseness, confidence calibrated to evidence, tone that adapts without drifting, the judgment calls where you can't write the rule but you know the better answer on sight. Preference tuning trains on that recognition directly.

• The data unit is the pair: (prompt, chosen response, rejected response). Not 'good vs. garbage' — the informative pairs are good vs. slightly better, because the model learns from the direction of the difference, and 'obviously bad' rejections teach almost nothing it didn't know.
• RLHF, honestly compressed: train a reward model on human preferences, then use reinforcement learning to push the policy model up the reward — the technique that made modern assistants possible, and a genuinely heavy apparatus (reward-model training, RL instability, reward hacking) that frontier labs run and you almost certainly shouldn't. You should know its shape because it explains the models you build on — and because its failure mode, reward hacking (the model finds what the reward model over-credits and exploits it), reappears in miniature in everything downstream.
• DPO — the reason this module has a lab: Direct Preference Optimization reformulates the objective so the model learns from preference pairs directly — no reward model and no RL loop (though it keeps a frozen reference copy of the SFT model to anchor each update), runs on the same tooling as your SFT with a second data file. DPO and its descendants made preference tuning a normal engineering step instead of a research program. The standard pipeline: SFT first (installs the skill), DPO second (calibrates the judgment) — never DPO alone on a base that can't do the task yet.

When the preference pass earns its place (and when it doesn't)

The test: can you write the failure as a rule? If yes ('never exceed 3 sentences', 'always include the order number') — that's SFT data or even prompt territory; fix it there, cheaper. If no — 'both outputs are valid, B reads better to every reviewer, and none of them can say why in a rule' — that's a preference-shaped gap. For the triage project, the preference-shaped gap after Module 3 is typically escape calibration: the SFT'd student either overuses [unclear] (safe but lazy) or underuses it (confident but wrong); the boundary resists rules because it is a judgment. That's Module 4's lab target — narrow, real, and measurable.