Building preference data

Preference data quality has one extra failure surface beyond SFT data: the comparison itself can be biased even when both outputs are real and the choice is sincere. Construction first, defenses second:

• Generating candidate pairs: sample two-plus responses per prompt from your SFT'd student (temperature up for diversity) — pairs from the model you're improving beat pairs from other models, because they target its actual failure surface. For the escape-calibration target: run the student on a prompt set rich in boundary cases; you'll harvest natural pairs where one response escaped and the other guessed.
• Choosing: humans, judges, or the honest hybrid. Human labeling is the gold standard and the budget item; LLM-as-judge scales cheaply and must be calibrated first (you know this discipline — hand-label 50 pairs, measure agreement with the judge, fix the rubric until ≥85%; an uncalibrated judge writes its own biases into your model at training strength). The working hybrid: judge labels everything, humans audit a stratified sample and own every disagreement region — pairs where judge confidence is low route to humans entirely.
• The rubric is the actual spec: one dimension per preference pass ('prefer the response that escapes when evidence is genuinely insufficient, and commits when it isn't'), with tie rules ('if both are correct, prefer neither — drop the pair'; ties trained as preferences are noise with a direction). Multi-dimension preference passes produce averaged mush; run narrow passes sequentially if you need more than one dimension.

The three bias controls (run all three, every dataset)

• Length: measure chosen-vs-rejected length distributions; if 'chosen' averages meaningfully longer, your data teaches verbosity regardless of your rubric. Fix by construction (length-matched sampling) or by pruning until balanced.
• Position: judges (and tired humans) favor the first-shown response. Randomize presentation order and verify the choice rate by position lands near 50/50 — a cheap query that has caught expensive bias in most projects that bothered to run it.
• Sycophancy/agreement: check whether 'chosen' systematically flatters, hedges, or agrees more. For the triage project this is the whole point inverted — an escape-calibration dataset contaminated by hedging-preference teaches the model that uncertainty is always preferred, recreating the over-escape problem you set out to fix. Audit the balance: chosen responses should include plenty of confident commits on clear cases.