Lab: The preference pass

A complete preference cycle on the project's real gap. The lab's shape is the professional shape: narrow target, measured before, biased-controlled data, one pass, measured after — with the honest possibility that the answer is 'it helped a little,' because that's frequently the truth about preference passes and you should experience reporting it.

The preference-file shape. DPO data is JSONL with three fields per row — the prompt and the two responses to rank:

{"prompt": "Subject: Is this charge mine?\nThere's a $12 charge I don't recognize but the merchant name is cut off.", "chosen": "{\"category\": \"billing\", \"urgency\": \"low\", \"extraction\": {\"order_id\": null}, \"escape\": \"unclear\"}", "rejected": "{\"category\": \"chargeback\", \"urgency\": \"high\", \"extraction\": {\"order_id\": null}, \"escape\": null}"}
{"prompt": "Subject: Cancel order A-9920\nPlease cancel order A-9920, I ordered by mistake.", "chosen": "{\"category\": \"order_status\", \"urgency\": \"high\", \"extraction\": {\"order_id\": \"A-9920\"}, \"escape\": null}", "rejected": "{\"category\": \"order_status\", \"urgency\": \"low\", \"extraction\": {\"order_id\": \"A-9920\"}, \"escape\": \"unclear\"}"}

A minimal DPO skeleton — again a reference shape, not guaranteed-runnable; confirm argument names against current TRL/PEFT docs. Note beta: it sets how hard the update is pulled toward the frozen reference model (higher = more conservative):

# Reference shape, NOT guaranteed-runnable. Training APIs move fast.
# Pinned for orientation: transformers 4.x, peft 0.x, trl 0.x (2026).
# Check current TRL/PEFT docs before relying on any argument name below.
from trl import DPOConfig, DPOTrainer

# Start from the SFT'd student (and its tokenizer) you trained above.
config = DPOConfig(
    output_dir="./triage-student-dpo",
    per_device_train_batch_size=2,
    num_train_epochs=1,          # a preference pass is usually short
    learning_rate=5e-6,          # much lower than SFT; DPO is sensitive
    beta=0.1,                    # KL strength vs. the frozen reference model
)

trainer = DPOTrainer(
    model=model,                 # the SFT'd student
    ref_model=None,              # None = an internal frozen copy is used
    args=config,
    train_dataset=pref_dataset,  # prompt / chosen / rejected rows
    processing_class=tokenizer,
)
trainer.train()

1. 1Measure the target precisely first: from Module 3's eval, isolate the escape-calibration numbers — over-escape rate (escaped when evidence sufficed) and under-escape rate (committed and wrong) by class. Write the target: 'halve under-escape without raising over-escape above X%.' A preference pass without a pre-registered target is a vibe with a GPU bill.
2. 2Harvest pairs: run the SFT'd student at temperature over 400 boundary-rich prompts (generate more via the Module 2 pipeline if needed), 3 samples each; keep prompts where samples disagree about escaping — those disagreements are your natural pairs (~200-400 expected).
3. 3Label with the calibrated hybrid: write the one-dimension rubric; hand-label 50 pairs; calibrate the judge to ≥85% agreement (expect one rubric revision); judge the rest; hand-adjudicate every low-confidence pair. Drop ties.
4. 4Run all three bias controls on the record: length distributions, position-balance check, and the commit/hedge balance (this dataset's sycophancy variant). Fix what fails — the commit/hedge balance usually needs deliberate rebalancing, per the lesson's warning.
5. 5Train the DPO pass and evaluate the full suite: not just the target numbers — the whole eval, because preference passes can move things you didn't aim at (watch schema adherence and the general slice specifically). Report the three-way honest comparison: SFT model / SFT+DPO / teacher, target metrics bolded, side effects named. Then the judgment call the whole module built toward: does the improvement justify carrying a second training stage forever? Write the recommendation either way — 'ship the DPO pass' and 'the SFT model plus a threshold tweak achieves 80% of this for free' are both passing answers when the numbers argue for them.