A/B testing prompts

With a golden set and checks, you can finally answer the daily question of prompt work — is version B actually better than version A? — with evidence instead of adrenaline. The method is ordinary experimental hygiene, applied to prompts.

The protocol

1. 1Change one thing. New example set OR reworded rule OR reordered sections — never all three. Bundled changes produce unattributable results, and unattributable results teach you nothing for next time.
2. 2Run both versions on the full golden set — same cases, same settings. Because outputs vary run-to-run, run each version 3× on borderline cases (self-consistency, now serving measurement).
3. 3Compare paired, not pooled. Don't just compare totals — list the cases where A passed and B failed, and vice versa. The flips tell the story: B fixing 3 hard cases while breaking 2 easy ones is a very different result from 'B +1 overall'.
4. 4Read every regression. A case that flipped from pass to fail is either a real regression or a gold answer that deserves rethinking. Decide which, explicitly.

Honest statistics at golden-set scale

With 20–50 cases you're not running clinical trials — a 1-case difference is noise, and pretending otherwise is self-deception with charts. Practical significance bar: a variant wins when it fixes multiple hard cases, breaks nothing you care about, and the flips survive a re-run. For subtler differences, grow the eval set before growing your confidence.

• Beware overfitting to the golden set. Tuning examples until exactly your 17 cases pass can memorize the set rather than learn the task. Defense: hold out 3–5 cases you never tune against, and refresh with production cases regularly.
• Judge dimensions A/B the same way: run the judge over both versions' outputs, compare paired scores, read the worst_sentence fields — they're a map of what the variant actually changed.
• Log every experiment — one line: date, versions, hypothesis, result, decision. Your future self will otherwise re-run this exact experiment in three months.