A/B testing
Comparing versions with evidence: offline A/B on the golden set vs. online A/B on live traffic, reading paired results, and honest significance.
You have two versions and a claim that B is better. LLMOps answers 'is it?' two ways — before shipping and after — and both matter, because they measure different things.
Offline A/B: the golden set (before shipping)
Run both versions through the eval harness on the same golden set. This is your Prompt Engineering protocol, now automated: change one thing, run both, and compare paired, not pooled — list the cases where A won and B won, and read the flips. B fixing three hard cases while breaking two easy ones is a completely different result from 'B +1 overall,' and only the paired view shows it. Offline A/B is fast, cheap, and deterministic-ish (average N runs) — it's the gate before anything reaches users.
Online A/B: live traffic (after shipping)
Offline evals can't measure everything — real user satisfaction, the true input distribution, downstream outcomes. Online A/B routes a fraction of real traffic to version B and compares outcome metrics: thumbs-up rate, escalation rate, task completion, cost. It's slower and needs volume, but it measures reality. The relationship: **offline A/B decides what's safe to try on users; online A/B decides what's actually better for users.** You need both — offline as the safety gate, online as the truth.
- Route by a stable key (user or session hash) so a given user gets a consistent version — flipping versions mid-conversation is its own bug.
- Pick the metric before you start. 'We'll see what looks good' invites reading noise as signal. Pre-register: 'B wins if thumbs-up rate rises ≥3 points with no cost regression.'
- Guard the downside. An online test is real users getting version B; keep the traffic slice small until offline evals and early online numbers say it's safe, and keep the kill switch ready.
Honest significance at your scale
Two traps. Offline: a 1-case difference on a 60-case set is noise — a variant wins when it fixes multiple hard cases and breaks nothing, and the flips survive a re-run. Online: small traffic + short windows produce swings that look decisive and aren't; wait for enough volume, and beware novelty effects (users react to change, not just quality). When in doubt, the offline number is the more trustworthy one — it's controlled.
Checking an online A/B test repeatedly and stopping the moment B looks ahead manufactures false wins — you're sampling noise until it flatters you. Decide the sample size or duration up front and read the result once at the end. The offline golden set doesn't have this problem, which is another reason it's your primary gate.