Rollbacks

You shipped v7. An hour later the sampled-quality score is sliding and complaints are trickling in. The single most important operational capability at this moment is rollback — getting back to a known-good state in seconds, before debugging, before blame. A team that can't roll back debugs under fire; a team that can debugs at leisure after the bleeding stops.

Rollback as a pointer change

This is why versioning was the prerequisite. If 'production' is a pointer to a version id, rollback is repointing it to the last-known-good — no code deploy, no prompt re-paste, no scramble to remember what v6 said. Seconds, not an incident bridge.

production_version = "triage-v7"     # a config value, not hardcoded
last_known_good     = "triage-v6"     # tracked explicitly, always

# rollback = set production_version = last_known_good.
# The kill switch's calmer cousin: don't stop the feature, revert it.

The last-known-good discipline

• Always know your last-known-good version — the one with the trusted eval score that ran clean in production. It's the floor you fall back to.
• Never delete old versions. The point of rollback is that v6 still exists, still runnable, exactly as it was. Immutable versions are what make reverting safe.
• Rollback first, root-cause second. Restore service, then pull v7's traces to understand what went wrong. Debugging a live regression while users hit it is a choice you don't have to make.
• A rollback is data. The failure that forced it becomes an eval case; v7's real problem, once understood, is a test that its replacement must pass.

Rolling forward safely

The fix for a bad v7 is not a panicked v8 hotfixed straight to production — that's how one regression becomes two. Roll back to v6, reproduce v7's failure in the eval harness (add the case), build v8 against the full golden set (old cases + the new failure), confirm it clears the gate, then ship with the same care as any release. The rollback bought you the time to do it right.