Fallback strategies
Designing the paths for when the happy path fails: the degradation ladder, confidence-based routing, and budget circuit breakers.
Retry handles the recoverable blip. Fallbacks handle everything else: the email your prompt genuinely can't classify, the model timing out, the case where confidence is honest LOW. A production prompt system is defined less by its happy path than by its ladder of dignified exits.
The degradation ladder
- 1Full success — validated output, HIGH/MEDIUM confidence → proceed automatically.
- 2Partial success — the classification validated but the reply draft failed its checks → use the classification, route the reply to a human. Salvage the trustworthy fields instead of discarding the whole output.
- 3Simplified retry — the rich task failed twice → fall back to a smaller ask: just category + urgency, no reply. A modest answer that validates beats a rich one that doesn't.
- 4Route to human — LOW confidence, split self-consistency votes, repeated validation failure, or any red-flag category (legal threats, safety) → into the human queue with the model's work attached (draft, reasoning, flags) so the human starts at 80%.
- 5Fail visibly — the system itself is down → queue and say so. The one forbidden state is silent failure or fabricated success.
Routing on confidence — now it pays off
Every module quietly built inputs for this decision: the confidence field (M2), self-consistency agreement (M3), answerable_from_context (M4), validator results (M6). Routing is just an honest table over those signals — this combination proceeds, that one gets a human. Write the table down; it is your risk policy, and it's what an auditor, a boss, or the governance course will ask to see.
Circuit breakers: budgets as guardrails
- Latency budget — triage worth having answers in seconds; hang past the budget → route on without it.
- Cost budget — retries + self-consistency multiply spend; cap per-item cost and alert on daily totals. A runaway retry loop should page you, not surprise finance.
- Rate circuit breaker — validation failures spike above X% for 10 minutes (model drift? provider incident?) → auto-route everything to humans and alert. The system degrades to 'how it worked before AI', which is the correct worst case.
Here's my triage flow and my routing signals [paste]. Stress-test my fallback design: for each of these events — model outage, sudden 30% validation-failure spike, an email in Portuguese, a legal threat, 10× normal volume on Black Friday — walk through my ladder and tell me where the design is silent or wrong.
Never fall back to pretending. No fabricated classification to keep the pipeline moving, no generic reply that implies a human read the email. Every dignified exit above preserves one invariant: the output honestly represents what the system knows.