Output validation

Everything so far made good output likely. Production needs more: a layer that makes bad output inconsequential. That layer is validation — ordinary code that inspects every model response before anything downstream trusts it. The mindset shift of this module: stop trying to make the model perfect; make the system safe around an imperfect model.

The three validator tiers

• Tier 1 — Shape: does it parse? Fields present, types right, enums legal? These are your assertions A1–A2, promoted from eval-time to every-call checks. Milliseconds, zero AI.
• Tier 2 — Business rules: the constraints that live in code because prose can be talked out of them (Module 1). order_id must match ^HL-\d{4}$ (a regex — a pattern language for text; you describe the check in plain English, e.g. 'HL then exactly four digits', and a developer writes the pattern) and exist in the orders table; reply must not match refund-promise patterns; urgency HIGH requires damage_mentioned or an explicit deadline in the email.
• Tier 3 — Grounding integrity: every citation quote string-matches its source; answerable_from_context false ⇒ reply contains no policy claims. Your hallucination tripwire from Module 4, automated.

One clarification the order_id rule invites: assertion A3 (from your eval suite) checks the model captured the id exactly as written in the email — that's a faithfulness check, separate from the ^HL-\d{4}$ format rule, which is a business rule. They don't conflict: an id that doesn't match the format is never silently rewritten to fit — it's flagged and routed to human review. Capture verbatim; validate the format; route the mismatches.

When a validator fails: retry with feedback

A failed check doesn't mean give up — it means the model gets one precise second chance:

Your previous output failed validation:
- order_id "HL-99" does not match format HL-#### — return the id exactly as
  written in the email, or null if none is present.
- citation quote for S2 was not found verbatim in S2 — copy the exact text.
Return the corrected JSON only.

• Feed back the specific violations, not 'try again'. Targeted feedback fixes most failures in one retry; vague feedback re-rolls the dice.
• Cap retries at 1–2. A model that fails twice on the same input is telling you the input is pathological — that's a routing problem (next lesson), not a retry problem.
• Log every validation failure with its input. Recurring failure patterns are your prompt backlog and your golden-set feed. A validator without logging fixes today's call and wastes tomorrow's lesson.