Module 4Human-in-the-loop & reliability 16 min

Lab: The approval gate

Ship v3: AI-drafted replies behind a real approval gate, a problem folder wired to every failure path, and a break-it-on-purpose test session.

Final build on Riley's workflow: v2 triaged; v3 acts — drafting customer replies and sending them through a gate you operate. Plus the reliability layer: problem folder, tiered alerts, and a testing session where you attack your own build. Same translation rule as every lab: part of the work is finding each construct — the delay, the reminder, the escalation — in your tool, and the pending-row gate is the universal fallback if yours has no native approval step.

workflow spec — OPS email->ticket v3 (additions to v2)text

PATH B+   (normal, confidence=high, category in [question,
           refund_request]):
  AI STEP  draft reply - contract prompt: friendly, 3 sentences max,
           facts ONLY from {{trigger.body}} + {{ai fields}};
           if a needed fact is missing, write [NEED: what] instead
           of inventing it
  GATE     approval message to [your chat/email]: trigger summary,
           AI reasoning fields, full draft, APPROVE / REJECT
           timeout: 4h -> remind; 8h -> escalate row 'stale'
  ON YES   send reply (from test mailbox), log 'sent by: [approver]'
  ON NO    row status='rejected', draft kept for review
RELIABILITY (whole workflow):
  problem folder: 'Failed runs' tab - every failure path lands here
  alert: 3 failures/hour -> direct ping to owner (you)

1Add the drafting AI step with its contract prompt. The [NEED: ...] convention is the drafting equivalent of confidence: low — a sanctioned way to be incomplete instead of inventive. Test it on an email that lacks an order number and confirm you get [NEED: order number], not fiction.
2Build the gate with your platform's approval construct. The message must carry everything (spec above) — test the five-second rule on yourself tomorrow morning: can you judge the draft without opening anything else?
3Wire the reliability layer: every failure path → 'Failed runs' tab; add the 3-per-hour owner ping. Confirm the ordering — log the ticket before attempting any notification.
4Break it on purpose (the fun part): send an email that produces an unanswerable draft (expect [NEED]), temporarily rename the tracker sheet (expect problem folder, not silence — then rename it back), reject a draft (expect 'rejected' status, nothing sent), and let one gate time out (expect the reminder). Each behavior you just verified is one 3am incident that now can't happen.
5Run v3 for a real week on your test mailbox with daily health minutes, then write the one-paragraph verdict: runs, approval rate, problem-folder count, minutes of your attention consumed. That paragraph is the shape of the capstone's evidence.

The gate stays until the data says otherwise

After a clean week, you'll itch to remove the gate — 'it approves everything anyway.' Resist, and re-read the approval-rate discipline: a high approval rate is the argument to narrow the gate (auto-send question replies, keep gating refund_request), never to delete it wholesale. Narrow with data, sample what you ungate, keep the log. That's how trust is extended — one category at a time, reversibly.

Problem set 4

Three gate designs to critique (an approval message missing the draft, a timeout that silently auto-sends, an alert scheme that pings on every failure), plus a retry-safety table to fill in: for eight step types, is retry safe, unsafe, or safe-only-with-a-check? The double-send question, drilled until it's reflex.

PreviousErrors, retries & alerts Next Capstone: Design