Human-in-the-loop
Designing checkpoints so review happens where it's effective — and staying sharp against automation complacency as trust grows.
'A human reviews it' is the safety story behind every workplace AI deployment. It's only true if the review is designed — placed where a human can actually catch what matters, not stapled on at the end where errors are polished and invisible.
Checkpoint placement beats checkpoint count
Recall Sam's workflow: extract facts → human reviews the facts → format the summary → human skims and sends. The review sits at the facts stage, where errors are raw bullets that are easy to spot — not at the polished-prose stage, where a wrong number wears a suit. The principle: review upstream, where errors are naked; approve downstream, where they're dressed.
- Review the inputs when garbage-in is the risk (did the paste truncate? right week's data?).
- Review the extraction when facts matter (bullets, tables, claim lists — fast to scan, hard for errors to hide).
- Review the judgment always — any recommendation, ranking, or decision the output implies is yours to make, not confirm.
- Skim the final for tone and constraint compliance only. If you're catching factual errors at the final stage, a checkpoint upstream is missing.
Automation complacency: the trap that opens after success
The dangerous phase isn't week one — you check everything in week one. It's month three, after fifty flawless runs, when your 'review' has quietly become a scroll-and-approve. Complacency isn't a character flaw; it's what human attention does around reliable automation. You counter it with structure, not willpower:
- 1Make review active, not passive. The reviewer must produce something — tick a 3-item checklist, correct one field, initial the verified number. Rubber stamps atrophy; tasks don't.
- 2Spot-audit on a schedule. Every tenth run (calendar it), verify one output fully as if it were week one. Log what you find.
- 3Track the catch rate. If your audits find zero issues for months, either the workflow earned lighter review (fine — decide that explicitly) or the audits went soft. Decide which on evidence, not vibes.
You'd never let even a brilliant intern send client emails unreviewed in week one — but you also wouldn't still be re-checking every comma in month six. Graduated trust with periodic audits is how you treat people; it's exactly how to treat the model. What you never delegate to either: your judgment and your name.