Back to course overview
Module 4Human-in-the-loop & reliability 13 min

Errors, retries & alerts

Apps go down, data arrives malformed, limits get hit. Reliable workflows expect all three: retries for the transient, a problem folder for the persistent, and alerts a human actually sees.

A workflow that ran perfectly all week will, eventually, meet reality: the sheets service has a bad minute, an email arrives in a format nobody imagined, the AI step times out mid-morning rush. Unreliable automations and reliable ones meet the same failures — the reliable ones were built expecting them. Three mechanisms cover nearly everything:

  • Retries, for transient failures. Most API errors are weather — down for seconds, fine on the second try. Most platforms offer retries — some automatic, some enabled per step or via an error-handler route; find yours and turn it on for every step that calls an external service. The one caution worth learning early: retry reads freely, but retry a send only if you're sure the first attempt didn't land — double-charging and double-emailing are retry bugs. When unsure, fail to the problem folder instead.
  • The problem folder, for persistent failures. When a run fails past retries or data won't parse, the run's details land in one designated place — a 'Failed runs' sheet tab, a triage channel — with what came in, which step failed, and the error. This is the unsure lane's sibling (engineers call it a dead-letter queue): nothing is silently lost, everything weird has an address, and Monday's fix-it session has a worklist instead of a mystery.
  • Alerts, for humans — tiered or ignored. A failed run posts to the problem folder quietly. Repeated failures (3+ in an hour: something systemic) ping the owner directly. The tiering is the point: alert on everything and alerts become wallpaper within a week; alert on patterns and a ping still means something. Silence-as-health only works if failures reliably make noise — that's what you're building.

The pre-mortem: five minutes that finds most of it

Before any workflow goes live, walk it step by step asking two questions: 'what if this step fails?' and 'what if this step's input is garbage?' Write one line per step. Riley's v2 pre-mortem found: AI step down → retries, then problem folder (don't guess a category); malformed AI output → already validated, unsure lane; sheet full/renamed → problem folder + owner ping; chat down → ticket still logs, ping retries (ordering matters: log first, notify second, so notification failures never lose data). Ten minutes, four decisions, zero incidents later — versus discovering each answer live, one bad morning at a time.

Prompt to try

Here is my workflow, step by step: [paste the spec]. Run a pre-mortem: for each step, tell me (1) the most likely failure, (2) what happens with my current design when it fails - trace it honestly, (3) whether the failure is transient (retry), persistent (problem folder), or dangerous (must stop the run). Flag any step where a failure could cause a duplicate send or lost data - those two are the ones I can't accept.

AI is a strong pre-mortem partner — breadth is the game and breadth is its strength. You adjudicate which failures are worth designing for; at this scale, most one-line answers are 'problem folder'.

The weekly health minute

Once live, a workflow needs sixty seconds a week: open run history — runs roughly the expected count? problem folder — empty or explained? approval rate — still sane? Calendar it. Automations don't fail loudly on day one; they drift quietly in month three, and the health minute is how drift gets caught at 'huh' instead of at 'incident'. (When your team runs many workflows across many systems, this minute is what grows up into real observability — Vigil-for-your-data being the same idea at warehouse scale.)