Continuous improvement
The weekly loop that compounds: triage the signals, sort fixes by layer, ship behind the regression gate — and know the difference between drift and decay.
AI features are never done — they're operated, and the products that pull ahead aren't the ones that launched best but the ones whose improvement loop cycles fastest. The loop, as the PM runs it:
- Weekly triage, one hour, standing: the episode dashboard (acceptance, edit distance, miss-path rates vs. forecast, by segment), the worst-20 episodes read raw (the transcript hour — the PM reads product truth; engineering reads failure mechanics; do it together), and the week's harvest: 3-5 candidate improvements + new golden-set cases from anything that surprised.
- Sort fixes by layer, cheapest first: most 'AI quality' issues fix without touching the model — copy & UX (the confusing message that edits always rewrite the same way — fix the template), prompt & retrieval (the segment where context is stale), scope (the input family that should route to a miss path instead of being attempted), and only then model (the systematic gap that survives the cheaper layers — which is when the fine-tuning conversation is honest, and you now know the course that adjudicates it). PMs who learn this sorting stop spending model-sized budgets on template-sized problems.
- Everything ships through the regression gate: any change — prompt, template, threshold, model — runs the full golden set before production, because AI changes have side effects the way all your operator courses warned. The PM's role isn't running the gate; it's refusing to bless anything that skipped it, including their own copy tweaks. Especially those.
Drift vs. decay: the two ways good features go bad
Drift is the world changing under the feature: new delay patterns after a carrier switch, seasonal traffic shapes, customers learning to game the proactive resolution ('if I complain about delivery I get a credit'). Watch for it in the segment trends and the miss-path rates — drift shows up at the edges first. Decay is the system changing under the world: a model provider update shifting behavior (version-pin and re-eval on notification — the vendor-management clause every course from finance to fine-tuning demands), a knowledge base going stale, a prompt edited five times by three people into incoherence (the changelog discipline, again, forever). Both look like 'quality is slipping' in aggregate; they route to completely different fixes, and the instrumentation you specced in the last lesson is what tells them apart. (A terminology note so you're not thrown in a cross-team meeting: the ML industry usually reserves 'drift' for the world changing — data drift and concept drift — and has other names for the system side. This course's clean drift-vs-decay split is a deliberate PM-facing framing: two buckets that route to two different owners and two different fixes, which is the distinction a PM actually acts on. Know both vocabularies; use whichever your team does.) A quarterly 'drift review' — is the golden set still shaped like production? are the segments still the right segments? — keeps the loop itself from decaying.
The steady-state cost of an AI feature is not its API bill — it's the weekly hour, the eval upkeep, the golden-set curation, the periodic model/prompt work: roughly 15-20% of an engineer and a real slice of PM attention, indefinitely. Features that launch without this line in the plan degrade on schedule, and their retros all contain the same sentence: 'it worked great at launch.' Put the line in the PRD. Defend it in planning. It is the product.