Instrumentation
You can't improve what you didn't log: the event schema for AI features, implicit signals that outrank thumbs, segment views, and the privacy line drawn in advance.
Deterministic features log clicks; AI features must log episodes — what the model saw, produced, and what the human did about it — because the improvement loop (next lesson) eats exactly that. The PM specs this instrumentation in the PRD, not as a telemetry afterthought, because what isn't captured at launch is unknowable forever after:
- The episode schema: input context (what the feature knew), output (what it produced, verbatim), model/prompt version (the lineage — every ops course's versioning lesson arriving in your PRD), user action (accepted / edited / rejected / ignored), and outcome where the workflow reveals it (ticket avoided? customer replied happy? recontact within 7 days?). Five fields; the whole improvement engine runs on them. One schema subtlety a PM must get right: the outcome fields are backfilled, not captured at write time. Recontact-within-7-days can't exist when the episode row is written — the seven days haven't happened. So the episode is logged with its outcome fields empty, and a downstream job joins the outcome in later (matching the episode to the customer's next 7 days of activity). Spec it that way; a schema that demands 'log the 7-day outcome at interaction time' is asking the row to predict the future, and engineering will (rightly) bounce it.
- Implicit signals outrank explicit ones. Thumbs up/down gets 1-3% participation from the annoyed and the delighted — real but skewed. The signals that scale are behavioral: edit distance (accepted verbatim vs. lightly edited vs. rewritten — the single best proxy for draft quality), acceptance rate by segment, abandonment (started reading the AI's message, closed it), and the follow-up question (a customer who immediately asks 'wait, which order?' just graded your message's clarity). Design the UX so these signals exist — an accept button that's distinguishable from an ignore is instrumentation disguised as interface.
- Segments or it didn't happen: aggregate acceptance hides everything you'll need — new vs. tenured customers, order value tiers, delay severity, time of day, mobile vs. desktop. The launch questions that matter ('is it working?') are all secretly segment questions ('for whom?'), and the segments must be in the schema on day one because they cannot be reconstructed later.
The privacy line, drawn before the first event fires
Episode logs are customer conversations and behavior — among the most sensitive telemetry a product emits. The PM's pre-launch obligations: the logging appears in the privacy policy in honest language; retention has a clock tied to purpose (improvement review ≠ forever); PII is redacted before storage (the pattern every operator course drills — here it's your spec line to write); access is role-limited and logged; and the 'can we use transcripts to improve the product?' question is answered deliberately with counsel, not assumed. One paragraph in the PRD, one meeting with whoever owns privacy — versus the alternative, which is discovering at quarter three that your improvement dataset is a compliance incident with a dashboard.
The forecast percentages from your scope contract (how much traffic takes each miss path) become measured numbers the day you launch — but only if the miss paths emit events too. Silent-stay-silent, route-to-human, fall-back-to-reactive: each logged with its trigger. The gap between forecast and measured miss rates is your first and best signal about whether the feature is meeting reality as scoped — and it's the number Module 5's launch review reads first.