Probabilistic products

Meet Dana Whitfield, product manager at Harbor Lane. Dana has shipped a checkout flow, a loyalty program, and a mobile app — deterministic products, where 'does it work?' had a yes-or-no answer. Now Dana owns Harbor Helper, the AI support assistant, plus AI features sprouting across the roadmap — and the first thing to understand is that the PM job itself changes, because the product's core behavior is probabilistic: the same input can produce different outputs, excellent on average and wrong on Tuesdays, with no bug to fix because nothing is broken.

What actually changes (and what doesn't)

• The spec becomes a distribution, not a behavior. 'Clicking Send sends the message' was a spec. 'The assistant resolves the customer's issue' is a target with a hit rate — the honest spec says 'resolves ≥N% of in-scope issues at quality ≥X, and here is exactly what happens in the other cases.' Writing the second half — the miss behavior — is the new core PM skill; deterministic PMs never had to spec what 'wrong' looks like.
• QA becomes evaluation. You can't test all paths when outputs vary; you measure rates on representative sets. (An eval is a repeatable measurement of output quality on a fixed set of cases — the AI version of a regression test; Module 3 makes evals a PM tool, not just an engineering one.) 'It worked in the demo' is now a meaningless sentence, and PMs who don't internalize that ship demos.
• Failure is a product surface, designed like one. Some percentage of interactions will go wrong at steady state, forever. The product decisions that matter most live there: what does the user see, how do they recover, who gets told, what does it cost trust? A probabilistic product is judged by its worst 5%, not its median.
• What doesn't change: users, problems, and value. The oldest PM discipline — is this a real problem, for real users, worth solving? — is more important, because AI's coolness generates solutions in search of problems at industrial scale. Half of Dana's job is still saying 'no, that's a tech demo, not a feature.'

The three questions on every AI feature's door

Dana's desk rule — before any AI feature enters the roadmap, three questions in order: (1) What's the cost of a wrong answer? (annoyance? money? trust? safety? — this sets the human-oversight design and often decides feasibility alone); (2) How will we know it's good? (if quality can't be measured, it can't be managed, shipped responsibly, or improved — 'we'll see how it feels' is a pre-registered failure); (3) Why does AI beat the boring alternative? (a lookup table, a form, a human — AI earns its complexity only where judgment-at-scale is genuinely the job). Features that survive all three are worth discovery. The order matters: teams that start with 'what can the model do?' answer none of them.