Back to course overview
Module 3Evals as a PM tool 13 min

Defining quality

'Good' is a PM deliverable: decomposing quality into dimensions, turning dimensions into rubrics a grader can apply, and setting bars an argument can't move.

Here is the division of labor that makes AI teams work: **engineering owns making the number go up; the PM owns what the number is.** 'What does good look like?' is a product question wearing technical clothes — it encodes user needs, brand standards, and risk appetite — and PMs who delegate it discover their feature optimized toward whatever was easiest to measure. The craft:

  • Decompose 'good' into 3-5 named dimensions. For the proactive-delivery message: correct (the delay is real, the order is right, the facts check), appropriate (right customer, right moment, honors the non-goals), on-brand (Harbor Lane's voice — warm, brief, no groveling), actionable (the proposed resolution is one the customer can actually accept). Dimensions force clarity aggregate scores hide: a feature can be 90% 'good overall' while failing appropriate exactly where it's expensive.
  • Turn each dimension into a rubric a stranger could apply: pass/fail criteria with two worked examples each — one clear pass, one instructive fail. The rubric-writing test: if two graders disagree on a case, the rubric is ambiguous, and ambiguity here becomes noise in every number downstream. (Recognize this discipline? It's the labeling guide from the data courses and the judge rubric from the agents course — where a judge is a model that grades outputs against your rubric, the machinery the next lesson builds — and the PM writes the product version.)
  • Set the bar per dimension, weighted by the error map: correct might need 98% (facts are cheap to verify and expensive to miss); on-brand might live at 90% (an occasionally stiff sentence is survivable). One aggregate bar ('95% good') is a negotiation trap — under pressure, the cheap dimensions get optimized to subsidize the expensive one. Per-dimension bars, pre-agreed, are how the PM's quality contract survives a deadline.

Quality is a contract, and contracts get signed before the work

The bars get agreed — PM, engineering, and the accountable business owner — before the build sprints, for the same reason kill criteria get written at funding time (strategy course) and baselines get frozen before launch (finance course): afterward, every party's judgment is bent by sunk cost. The signed artifact is one page: dimensions, rubrics, bars, and the sentence that gives it teeth — 'v1 ships when the eval set clears every bar; no bar moves without the three signatures that set it.' Three signatures, named: the PM (owns what 'good' is), the engineering lead (owns whether the number can be moved), and the accountable business owner (owns the risk of shipping). Dana's proactive feature will hit a week where it's at 96.5% correct against a 98% bar and the quarter is ending; the one-pager is what makes that week a data conversation instead of a pressure conversation.

Selling the signatures is its own skill, because a skeptical exec hears 'quality contract' as bureaucracy — a gate that slows the thing they want shipped. Dana reframes it as risk control, not process: 'we agree what "good enough to ship" means BEFORE we've sunk the cost, so the launch decision isn't a fight later — it's a checkbox we already designed.' That lands, because the exec has lived the alternative: the quarter-end argument where nobody wrote down the bar and the loudest narrative wins. When a signature stalls — the business owner won't commit to 98% correct — the unblock is to ratchet, not stall: start with a lower bar you can defend from the prototype evidence ('90% correct at launch, gated to the tolerant workflow'), ship, and raise it on a written schedule as the loop earns each notch. A defensible bar today beats a perfect bar that never gets signed.

Beware the measurable crowding out the meaningful

Some dimensions rubric easily (correctness); others resist (does the proactive message feel caring or surveilling? — same facts, opposite products). The resistible ones don't get dropped; they get human panels and user research instead of automated graders, and they keep their seat in the contract. The most common failure of eval-driven product culture isn't measuring badly — it's quietly narrowing 'quality' to what measures cheaply. Holding the hard dimensions open is a PM responsibility nobody else will pick up.

Three terms you'll live in from here

The next lesson operationalizes this contract, so meet the vocabulary now. A golden set is the fixed, curated collection of cases you run your rubrics against — the 'eval set' this lesson keeps referring to, made concrete and version-controlled. A judge is a model that applies your rubric to outputs automatically, so you can score every change instead of only what a human has time to read. Calibration is the check that the judge agrees with human graders on the same cases — the receipt that lets you trust the judge's numbers. This lesson wrote the contract; the next builds the instruments that enforce it.