Metrics that keep deflection honest
Containment is the metric everyone quotes and the easiest to corrupt: the balanced scorecard — resolution, escalation quality, CSAT split by path, and the honest-miss rate.
Customer-agent programs live and die by one measurement mistake: optimizing containment (conversations the bot finished alone — a.k.a. deflection) as if it were success. Containment counts the customer who gave up in disgust as a win. The balanced scorecard that keeps everyone honest:
- Resolution rate, verified — not 'conversation ended' but 'problem actually solved': measured by explicit confirmation, no-recontact within 7 days on the same issue, and sampled human review. The recontact check is the killer: high containment + high 7-day recontact = a bot that postpones work while degrading trust. This pairing is the single most diagnostic number pair in the whole program.
- Escalation quality — of conversations that handed off: did the right triggers fire (sampled review), how fast (turns-to-handoff when customer asked), and did the package hold (rep re-ask rate from the drills, now measured live). Great escalation numbers are a feature metric, per Module 4's framing.
- CSAT, split by path — one aggregate CSAT hides everything. Split: bot-resolved vs. handed-off vs. bot-attempted-then-recontacted. Healthy pattern: bot-resolved CSAT ≈ human CSAT on easy issues, handed-off CSAT close behind (good sorting feels good), and the third bucket small. An agent with great bot-resolved CSAT and a large angry third bucket is doing selective accounting.
- The honest-miss rate — how often the agent said 'I don't know' + routed, vs. improvised. You want a floor above zero (zero means it's guessing somewhere), and each miss is logged as improvement fuel: knowledge gap, scope gap, or retrieval bug.
- Cost & speed, in context — cost per resolved conversation (not per conversation — resolving matters), median time-to-resolution vs. the pre-agent baseline. The business case lives here; just never present it without the quality row above it.
One target, stated the right way around
Set the target as a quality floor with a volume ambition, never the reverse: 'Resolve ≥N% of in-scope conversations at CSAT ≥ human-baseline and 7-day recontact ≤ X%' — not 'contain 60% of tickets.' Teams given the second target will hit it, and the way they hit it is the stonewall. The metric definition is the incentive design; the BA and Data Foundations courses drill this, but you don't need them for the takeaway, which is right here: write the definitions down, get them signed, and let no dashboard compute them differently.
No dashboard replaces reading: one hour weekly, 15 transcripts — 5 random, 5 worst-CSAT, 5 handoffs. Every mature team that runs customer agents converges on this ritual, because the dashboard tells you that something moved and transcripts tell you why. Rotate who reads; bring one finding each to Module 5's improvement loop.