Logging & metrics

Traces debug one request; metrics — traces aggregated over time — reveal the health of the whole feature. The art is choosing metrics that expose the reliability gap, not just the infrastructure. Uptime and error rate are necessary and completely insufficient.

The metric families that matter for AI

• Quality proxies — you can't grade every live request, but you can track signals: user thumbs-down rate, validator-rejection rate, escalation rate, abstention rate, the share of low-confidence responses. A rising validator-rejection rate is drift or an attack, visible before any complaint.
• Cost — spend per request, per feature, per day; tokens in/out; calls-per-request (a creeping average > 1 means retries or agent loops multiplying). This is the metric that gets features cancelled; watch it like uptime.
• Latency — p50 and p95 (never just average — the tail is where users rage-quit), broken down by span so you know which stage is slow.
• Volume & mix — requests over time, and the distribution of inputs (categories, lengths, languages). A shift in the mix is the drift signal that a frozen eval set will miss.
• Infrastructure — the traditional stuff (uptime, 5xx, timeouts) still matters; it's just not sufficient.

Sampling and continuous evaluation

You can't judge every production response, but you can sample — run your Module 2 judge over a random slice of live traffic (say 5%) and track the score over time. This 'online eval' is the bridge between your offline golden set and reality: it catches quality regressions the golden set can't, because it's scoring the actual weird inputs users send. When online quality diverges from offline quality, your golden set has drifted from production — refresh it. Mind the volume: at low traffic a 5% sample can be a handful of requests a day — pure noise you'll misread as a trend. Sample a higher fraction, or aggregate over a longer window, until the number is stable (the same significance caution as online A/B in m4).

The dashboard that earns its screen

• One view: quality proxies, cost/request, latency p50/p95, volume, and sampled-judge score — each as a line over time, because trends beat snapshots. A number is fine; a number bending is the alert.
• Top failures and top abstentions, ranked by frequency — your prioritized bug list and content-gap roadmap (the abstained-queries goldmine from RAG, generalized).
• Segment by prompt version — so a metric that moves lines up against 'we deployed v7 on Tuesday.'