Monitoring
Watching a RAG system live: the metrics that leant early, the traces that make debugging possible, and the weekly loop that feeds the eval set.
Evals tell you how the system performs on questions you thought of. Monitoring tells you what's happening with the questions users actually ask — and in RAG, the gap between those two is where systems quietly rot. You already log full traces (Module 4). Monitoring is deciding which numbers from those traces get watched, and by whom.
The dashboard that matters (one screen)
- Retrieval health: top-score distribution over time (a sagging median = corpus or drift trouble), and the no-good-chunk rate (queries below FLOOR). A rising abstention rate is your earliest signal that users want something the corpus doesn't have.
- Trust layer: citation-verification failure rate (should be near-zero and boring — any spike is a model change, a prompt regression, or an index/content mismatch), confidence mix (HIGH/MEDIUM/LOW percentages), sampled groundedness score.
- Operations: p50/p95 latency per stage, per-query cost, cache hit rates, calls-per-question.
- Demand: query volume by category filter, top queries, and — the goldmine — top abstained queries. The questions you couldn't answer, ranked by frequency, is a literal content roadmap for whoever owns the docs.
Alerts: few, meaningful, actioned
Alert on: verification failure rate above 2× baseline for 30 minutes (something changed — find which model/index/prompt version moved in the trace metadata); no-good-chunk rate doubling day-over-day (corpus gap or ingestion failure — check the ingestion log first); cost or latency breaching budget. Everything else is a dashboard, not a page. An alert nobody acts on trains everyone to ignore the one that matters.
The weekly loop (the whole system's heartbeat)
- 1Sample 20 traces: 10 random, 5 lowest-confidence, 5 most-frequent queries. Read them like the Module 5 red-team: clean / gap / blend / synthesis / stale.
- 2Feed findings where they belong: bad answers → golden set; abstained-but-should-answer → docs team (with the frequency ranking); stale answers → ingestion pipeline check.
- 3Re-run the eval suite (scheduled anyway) and read it next to the production metrics. Eval-fine-but-production-degrading = your golden set no longer smells like traffic — refresh it.
The abstained-queries report alone justifies the logging: it turns 'the bot couldn't answer' from a failure into a prioritized list of documentation to write. Teams that ship this report to their content owners watch coverage — and answer rates — climb quarter after quarter.