Lab: Add tracing

Your feature gets eyes. You'll wrap it in tracing, run varied traffic through it, aggregate a metrics view, and then debug a failure using only its trace — the full observe-and-diagnose loop.

Step 1 — Instrument

1. 1Add the Tracer from the lesson. Wrap your feature's entry point so every request emits a trace with: full prompt sent, full response, model+version, tokens in/out, cost, latency, and a trace_id.
2. 2If your feature is multi-step (RAG or agent), open a child span per stage (retrieval, each model call, each tool). The trace should read as the request's whole journey.
3. 3Stamp each trace with the feature/prompt version (a constant for now; Module 4 makes it real).

Step 2 — Generate traffic

1. 1Run 40–50 varied inputs through the feature: your golden set plus a dozen deliberately weird ones (empty, huge, adversarial, off-topic). You want a realistic spread of traces to aggregate.
2. 2Confirm traces are landing in traces/ and are complete — open two by hand and verify you could reconstruct exactly what happened.

Step 3 — The metrics view

1. 1Write dashboard.py: read all traces and print the health view — count, cost total + per-request, latency p50/p95, token averages, and a table of any flagged/validator-rejected requests.
2. 2Add a sampled online-eval: run your Module 2 judge over a random 20% of the traces and report the average score. This is your production-quality proxy.
3. 3Add 'top failures' — group the low-scoring or flagged traces and show the most common pattern.

Step 4 — Debug from a trace

1. 1Introduce a planted bug: truncate assembled context to 200 chars before the model call (simulating a retrieval/assembly failure). Re-run traffic.
2. 2From the dashboard, spot the quality drop. Pull one bad trace. Practice the debugging order: read what the model saw → localize the span → classify (it's 'bad input to the model', not a dumb model) → confirm.
3. 3Fix it, add the failing case to golden/v2, re-run evals to confirm, and note in LAB-NOTES.md: how the metrics showed it, how the trace localized it, and that the eval now guards it.