Lab: Build an eval suite

Step 1 — Write the cases

1. 1Create eval.py inside the agent/ folder with the sys.path preamble from the lesson — that's what lets it import both the semantic layer and the AnalyticsAgent class no matter where you run it from.
2. 2Write ~10 questions spanning lookups (total revenue), slices (revenue by region), rankings (top channel by CAC), and refusals (what's our NPS?).
3. 3For factual ones, compute ground truth with the semantic layer so your answer key is always correct.
4. 4For refusals, assert the reply admits it can't answer rather than containing a fabricated number.

Step 2 — Run and read the failures

1. 1Run it — python3 agent/eval.py from the starter-kit root (venv active, ANTHROPIC_API_KEY set) — and note which cases fail.
2. 2Read the agent's actual replies on failures. Is it a formatting mismatch (right number, different format), a wrong metric choice, or a real error?
3. 3Failures are information — they tell you exactly what to fix.

Step 3 — Improve and re-measure

1. 1Pick the most common failure and address it in the system prompt (e.g. clarify a formatting rule, or add an example of the right metric to use).
2. 2Re-run the eval. Did the score improve without breaking other cases?
3. 3This measure → change → re-measure loop is how you improve any LLM system with confidence.