Evaluating the agent

"It seemed to work when I tried it" is not good enough for something people make decisions on. You need an eval set: a list of questions paired with the correct answer (which you get from the semantic layer directly), so you can measure the agent objectively.

Building the eval set

For each question, compute the ground-truth answer with the layer, then check the agent's response contains it. Because the layer is your source of truth, generating correct answers is trivial:

# Save this INSIDE the agent/ folder, next to agent.py and tools.py.
import os
import sys

# Same path trick as tools.py: make the sibling semantic_layer importable.
sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))

from semantic_layer import SemanticLayer
from agent import AnalyticsAgent

L = SemanticLayer()
agent = AnalyticsAgent()

CASES = [
    ("What was total revenue?",           str(int(L.query(["revenue"])["rows"][0]["revenue"]))),
    ("How many active subscriptions?",    str(L.query(["active_subscriptions"])["rows"][0]["active_subscriptions"])),
    ("Which channel has the lowest CAC?", "Organic"),
]

passed = 0
for question, expected in CASES:
    reply = agent.answer(question)["text"]   # fresh history for each case
    ok = expected.replace(",", "") in reply.replace(",", "")
    passed += ok
    print(f"{'PASS' if ok else 'FAIL'}  {question}  (want {expected})")
print(f"{passed}/{len(CASES)} passed")

Because eval.py lives in agent/ and inserts the kit root onto sys.path (the same preamble tools.py uses), its imports resolve whether you run python3 agent/eval.py from the starter-kit root or python3 eval.py from inside agent/. And note the signature: the shipped class is agent.answer(user_message, history) returning a dict — passing no history starts each eval case from a clean slate.

What evals unlock

• Safe iteration. Change the system prompt and re-run — did accuracy go up or down? No more guessing.
• Model comparison. Swap the model id and measure the quality/cost trade-off on your questions.
• Regression protection. When you add a metric or refactor, the eval catches anything you broke.