Lab: Eval suite

The measurement capstone: the resolution agent gets a real evaluation system that scores outcome, process, safety, and cost — the scorecard you'll defend in Module 8. Everything you've built for the agent finally gets held to account.

Step 1 — The task suite (~20 specs)

1. 1Convert your accumulated test tickets into task specs: initial state (ticket + world snapshot), gold outcome (decision, amount, escalation), required actions, forbidden actions.
2. 2Cover the archetypes: ~8 straightforward, ~4 multi-step, ~3 should-escalate (ambiguous/out-of-policy), ~3 adversarial (injection, over-limit), ~2 unanswerable. Include the weaponized-note ticket.
3. 3Mark 4 holdouts. Log every gold-outcome judgment call.

Step 2 — The harness

1. 1Write eval_agent.py: run each task 3× (variance!), capture full trajectories and the action log.
2. 2Score outcome mechanically where possible (right amount? right final status? escalated when required?) against the gold spec.
3. 3Score process + classify failures with the trajectory-judge prompt from the lesson. Calibrate the judge against your hand-labels on 10 trajectories first.
4. 4Compute safety (wrong actions executed — must be 0; invalid actions requested — track) and efficiency (turns, tokens, cost per task) from the logs.

Step 3 — The scorecard

1. 1Produce one scorecard: task success rate (with variance across the 3 runs), process-quality distribution, safety counts, cost/turns per task, escalation precision & recall.
2. 2Build the failure-type histogram from the judge's classifications. The tallest bar is your next fix.
3. 3Sample 3 successful trajectories and read them by hand — hunt the lucky-success latent failures the scorecard blessed.

Step 4 — The honest read + one fix

1. 1Write the one-paragraph verdict: success rate, the dominant failure type, the safety result, cost per task, weakest archetype.
2. 2Apply the single highest-value fix the histogram points to (a tool description, a prompt rule, a guardrail, a memory tweak). Re-run the suite. Did the target failure type shrink without breaking others? Log the delta.
3. 3Run the holdouts. Gap between tuning and holdout success rates → interpret honestly (overfit, or holdout drift?). This paragraph seeds your capstone eval report.