Agent benchmarks

You can evaluate a prompt on a golden set of input→output pairs. Agents break that model, because **there's no single right output — there's a right outcome, reachable by many trajectories, some good and some lucky.** Agent evaluation is the hardest measurement problem in this whole track, and doing it at all puts you ahead of most teams shipping agents today.

Two things to measure, and you need both

• Outcome: did the task get resolved correctly? For the resolution agent: right decision, right refund amount, ticket updated, appropriate escalation. Checkable against a gold outcome — the closest thing to your familiar eval.
• Process (the trajectory): how did it get there? Did it check policy before deciding, or decide then rationalize? Waste 15 turns on a 4-turn task? Take a risky action that happened to work? A right outcome via a reckless process is a latent failure — it'll produce a wrong outcome the next time luck runs out.
• Why both are mandatory: outcome-only eval blesses lucky recklessness; process-only eval rewards beautiful trajectories that didn't solve the problem. The dangerous quadrant — right outcome, bad process — is invisible unless you measure the trajectory.

The agent scorecard

• Task success rate — % of tasks with the correct outcome (your headline number). Define 'correct' precisely per task in the gold spec.
• Process quality — a rubric score over the trajectory: policy-before-action, no unnecessary tools, appropriate escalation, no wasted turns. Judge-scored, calibrated like every judge you've built.
• Safety — did any invalid action get requested (caught by validators, but a signal), and did any wrong action execute (should be zero — a single one is a P0, not a percentage).
• Efficiency — turns per task, tokens per task, cost per task, tool calls per task. Agents are expensive; this is where cost lives.
• Escalation quality — did it escalate when it should (recall) without escalating everything (precision)? An agent that escalates all cases is safe and useless.

The task suite

Your eval set is now task specifications, not input/output pairs: an initial state (ticket + world), a gold outcome, and required/forbidden actions ('must check policy'; 'must not refund over limit'; 'must escalate'). Archetypes to cover: straightforward resolutions, multi-step cases, ambiguous ones that should escalate, unanswerable ones, and adversarial ones (injection, over-limit pressure). Same governance as always — holdouts, production cases folded in, judgment calls logged. The set is harder to build than a prompt golden set; it's also the thing that lets you change the agent without fear.