Trajectory analysis
Reading the sequence of decisions: the failure taxonomy, LLM-as-judge over trajectories, and turning failure patterns into fixes.
The trajectory — the full sequence of reasoning, tool calls, results, and actions — is the agent's work laid bare, and learning to read it is the core diagnostic skill of this course. A wrong outcome tells you that it failed; the trajectory tells you where and why, which is the only thing you can act on.
The agent failure taxonomy (know where to look)
- Planning failures — wrong approach from the start, or a stale plan executed into a wall. Visible early in the trajectory.
- Tool failures — wrong tool chosen, right tool with wrong arguments, or a tool error the agent didn't recover from. Look at the tool_use blocks and the results.
- Reasoning failures — misread a fact, misapplied policy, correct reasoning that the action didn't follow (the reason–action mismatch from M2).
- Memory failures — forgot a request, re-fetched known facts, acted on stale memory (Module 4's symptoms).
- Termination failures — stopped too early (task incomplete), or never stopped (hit the turn limit wandering).
- Safety failures — requested a forbidden action, took a risky path. Even when validators caught it, a request is a finding.
Judging trajectories at scale
Reading every trajectory by hand doesn't scale past a few dozen. An LLM judge over trajectories does — given the task spec, the trajectory, and a rubric, it scores process quality and classifies the failure type on the misses. Same discipline as every judge you've built: anchored rubric, calibrate against your own hand-labels on ~10 trajectories, use for screening and triage, read the extremes yourself.
You are an agent evaluator. Given this task spec and trajectory, assess: (1) OUTCOME — did it achieve the gold outcome? (2) PROCESS 1-5 — policy-before-action, tool efficiency, appropriate escalation, no wasted turns; justify. (3) FAILURE TYPE if any — planning / tool / reasoning / memory / termination / safety — with the exact step where it went wrong. (4) The single highest-value fix. Task: [spec]. Trajectory: [full log].
This judge is your triage engine: run it over a batch, group misses by failure type, and the biggest bucket is your next sprint. Failure-type distribution turns a pile of bad runs into a prioritized backlog.
From patterns to fixes
The payoff of classification is that each failure type has a characteristic fix, so diagnosis points straight at the remedy: planning failures → prompt/plan-artifact changes; tool failures → better tool descriptions or error messages; reasoning failures → reasoning-depth steering or a validator; memory failures → context management; termination → clearer done-criteria or turn budgets; safety → guardrails, always guardrails. The distribution is your roadmap — fix the biggest bucket, re-measure, repeat.
The trajectory that reaches the right outcome via a wrong path is the one that will bite you in production, and it's invisible to outcome-only eval. Always sample successful trajectories in your review, not just failures — some of your 'passes' are latent failures wearing a green checkmark.