Gen AI

Agent Evaluation

Scores an agent by whether its steps were sensible and efficient along the way, not just whether the final answer happened to land on the right number.

Outcome evaluation only checks the last box in a run's step sequence, while trajectory evaluation checks the whole row, including the redundant, wasted calls hiding earlier in it

Why Does This Exist?

Take the travel-booking agent from AI Agent Architecture: find a flight and a hotel under budget, then report the total. Run it once, and it comes back with the right number. Ship it, right?

Pull up the transcript and it's messier. The agent called the flight-search tool five times in a row for no good reason, drifted $80 over budget for two steps, and only landed back under budget because a later search turned up a cheaper fare — not because it caught the problem itself. The final total is correct. Almost everything about how it got there says don't trust it again.

That gap is why agent evaluation is its own problem, separate from grading a chatbot's answer. Outcome evaluation — scoring only the final result — is cheap and the natural first thing anyone measures. But it's blind to how an agent got there, and for something taking multiple actions in sequence, that's exactly what predicts success next time. Trajectory evaluation closes that gap by scoring the run itself: the steps, their order, whether any were wasted.

Think of It Like This

Grading a parallel-parking test on the parked car alone

A driving examiner who only glances at the car once it's stopped can't tell a wildly bad parking job from a smooth one — both end up parked between the lines. An examiner watching the whole maneuver sees what the glance misses: two taps on the curb, backing up twice as far as needed, a correction that lined the car up right at the last second. That student passed this attempt. Nothing about it says they'll pass the next one.

Outcome evaluation is the glance at the parked car. Trajectory evaluation is watching the whole turn — the only view that tells a clean park apart from a lucky one.

How It Actually Works

Outcome evaluation: scoring where it landed

Outcome evaluation compares the agent's final output to a correct answer or a rubric, and stops there. For the travel-booking agent, that means checking one thing: does the reported total match the true cheapest total and respect the budget. Nothing about the steps in between enters the score.

That's cheap for a reason — one comparison per run, no logging of intermediate steps, and it maps directly onto what a user experiences. It's also usually the first thing anyone builds, since it's the metric a launch decision needs most. On the run above, outcome evaluation reports a clean pass. The number is right. That's the entire input to the score.

Trajectory evaluation: scoring how it got there

Trajectory evaluation scores the step sequence itself, not just where it ended: right tools, sensible order, no redundant calls, and a clean recovery instead of flailing when something breaks.

That requires logging every tool call, its arguments, and the result — not just the final message. Run that log against the travel-booking session and the picture changes. Five calls to the flight-search tool where one or two would have done. A stretch where the running total sat above budget with no visible attempt to fix it. Then, on the fifth search, a cheaper fare shows up and the total drops back under budget — a coincidence the agent had no way of engineering, since nothing in the log shows it reasoning about the earlier overage. Trajectory evaluation flags the redundant calls and the budget excursion, on a run outcome evaluation graded a clean success.

Why outcome-only scoring is incomplete for agents specifically

A correct final answer reached through a chaotic, redundant, or lucky path isn't evidence the agent works. It's evidence of a process that landed somewhere acceptable this once — and will very plausibly go wrong under slightly different conditions: a fare that doesn't drop on the fifth search, a hotel API down on retry, a budget ten dollars tighter. Outcome-only scoring can't see any of that, since it never looks past the last message. It can't tell a sound run from a mess that resolved in its favor, because both produce the same number.

This is where benchmark families come in: fixed suites of multi-step tasks, run the same way across agents or versions so scores stay comparable. They exist because one anecdotal run tells you little. But most still score outcome-heavy — did the agent complete the goal — unless built to also grade the trajectory. A high pass rate can describe a genuinely reliable agent, or one that gets lucky often on that suite. From the aggregate number, you can't tell which.

💡 Note That run is the argument for logging trajectories at all. Without the log, it and a clean, single-search success look identical.

Show Me the Code

1def redundant_call_count(steps: list[dict[str, object]]) -> int:2    """Count tool calls that repeat the same tool+args as an earlier step."""3    seen: set[str] = set()4    redundant = 05    for step in steps:6        key = f"{step['tool']}:{step['args']}"7        if key in seen:8            redundant += 19        seen.add(key)10    return redundant11
12
13def outcome_correct(final_total: float, budget: float, true_total: float) -> bool:14    return final_total == true_total and final_total <= budget15
16
17trajectory = [{"tool": "search_flights", "args": "TYO"}] * 518print(redundant_call_count(trajectory))           # -> 4 -- one original call, four repeats19print(outcome_correct(1180.0, 1200.0, 1180.0))    # -> True -- outcome eval sees only this

redundant_call_count catches the five-search pattern directly. outcome_correct only sees the final total — exactly why it reports success on the same run the first function flags as messy.

Watch Out For

Shipping on a passing outcome score without ever opening the step log

A team that measures only the final answer, and never inspects the tool-call sequence, has no way of knowing the travel-booking agent's "successful" run involved five redundant searches and a budget breach fixed by luck, not by the agent noticing anything. The outcome score looked fine. The process it hid wasn't, and it'll very plausibly fail differently next run.

Treating a benchmark's aggregate pass rate as proof the agent is reliable

Most benchmark families score mostly on whether each task's goal was completed, not whether the steps were sound. A high pass rate says the agent reached acceptable endpoints often enough on that suite — nothing about how many passes involved wasted retries or lucky paths, unless the benchmark grades trajectories too. Treat an outcome-heavy pass rate as a floor, not a guarantee.

The Quick Version

Outcome evaluation scores only the final answer against a correct result or a rubric — cheap, and usually the first thing anyone measures.
Trajectory evaluation scores the full step sequence: right tools, sensible order, no redundant calls, clean recovery from errors.
A correct final answer reached through a chaotic or lucky path is a sign of an unreliable process, not a working one — it can easily fail differently next time.
Outcome-only scoring can't see that risk at all, because it never looks past the last step.
Benchmark families are fixed, repeatable task suites for comparing agents over time, but most still score outcome-heavy unless they explicitly grade trajectories.