LLM-as-Judge Eval Design for Agentic Task Outcomes
Evaluating agents requires inspecting execution traces, not just final outputs.

An agent can plan sensibly, call five tools in a reasonable order, reason through intermediate steps that look sound, and still fail the task, while the final response reads as competent and a judge scores it accordingly. An agent is a different kind of object. It plans, calls tools, and reasons across many steps, and the final response is only the last artifact of that sequence, not a summary of it. Confident AI's agent evaluation guide frames the actual evaluation surface as four distinct areas: tool calling, planning, task completion, and reasoning, and none of those four areas is visible from the terminal output alone. DAREBench names this directly as a structural reliability problem, stating that an LLM judge may incorrectly award positive credit when tools were not executed, the task timed out, or required artifacts were missing.
The most reliable approach evaluates the full execution trace of an agent run, not the terminal output alone. Confident AI's framework identifies four distinct evaluation surfaces that are invisible from the final response: tool calling, planning, task completion, and reasoning. Each span in the trace, from the initial input through the plan, retrieval, tool arguments, tool output, and intermediate reasoning, is a separate site of potential failure that requires its own judge, whether deterministic for exact facts like tool selection or LLM-based for contextual questions like argument correctness. Closing the loop requires three layers running together: unit evals on discrete steps, LLM-as-judge regression suites, and continuous sampling of live production traces with score-based alerting so that new failure modes are caught while they are still isolated.
Four properties of agentic systems make the problem worse than it first appears. They can correctly detect that something broke, but they cannot localize which component broke.
The failure modes that terminal-output judges cannot see
Agent failures fall into five broad categories: planning errors, tool errors, retrieval errors, reasoning errors, and safety or policy violations, and the most dangerous instances in each category produce a final output that looks entirely plausible while concealing the break that produced it. Two of these failure modes deserve particular attention because they are the least intuitive and the most likely to pass an output-only review undetected.
Tool schema drift is the first. The final output can still read as reasonable, because the breakage happened in the translation layer between tool and consumer, not in the visible text.
Workflow structural failures are the second, and they account for the majority of execution failures before the underlying model is ever the actual issue. If you fix these three patterns, you clear out most execution failures, so put engineering attention there before you chase more exotic failure modes.
The remaining categories are narrower but no less real. Pioneer Agent (2025) finds emergent failures in multi-agent coordination, consensus loops and hallucinated communication protocols among them, and you can only see these in the span-level trace of handoffs, never in a single terminal output. Standard application performance monitoring tools compound the problem: they cannot surface tool-call failures, context truncation, or runaway loops without instrumentation built specifically for agents, so the failure is invisible at the infrastructure layer at the exact moment it is invisible to a terminal-output judge.
What execution evidence a judge needs to see
Catching the failure modes above requires inspecting the ordered trace of a run, not just its terminal artifact: the input, the plan, the retrieval step, the tool arguments, the tool output, the intermediate reasoning, and finally the answer. Each span in that sequence is a distinct site where something can go wrong, and each carries its own evidence of whether it did.
The eval surface maps cleanly onto four areas. Tool calling asks: was the right tool invoked with the right arguments? Reasoning asks whether each intermediate decision followed logically from the task, the available context, and the results tools had already returned.
Confident AI's guide breaks this into specific metrics: Task Completion, Step Efficiency, Tool Correctness, Argument Correctness, Plan Adherence, and Reasoning Relevancy and Coherence, each tied to a specific span in the trace rather than to the output as a whole. This is the practical checklist a team instruments against: capture the input, capture the plan the agent commits to, capture every tool call with its arguments and its raw response, capture the reasoning that connects tool output to the next decision, and capture the final answer separately from all of it.
Debugging a multi-agent failure means walking that span tree, from input through plan, retrieval, tool arguments, tool output, any intermediate rewrite, and finally the answer, scoring each span individually to find the one that broke. A single end-to-end score cannot do this work, because it cannot tell you which span broke. For systems built from multiple cooperating agents, the span tree has to extend further still, covering what one sub-agent handed to another, whether the receiving agent built its plan on accurate input, and whether the orchestrator assembled the final result faithfully from what its sub-agents actually returned. The trace, captured at this granularity, is the substrate that any serious judge has to work from. What remains is deciding how to judge each piece of it.
Matching the judging strategy to the type of evidence
Not every span in a trace calls for the same kind of judgment, and the right technique follows directly from the kind of evidence a given span produces. Deterministic checks belong wherever the fact in question is exact: an LLM judge belongs wherever the question is semantic or depends on context; trajectory-level evaluation belongs wherever the sequence of decisions, not any single decision, is what's being assessed.
Tool correctness, whether the agent called the right tool at all, is a deterministic fact and does not need a language model's opinion to resolve. Argument correctness is a different matter: whether the arguments passed to that tool were the right arguments depends on what the task actually required, not merely on whether the expected fields were populated, and that judgment is contextual enough to warrant an LLM judge. Reasoning relevancy and plan adherence form a third, distinct level. You need to see the ordered sequence of decisions, not an isolated snapshot, to tell whether a given step followed sensibly from prior context and prior tool results, so these metrics demand trajectory-level evaluation rather than a single-span check.
DeepEval's guide to LLM-as-judge formalizes three techniques suited to these different evidence types: G-Eval, for custom subjective criteria applied to a single output; DAG, for deterministic branching logic across multiple conditions; and QAG-style metrics, which decompose a judgment into a series of closed-ended questions. You choose among them per span, as an engineering decision specific to that span's evidence. Pairwise comparison is a separate and useful pattern: it asks which of two prompt versions or tool schema versions performed better on the identical trace. You reach for this specifically to validate a proposed harness change, not to produce an absolute quality score.
For the longest trajectories, a single static judge model starts to strain. The agent-as-a-judge framework extends the idea by giving the evaluator its own tool use, memory, and multi-step reasoning, so it can observe and critique another agent's run step by step rather than reading a static transcript after the fact. That architecture is the appropriate one for long autonomous trajectories, where the judge needs the same operational depth as the system it is evaluating. LLM-as-judge has known failure modes of its own, inconsistency across runs and sensitivity to prompt phrasing among them, but you fix those with better judge design along the lines above, not by retreating to output-only scoring.
Why static golden sets miss production failure modes
A harness change ships. An offline evaluation set is a snapshot of the failure modes a team knew about at the moment the set was built. Production keeps generating new failure modes, but a judge that only ever runs against frozen traces has no way to catch them.
Three layers of evaluation are needed to close this gap: unit evals on discrete steps, LLM-as-judge regression suites for subjective output quality, and continuous sampling of live production traces to catch drift as it actually happens. The third layer is the one most teams skip. Alerting should fire on drops in evaluation score, not only on latency spikes, since a system can get slower in failure mode just as easily as it can fail silently at full speed.
The sources of this drift are rarely the underlying model. Model updates do contribute, but prompt edits and tool schema changes cause silent drift just as often, and all three are harness-layer changes, so they can break previously reliable behavior even while the foundation model itself stays frozen. So you run LLM-as-judge asynchronously against a sampled slice of live production traces, and you alert when rolling quality scores fall below a set threshold, so you catch new failure modes while they are still isolated rather than after they have accumulated. A score that looks stable overall can mask a regression concentrated in one task type or modality, and per-flow drift detection exists to surface exactly that kind of regression.
Replay validation as the bridge from observed failure to verified fix
Once you have found a production failure and localized it to a specific span, you still need to prove that a proposed fix resolves it without breaking something else. Re-running the agent live against the same inputs does not answer this cleanly: live re-execution reintroduces the same non-determinism that made the original failure difficult to isolate. Replay testing solves this by holding everything constant except the one variable under test.
The method starts with capture: every user message, every tool call, every tool response from a real production run, stored as a deterministic fixture. When the trace is replayed, the engine plays back the exact same tool responses rather than hitting live APIs, so the only thing that can differ between the original failing run and the replayed run is the harness change being tested. Scheduled replay of a golden set of these traces against the current production configuration is the most reliable early warning available for exactly the three silent-drift sources named above, model updates, prompt edits, and tool schema changes, any of which can break a trace that previously passed without anyone touching the trace itself.
Sizing the golden set depends on system complexity. A single-purpose agent can be served reasonably well by a reviewed set of traces revalidated monthly, enough to catch most regressions without becoming a maintenance burden of its own. A multi-agent system built around an orchestrator and several sub-agents needs considerably more: enough traces per sub-agent to exercise each one's behavior individually, plus a separate set of end-to-end trajectories covering the full handoff chain. These figures are guidelines calibrated to how complex the system actually is.
Attribution inside replay is its own discipline. AgenTracer is the first automated framework for annotating failed multi-agent trajectories through counterfactual replay and programmed fault injection, and you build it to pinpoint the agent or step responsible for an error inside long execution traces (describing the attribution mechanism this section depends on). This operationalizes span-level attribution directly inside the replay workflow, turning a vague sense that "something broke" into a specific, reviewable claim about which span broke and why.
Most of the time, the harness, not the model, is the right target for the fix. Correcting the three structural patterns named earlier, no exit condition, no fallback handling, no format specification, eliminates the large majority of execution failures before the underlying model is touched at all, and replay lets each of those fixes be validated against historical evidence before it ships. The research backs the harness-first instinct over the reflexive instinct to retrain. The Pioneer Agent work from 2026 found that naive retraining can degrade performance on AdaptFT-Bench by as much as 43 points, while targeted, diagnosis-driven improvement preserves or even improves performance on the same benchmark. Live re-execution cannot give you a reviewable, inspectable diff between the broken trace and the fixed trace, with the harness change isolated as the only variable between them. That diff is what it means to ship a fix with evidence behind it, rather than shipping it on faith that the change probably helped.
How do you build a production eval loop that keeps improving over time?
Instrumenting traces, running span-level judges, sampling production continuously, maintaining a replay-validated golden set, and feeding failing traces back into the regression suite are not five separate projects. They form one loop, and the value of that loop compounds as the trace corpus it runs against grows larger and more representative of what the system actually faces in production.
This loop has recently been given a formal name and structure: Harness Continual Learning, in which the harness evolves around a frozen foundation model through a repeating cycle of candidate generation and evidence-gated commitment. A Continual Optimizer proposes candidate harness changes based on post-execution feedback, and a Continual Evaluator only commits a candidate after it checks that candidate's current improvement, its retention of prior fixes, and its overall validity. The structure matters because it separates the act of proposing a change from the act of accepting it, the same separation of concerns that code review and staged deployment enforce in ordinary software engineering.
A guarded version of the same discipline applies to the eval rubric itself, not just to the harness it evaluates. You validate a proposed rubric change against the existing golden set before you adopt it, and it earns adoption only when it improves attribution of failures without losing coverage of failure modes the team already knows about. If you close the loop fully, production traces that fail a replay or fail live sampling get added automatically to the offline dataset the next regression suite is built from, so the set under test keeps growing to match what production has started throwing at the system. Production finds the next failure mode the golden set didn't anticipate, and the loop has to make sure that failure mode never gets the chance to repeat unnoticed.


