Replay-Based Eval Against Production Traces Before Shipping Changes

Replay-based eval means re-running an agent's evaluation using pre-recorded inputs and tool responses, without making a single new live call. Nothing hits a real API. Nothing depends on a vector database returning the same result twice in a row, because it won't, and that's exactly the problem replay solves.

A production trace, at minimum, captures the user's messages, every tool call along with its exact arguments, every tool response, the context retrieved at each step, the final output, and, where the framework supports it, intermediate reasoning steps. Put together, that's a full record of what the agent actually did, not a guess at what it probably did.

That stored trace becomes a fixture. The replay engine plays back the exact tool responses that were recorded, bypassing the real APIs entirely, so the agent "runs" against a recording instead of the live world. Mechanically, this comes down to a small number of components: a function that loads the recorded trace into memory, and stub clients, a replay tool client and a replay LLM client, that intercept what would normally be real calls and answer them from the trace instead.

The payoff is concrete. A user files a bug report on a Tuesday. The debugging happens on Wednesday, against the user's exact session, with the exact same inputs the user actually sent, not a reconstruction built from a support ticket and a guess. Agents are dynamic enough that a "close enough" reproduction just isn't close enough, and teams that settle for close enough end up chasing the wrong fix, patching a symptom that was never the cause.

The unit of evaluation isn't the completion. Treating it as one is where most replay setups go wrong from the start. An agent run is a tree of observations, LLM calls, tool executions, retrievals, and different quality questions attach to different nodes in that tree. A check on whether the arguments to a tool call were valid belongs on that tool-call observation. A check on whether the task got done belongs at the root of the trace. Collapsing all of that into one pass/fail number throws away the exact information needed to fix anything, which is the real cost of treating a trace as a single blob instead of a tree.

On the infrastructure side, OpenTelemetry has become the de facto, vendor-neutral standard for AI telemetry, even though its GenAI semantic conventions are still officially in Development status as of 2026. Any framework that emits OTel-compatible GenAI spans, LangChain and LlamaIndex among them, produces traces usable for replay, provided content capture is explicitly turned on and the replay tooling is configured to read it. That "provided" is doing real work: plenty of teams have OTel wired up and still can't replay anything, because content capture was left off by default and nobody noticed until the day they needed it.

For conversational agents specifically, the session, not the trace, is the unit users actually experience. Five turns can each be individually defensible, reasonable answers given reasonable context, and still add up to a conversation that never resolves the user's actual problem. That kind of failure is invisible if the eval only ever looks at one trace at a time.

The offline/online split and where replay sits in the full eval loop

Two evaluation modes exist, and they catch different failures. Neither substitutes for the other. Picking one and skipping the other isn't a tradeoff worth making, it's a gap in coverage dressed up as a decision. Offline eval runs the agent against a fixed dataset before a change ships. It catches regressions from a prompt edit, a model swap, or a tool change, usually against curated data with expected outputs attached, and its cost is bounded because the dataset size is fixed and known. Online eval scores a sample of real production traffic after the change has already shipped. It catches drift, novel input the offline set never anticipated, and tool failures that only show up under real-world load, using real user inputs that come with no ground truth attached. Its cost is controlled by the sampling rate, not by dataset size.

The canonical loop connecting the two runs like this: collect failing production traces, turn them into dataset items, reproduce the failure inside an experiment, fix it, then keep that item permanently as a regression test. Replay is what makes the reproduction step actually work. Without it, "reproduce the failure" is an approximation dressed up as a repro, and approximations are exactly what let regressions slip back in six weeks later under a slightly different name.

Cadence matters here as much as method. On every pull request, the full replay regression suite should run against a set of golden cases, blocking the merge if any regression metric drops below its threshold. Keeping that check fast enough that nobody skips it is the entire point. Model version updates get a different, heavier cadence: an extended eval suite, because a model update can introduce behavioral drift across task types in ways a fast PR check isn't built to catch.

Online sampling doesn't try to score everything, and it shouldn't try. A fraction of live sessions gets pulled and run through automated eval close to real time, enough to catch an emerging pattern before it compounds into a wider outage of trust, not so much that the pipeline drowns in its own data.

None of this holds together unless production traces and experiment runs share the same underlying schema. If a score means something slightly different in the offline dataset than it does in the live sampling pipeline, the two numbers can't be compared. And the whole point of closing the loop, catching in the next PR what broke in production last week, falls apart the moment that comparison stops meaning anything.

Why failures surface where they do, and why that location is almost never the real cause

The step where an error becomes visible is often not the step that caused it. That gap is where most debugging effort gets wasted: chasing the most error-dense part of a trace is chasing the wrong thing, full stop. Research on agent debugging (arXiv:2607.18754) found that one root error tends to trigger further violated checks downstream, and a large majority of failures end up violating more than one check by the time anyone notices. A heuristic that just flags the messiest region of a trace can't tell the difference between the step that caused the failure and the steps that merely inherited it. That distinction needs causal tracing along the actual information flow, not a scan for wherever things look worst.

AgentTether (arXiv:2607.06273) is one architecture built around exactly that distinction: graph-guided root-cause diagnosis that follows dependency edges and information flow through the trace, rather than treating trace position as a proxy for causation.

The taxonomy work backs this up from a different angle, and it points at the model less than most people expect. Research tracked across tested models found that hallucination in interpretation accounted for 71.2% of failures and incomplete exploration for 63.9%, and both held steady across every model tested, regardless of that model's general capability tier. Swapping in a stronger model doesn't fix either one. Separately, UC Berkeley's MAST taxonomy attributes 41.8% of failures to specification issues and 36.9% to inter-agent misalignment, categories that live in the harness and the coordination protocol, not in the model's weights. MAST has been applied across more than 1,600 execution traces, with the taxonomy itself validated on 150 traces at NeurIPS 2025, and it maps 14 distinct failure modes back to three root categories: specification issues, inter-agent misalignment, and task verification. None of the three is a model generation problem. The fix rarely lives where teams first go looking for it, which is the model: teams that respond to agent failures by upgrading the model are treating a harness problem as a capability problem.

Most of these failures are deterministic given the inputs. They recur, reliably, whenever the same conditions show up again. Replay is how a team confirms a fix addresses the actual condition rather than just quieting the symptom once.

AgentDebugX (arXiv:2607.18754, July 2026) addresses the full arc of agent debugging, from detecting an error through attributing its cause, recovering from it, and rerunning to confirm the fix. Its DeepDebug method hits 28.8% strict attribution accuracy on the Who&When benchmark and still lifts overall GAIA task accuracy to 63.6% through a single rerun. Read that pairing carefully: even with attribution accuracy well under a third on strict scoring, a rerun still produces a meaningful accuracy gain. That gap is precisely why the rerun step, replay in production terms, isn't optional. Attribution is imperfect. Replay is how the imperfection gets caught anyway.

The specific failure categories that replay catches and synthetic tests miss

Tool schema drift is the quiet one, and it's the one worth worrying about most, because the loud version of this failure isn't the dangerous one. A schema mismatch, wrong type, missing field, throws a runtime error, which means someone notices right away. A description mismatch is worse: the tool's interface is technically fine, but its description no longer matches what it actually does, so the model calls it in situations where it shouldn't, or skips it in situations where it should. No error fires. Nothing in a type checker or a unit test catches it, because both check that the tool call is well-formed, not that the tool's actual behavior matches what the model was told to expect. Tool calling already fails at a meaningful rate in production, varying by model size and task complexity, and some workflows fail well above that baseline, at rates nobody sees until traces make them visible. Tool count plays into this too: a focused set of tools tends to outperform a much larger menu of overlapping ones, because the model is less likely to confuse them. That's a harness design choice, not a model capability limit, and most teams get it backwards, adding tools when accuracy drops instead of cutting the ones that overlap.

Prompt drift is a related but distinct failure. It shows up when an LLM starts producing subtly worse output even though nobody touched the prompt, usually because a model update, a safety-tuning pass, an alignment adjustment, interacts with prompt wording in a way nobody predicted. In a single-turn chatbot, that's a UX annoyance. In an agentic system, where a whole network of prompts coordinates each stage of a multi-step workflow, it's a systems-engineering problem. A model that slightly misreads user intent might pick the wrong tool. A model that misreads a tool's error message might loop forever, retrying a call that will never succeed. Both produce the same visible symptom, a wrong final answer, from entirely different root causes. The only way to catch drift at all is to compare current behavior against a known-good baseline, which is what a replay regression suite is, functionally: a permanent, growing set of known-good baselines.

Workflow loops and coordination failures round out the list, and they're arguably the costliest of the three because they hide in the budget rather than the output. Multi-agent systems fail at high rates in production, at substantial rates depending on the setup, per the MAST research, largely because specification issues and inter-agent misalignment, the two categories MAST identifies, let agents misread their own roles, duplicate each other's work, or skip verification steps. A loop that keeps retrying the same failing call runs silently without throwing an error. It just burns tokens and time, quietly, until someone notices the bill. Cost per task becomes the actual signal there, not error rate, because error rate stays at zero the whole time the loop runs. These patterns are, again, reproducible given the same trace inputs, which is exactly why replay can expose them and then confirm a fix stops the loop rather than just changing its shape.

None of these three produce a clean error signal at the output layer. Catching any of them requires looking at the execution path itself, and the execution path only exists to look at if it was captured as a trace in the first place.

What you need to measure across a replay eval run

Agent quality breaks down into four separate dimensions, and treating them as one aggregate score is the single most common mistake teams make here. A blended number can tell someone that things got worse. It can never tell them where, and a metric that can't localize a failure isn't doing the job a metric is for. Trajectory asks whether the agent took a sensible path: step count, unnecessary tool calls, loops and retries, whether required steps were present and in the right order. Tool use asks whether it called the right tools, correctly: right tool selected, valid arguments, how often tool calls errored out, and whether the agent recovered when they did. Task completion asks the most basic question, did the user actually get what they asked for, measured through goal achievement, answer correctness, resolution rate. Multi-turn quality asks whether performance holds up across an entire conversation: context retention across turns, goal drift, turns to resolution, outcome at the session level rather than the turn level.

These four can fail in different ways and at different points in a run, which is the whole reason to track them separately. An agent can complete a task correctly while taking a wasteful path, twelve tool calls where two would do, which shows up as a latency and cost problem rather than a correctness one. Another agent can execute a clean, efficient trajectory and still fail the task, because the final synthesis step got the answer wrong despite every step leading up to it being sound.

Tool-call correctness is the dimension teams skip most often, and it's the one they come to regret most. A large share of confidently-wrong behavior originates right there, and it stays invisible in the final answer whenever the agent happens to recover from the bad call downstream. The output looks fine. The path that produced it wasn't, and nobody checked the path, because checking the output is easier and feels like enough until it isn't.

Cost per task deserves its own line item as a failure signal, not just a line on a budget spreadsheet. An agent that loops an extra two or three times before landing on the same answer it could have reached directly is correct, by any accuracy metric. It's shipping a margin problem instead, one that accuracy metrics have no way to see, which is exactly why cost has to be tracked as a first-class signal rather than an afterthought pulled from the finance team's dashboard.

Three layers of evaluation method, used together rather than as alternatives, cover this ground. Code evaluators are deterministic and cheap, running in milliseconds at no marginal cost: did the required tool get called, do the arguments parse against the schema, was the step budget respected, is the output valid JSON. LLM-as-judge handles semantic judgment at scale, things like task completion without a ground-truth answer to check against, reasoning quality, whether a claim is grounded in retrieved context, and it needs to be calibrated against human review rather than trusted on its own. Human annotation supplies the ground truth itself: labeling the genuinely ambiguous trajectories, building the reference set the LLM judge gets calibrated against in the first place.

Separating the agent that generates output from the agent that evaluates it beats letting a system grade its own work, a distinction Anthropic's research has been explicit about. Agents that self-evaluate skew positive, reliably. That's a systemic bias, not a bug that gets patched later, and the correction has to be structural: build the generator and the evaluator as genuinely separate processes from day one, not as a single system asked to mark its own homework.

How tooling connects production traces to pre-ship regression tests

The mechanics above only matter if the tooling actually closes the loop between a production failure and a permanent test. A few capabilities separate tools that do this well from ones that just log traces and stop there.

The fastest loops let someone click directly on a failed trace and convert it into a test case that runs in CI on the very next pull request, collapsing the distance between a production incident and a permanent regression test down to minutes rather than the days it takes when someone has to hand-write a repro from a bug report. Good trace logging captures LLM duration, time to first token, prompt tokens, cached tokens, completion tokens, reasoning tokens, estimated cost, tool calls, and errors automatically, without requiring custom instrumentation for each one. A timeline replay view, showing exactly when each operation started, how long it ran, and what it returned, turns a multi-step agent run with a dozen tool calls into something a person can actually step through, rather than a wall of JSON to parse by hand.

On the CI side, native integration that posts eval results directly onto a pull request keeps the check where engineers already look, instead of adding a separate dashboard nobody opens. Ranking failure modes across all of production traffic, rather than one trace at a time, is what turns an isolated bug report into a pattern worth fixing at the harness level. And enabling all of it, the query engine has to stay responsive across millions of stored traces, because a regression suite that takes ten minutes to search isn't one anyone will run before every merge.

The gap between benchmark performance and production reliability is a systems problem, not a model problem, and no amount of model quality closes it on its own. Most teams still find that out the hard way, in production, after the fact, having spent the upgrade budget on the wrong layer. Replay-based eval, tied into a CI loop with real production traces feeding it, is what turns that discovery from something painful into something routine.