The Agentic Harness
FeaturesLong read

Replay-Based Eval Against Production Traces Before Shipping Changes

Test production agent failures by replaying recorded incidents with fixed code.

· 10 min read · Updated
Cover illustration for “Replay-Based Eval Against Production Traces Before Shipping Changes”
Features · September 15, 2026 · 10 min read · 2,284 words

An agent fails in production, an engineer pulls up the logs, re-runs the same workflow, and watches it behave differently the second time. This is the central obstacle replay-based evaluation exists to solve: agent failures in production cannot be reproduced by simply re-running the agent, because the conditions that produced the failure, the inference output, the tool state, the shape of the trajectory, have all shifted by the time anyone goes looking. Tools read external state that has since moved on, and retries and routing logic change how many times a given step fires, so the sequence that comes out of a re-run is rarely the sequence that actually broke. A failure in an agent system is almost never just one wrong output sitting by itself. It is a chain of model calls, tool calls, and routing decisions, and the standard engineering loop, reproduce the bug, fix it, run the test, assumes you can hold that chain still. Agent failures do not hold still, and the loop falls apart without infrastructure purpose-built to freeze them in place. Traditional logs cannot stand in for that infrastructure, because they cannot reproduce non-deterministic LLM failures, and the 2026 observability consensus treats span-based trace replay as a baseline requirement for any team running agents at scale, not an optional add-on for mature teams.

What tracing and scoring alone cannot do

Most teams running agents in production already have distributed tracing and some form of output evaluation, and both are necessary. Neither is enough on its own. Tracing spans that wrap LLM calls, tool invocations, retrievals, and internal reasoning steps let an engineer reconstruct what the agent actually did during a failed run, and without that reconstruction there is no starting point for debugging a multi-step failure at all. Evaluation frameworks add the next layer: they score whether a given output was acceptable. But a score is a verdict on what already happened. It says nothing about what would happen if one piece of the pipeline changed and everything else stayed fixed, which is the exact question an engineer needs answered before shipping a fix. Chronicle's framing of this gap is precise: tracing records, evaluation scores, but neither operation lets a developer isolate one layer of a recorded run and run new code against it while the rest of the incident stays frozen.

Either they guess at a fix based on reading the trace and ship it without a test that proves the guess was right, or they sit down and hand-write mocks meant to stand in for the real tool calls and model responses. Chronicle's own observation about the second path is blunt: mock-heavy tests can be worse at validating real interactions than no test at all, because the mocks are invented by the engineer rather than pulled from what the system actually did during the incident. A mock reflects what someone assumes a tool should return, not what it returned.

Even attempts to approximate the original workload by replaying at the level of sequence length, rather than full token content, run into trouble. The AgentReplay findings show that fixing sequence lengths while discarding the actual tokens still loses prefix-caching reuse and breaks mixture-of-experts routing fidelity, so the replayed workload is not a faithful stand-in for the original one. Getting replay right requires preserving the content that determined how the system actually routed and computed.

Recording at non-deterministic boundaries to turn a run into a testable artifact

The architectural move that closes this gap is to record at the points where non-determinism enters the system, rather than log every byte an agent touches. Each LLM inference call, each tool invocation, each routing decision becomes a boundary, and Chronicle captures the crossing of that boundary as an immutable envelope. A run recorded this way can be replayed in full with zero model calls reissued, and that full replay is bit-stable across repeated runs, with the same recorded trajectory coming back out every time it is replayed.

This only works because the overhead of recording is small enough not to distort the thing being recorded. Chronicle adds 23 microseconds per boundary crossing, a figure that is a rounding error next to the latency of a typical model call, so instrumenting an agent this way does not change how the agent behaves while it runs. The capture also does not require a separate step dedicated to generating test workloads. OTel-instrumented frameworks, which Chronicle supports for OpenAI- and Anthropic-style clients as well as any OTel-instrumented framework generally, emit these boundary traces as part of normal operation, so capture happens in production without a separate workload-generation step.

What comes out the other end is a committed record: the same inputs that triggered the incident, the same tool responses the agent actually received, the same context present at each boundary crossing. An engineer gets a fixture built from a real failure rather than one assembled from guesswork about what the failure probably looked like.

Cut-point replay: running new code against a recorded incident

The operation that turns this recorded artifact into something resembling a regression test is cut-point replay. The idea is to serve some recorded boundaries from the record itself while the rest run live, executed by the new code under test. A fix gets evaluated against the actual incident.

In practice, you choose which boundaries run live, typically the layer where the fix was actually made, and Chronicle serves everything else straight from the recorded envelope. Each boundary that gets served from the record, rather than executed live, carries a per-name call-count guard: Chronicle hands over the k-th recorded envelope for the k-th crossing of a given boundary name, so if the new code crosses that boundary more or fewer times than the original incident did, perhaps an extra loop iteration, perhaps a dropped retry, the lookup fails immediately and replay raises an error on the spot. The test does not pass silently because the code happened to call a tool one extra time in a way nobody noticed.

A benchmark built from 6 recorded failures shows what this buys. Cut-point replay tests fail correctly on faulty code and pass correctly on both guarded fixes and benign changes, across all 6 incidents. A baseline approach stubs every single boundary, using the same pass/fail assertion, and it catches none of the mutants that let an unsafe action through in that same benchmark. That contrast is a correctness finding: when every boundary in a test is stubbed, the new code never actually touches a real non-deterministic crossing, so a mutation that would let a dangerous action slip through in production passes the stubbed test without protest. The call-count guard is also what makes workflow loops detectable without someone manually inspecting the trace. If a workflow loops more times than the recorded incident did, the crossing count for that boundary diverges from what was recorded, and replay raises before the divergence has a chance to propagate into a passing test.

What a developer walks away with is a test file that exercises the precise failure that happened in production, passes once the fix is applied, and can be committed to the repository alongside the code. Once committed, the incident becomes a permanent member of the regression suite rather than a one-time post-mortem document.

The failure categories where replay provides detection that other methods miss

Replay-based detection earns its keep most clearly in the failure categories where a single-run evaluation or a static code review has no way to surface the problem.

Tool schema drift is the most concrete case. An agent can pass every happy-path evaluation in staging and still fail silently in production because one tool argument's name or shape changed somewhere upstream. There is no outage banner and no obvious stack trace, just a gradual decline in task completion and outputs that start to look slightly off in ways that are hard to pin down. Without tool-level instrumentation that captures the argument, the schema context, and the response in sequence, reconstructing this failure after the fact means manually piecing together execution traces by hand, which does not scale past a handful of incidents. Replay closes that gap directly: the recorded envelope holds the argument names present at the time the run succeeded, so when new code uses a changed argument name, replay fails at that exact boundary and attributes the failure to the tool layer before the change ever ships.

Prompt ambiguity collapse is harder to catch, precisely because it does not look like a failure in isolation. An agent facing an ambiguous instruction resolves it by silently picking one interpretation, and that interpretation can look entirely reasonable when you examine a single run on its own. Production traces are the only reliable evidence that this is happening at scale, because the pattern only becomes visible once many recorded runs are compared side by side. Replaying a suite of recorded traces against varying input phrasing exposes the inconsistency directly: the same underlying prompt produces different boundary crossings depending on how the input was worded, and that divergence becomes something an engineer can measure rather than something a user eventually complains about.

Workflow loops round out the third category. Infinite loops, state drift, and ambiguity about whether a task actually completed are operational failure modes that generic tracing tools miss entirely, since those tools typically trace LLM calls and stop there, never reaching the message-bus-level conditions that actually cause a loop to spin. Chronicle's call-count guard catches this class of failure directly at the boundary: an extra loop iteration changes the crossing count for that boundary, the lookup against the recorded envelope fails, and replay raises before the loop has a chance to burn through budget or leave the system's state corrupted.

Root cause attribution as the prerequisite for choosing the right cut point

Cut-point replay assumes the developer already knows which boundary to run live. That assumption is far from trivial. Deciding where the fix belongs, and so where the cut point should sit, depends on root-cause attribution, and that remains a hard problem in its own right on recorded agent traces. Chronicle treats attribution as a distinct line of work from replay itself: intervening in a recorded run by rewinding it, editing a crossing, or modifying a step in order to localize where a failure actually originated. That work informs where a cut point should be placed, but it is a separate exercise from the act of replaying.

Chronicle's own proposed remedy for this, called Continual Search, nudges a judging process across successive turns to keep searching for diagnostic evidence that has not yet been resolved, so it treats attribution as iterative rather than something decided in one pass. Separately, AgentDebugX, described in a 2026 paper (arXiv:2607.18754), operationalizes a closed loop with four stages: Detect identifies step-level errors, Attribute locates their root causes, Recover proposes a ranked set of fixes, and Rerun regenerates the trajectory around the point of failure. Cases that resist this loop escalate to a component called DeepDebug, and it produces an auditable root-cause report complete with supporting evidence and a recommended fix.

Attribution is not a solved prerequisite that simply precedes replay in a tidy sequence. Teams need an iterative process for figuring out where a failure actually originated, and the cut point chosen for a given replay test should be open to revision if the resulting test behaves in a way the engineer did not expect.

Wiring replay tests into CI so a regression blocks before it ships

A replay test run by hand on a developer's machine is useful for debugging a single incident. A replay test run automatically in continuous integration is a gate that stops a reintroduced failure from reaching users. Whether the regression is caught before it ships or after someone reports it depends on this.

Chronicle's stated design goal is exactly this: cut-point replay should produce a committed regression test that runs inside CI, and the 6-incident benchmark demonstrates this works in practice rather than as an aspiration, with correct verdicts delivered on faulty changes, guarded fixes, and benign changes alike. Two properties make this viable as an everyday CI gate rather than a slow, flaky integration check that teams learn to ignore. Full replay issues zero model calls, and it stays bit-stable across repeated runs. A test suite with those properties runs fast and gives the same answer every time, and that is what a CI pipeline needs from a gate it runs on every pull request.

The economics follow from the same properties. Replay suites built from many recorded traces run in minutes rather than hours, and they cost cents rather than dollars once real API calls are bypassed in favor of recorded envelopes. That cost structure favors building large replay suites rather than keeping them small and selective, because scale stops being expensive once you no longer have to reissue the model calls themselves.

Broader 2026 practitioner guidance points the same direction: automated quality thresholds tied directly to the CI/CD pipeline so a prompt regression blocks deployment before it ever reaches production, with prompt versioning and A/B testing under live traffic serving as the runtime complement to what replay catches before deployment. Moda, a production-trace-to-fix platform for AI agents that diagnoses harness failures and validates proposed changes against recorded production traces before they ship, is built around this same principle of treating replay against real traces as the confirmation step, not synthetic benchmarks. The 2026 evaluation stack, viewed as a whole, is commonly described in four layers: an offline benchmark, a CI regression layer, inline guardrails, and production observability. A regression caught at the CI layer never has to be caught by the layers that come after it.

Diagram: The Four-Layer 2026 Agent Evaluation Stack. Visualizes: Show a vertically stacked progression of the four evaluation layers that the 2026 practitioner consensus describes: (1) Offline Benchmark, (2) CI Regression Layer, (3) Inline…

Sources

  1. Top LLM Observability Tools in 2026: A Pro Guide
  2. Chronicle: Cut-Point Replay for Regression Testing of LLM Agents
  3. AgentReplay: Token-Wise Trace ReplayIs Essential for Fair Serving SystemPerformance Benchmarking
  4. AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents

More in Features