The Agentic Harness

Replay-Based Regression Loops for Prompt Edits Across Real User Trace Segments

Production traces, not synthetic tests, expose the exact conditions that made agents fail.

· 8 min read
Cover illustration for “Replay-Based Regression Loops for Prompt Edits Across Real User Trace Segments”
Eval & Validation · October 29, 2026 · 8 min read · 1,793 words

Before shipping an agent change, validate it by replaying the production trace segments that exposed the failure, with the changed component running live and unrelated boundaries supplied from the record. Replay-based regression loops for prompt edits make real user trajectories the authoritative test corpus. Synthetic benchmarks remain useful for breadth, but they cannot establish whether a change fixes the exact sequence that failed in production.

Why re-running a failed agent trace does not reproduce the failure

On turn four, an agent calls issue_refund when check_status is required. The trace shows the wrong action, but a re-run takes another path. The failure belongs to a trajectory because earlier model calls, tool responses, and routing decisions established the conditions for that choice. Sending the same request again does not bring back those conditions.

Chronicle (Chawla & Koul, 2026) identifies inference that is not bitwise reproducible, even at temperature zero, as one source of divergence. Tools also read external state that may have changed, while retries and routing logic alter how often a step executes. Identical code can therefore pass or fail across re-executions. For multi-step agents that use live tools and non-deterministic inference, re-running a recorded incident alone cannot make it a repeatable test.

Why synthetic test cases cannot substitute for the original trace

Writing an approximate test is a tempting shortcut. A synthetic case starts from a description of expected behavior, while the production failure may depend on something that description never anticipated. Unusual phrasing, rare context, an unexpected tool result, or a multi-turn history can determine which action the agent takes. Reconstructing the apparent intent of the conversation does not preserve that causal sequence.

Synthetic coverage still serves a necessary purpose. It explores imagined cases beyond the incidents already observed, whereas replay tests the specific bug that real traffic exposed. These are complementary forms of evidence. Once an actual interaction fails, passing an approximation leaves the original failure untested.

How record-and-replay converts a production incident into a regression test

Chronicle records runs at non-deterministic boundaries as immutable envelopes, then supplies those recorded outputs during replay. The record replaces the changing sources of execution. Full replay makes zero model calls and is bit-stable from run to run, so it can reproduce the failed trajectory without live inference having to produce it again. Chronicle measures recording overhead at 23 µs per boundary crossing, negligible beside a typical model call’s latency.

Tooling that traces or scores a run helps establish what happened or whether an output was acceptable. Replay supports testing a code change against the incident while selected conditions remain fixed.

A saved scenario can then become a permanent regression test. Its assertions identify whether a later change allows the same failure to return. The suite grows through failures production has already demonstrated, rather than relying entirely on cases an engineer can anticipate.

Cut-point replay: testing a specific fix without re-running the whole trace live

Cut-point replay tests a fix without having to re-run the whole trace live. It serves a chosen subset of boundaries from the record and executes the complementary subset with new code. The engineer can therefore expose the changed component to the conditions surrounding the original failure. Unrelated variability stays outside the test.

For a prompt edit, the prompt boundary runs live with the new prompt while non-target model calls, tool responses, and routing decisions come from the recorded trajectory. Freezing the component under test would preserve its old behavior rather than evaluate the change, so choosing the cut point is part of the test’s design. The resulting regression can run in continuous integration against the past incident.

Chronicle’s benchmark uses six recorded failures in which the model boundaries are simulated. Its cut-point tests fail on faulty code and pass on guarded and benign changes for all six incidents. In its mutation study, the tests catch every mutant that lets the recorded unsafe action through. A baseline that stubs every boundary with the same assertion catches none. Cone-bounded partial replay runs only the affected segment and uses fewer tokens than full-run replay, which makes per-PR testing more practical.

Diagram: Cut-Point Replay: How One Boundary Runs Live While the Rest Stay Frozen. Visualizes: Illustrate the cut-point replay mechanism as a linear agent trace of six or so boundary crossings (model calls, tool responses, routing decisions).

Choosing which trace segments to replay: the failure categories that make segments non-interchangeable

For tool schema drift, the test needs a segment that used the tool before its schema changed. Renaming customer_id to account_id, for example, can turn staging success into degraded task completion and additional retries without a stack trace. More complex or constrained fields can also lead the model to avoid the tool or choose a structurally easier alternative. Replaying that segment tests whether the prompt or schema fix preserves the intended action under the changed contract.

A failure-mode analysis in arXiv:2604.21816 found that 48% of failures in one tool-attention system came from ambiguous queries matching multiple semantically similar tools. Poorly written tool descriptions accounted for a substantial fraction as well. The relevant segment contains the query and context where routing actually went wrong. An example with similar tools does not keep that same query and context.

Workflow loops have their own distinct trace signature. The agent repeats the same tool, target, and arguments until it exhausts its turn budget. A fix for that behavior needs a segment that captured the repetition.

Root cause attribution determines which layer the replay should isolate

Choosing the failure segment still leaves the choice of which component should run live. A wrong tool call might originate in a prompt, schema, workflow, model, or product rule, and each attribution directs a different intervention. The cut point follows that causal judgment. If the wrong layer is exposed, the replay can test a change without testing the source of the failure.

Attribution requires searching the trajectory. Earlier states condition later decisions, and relevant evidence can be sparse or distributed across distant steps. LLM attributors can also anchor on a plausible explanation before exploring the full record. AgentDebugX (2026) addresses that problem through Detect, Attribute, Recover, and Rerun, with its DeepDebug method improving exact agent-and-step attribution on the Who&When benchmark over the strongest single-pass baseline.

HPFA (COLM 2026) compares the hyperedges of a failed reasoning path against a successful reference path to reduce the attribution search space. Prompt, tool, workflow, and memory failures direct harness changes, while model failures can inform post-training objectives. Environment or grader faults call for benchmark repair.

Harness-layer failures are where replay-based prompt iteration pays off most

Most production agent failures originate in the harness. The harness includes input interfaces, memory, tool and skill registries, and routing or workflow controllers, all of which can change without retraining. That makes it a particularly useful replay target. Its edits are frequent, incremental, and often intended to repair a specific failure trajectory.

Agentic Harness Engineering (Lin et al., Fudan/Peking, 2026) argues that observability, rather than model capability, constrains harness evolution. Its Component Observability makes editable components explicit and revertible through file-level representations. Experience Observability distills trajectories into layered evidence, while Decision Observability pairs each edit with a prediction that is later evaluated. The study reports pass-rate improvements from harness iteration and transfer to a separate benchmark with fewer tokens. HarnessX research also reports large gains from harness-only evolution before model co-evolution.

Those findings do not make individual edits safe. A prompt change that improves one answer category can break a tool call, omit a required field, or alter a previously working route. Replay evaluates that change against the trajectories it must repair and preserve, rather than only against fresh examples selected to demonstrate improvement.

Harness Continual Learning (HCL, 2026) identifies retention across committed updates as a separate objective. Repeated improvements can otherwise erode earlier fixes. A permanent replay suite is a retention corpus for testing each later harness change.

Retraining versus harness iteration: what replay evidence tells you about which to pursue

A model-level failure gives evidence for post-training objectives, while a prompt, schema, or workflow failure points to a fix that needs no weight update. The distinction prevents an improvement loop from conflating two different signals. Without it, teams can default to an expensive intervention without establishing which component needs repair.

SIA (Hebbar et al., 2026) shows how to use joint optimization without committing to one path for good. Its Meta-Agent, Task-Specific Agent, and Feedback-Agent improve the harness and update model parameters. The Feedback-Agent chooses what to change next based on recent trajectories and evidence of how well the agent did. Replay can supply trajectory evidence upstream of either choice. Its usefulness is therefore not limited to prompt iteration.

A methodological objection remains, and it deserves explicit treatment. Harness evolution is itself an iterative search procedure, so benchmark gains after many revisions may reflect that search, leaving general harness quality uncertain. The research identifies comparison with task-level search alongside a frozen baseline as necessary for assessing those gains. For practitioners, the corresponding validation is replay against held-out production traces, separate from those that drove the edits. Passing the development incidents establishes repair on known cases, while held-out traces test whether the improvement generalizes.

How do you test agent changes against real production traffic before deploying?

Replay should be automatic whenever a prompt, tool, or workflow change is proposed. Capture begins with immutable envelopes at non-deterministic boundaries, storing the input context and output envelope for each crossing. Triage then attributes the failure to a layer before selecting the segment and cut point. The candidate change runs against that segment, with non-target boundaries supplied from the record. This isolates the proposed intervention from today’s API state.

Deterministic checks cover the correct tool, required fields, forbidden calls, and schema compliance, while an LLM judge evaluates relevance, completeness, and reasoning quality. Neither layer can substitute for the other. A fluent answer missing a required disclosure can pass a judge while failing an exact check, and degraded reasoning can escape tool-call assertions. On pass, the replayed scenario joins the permanent regression suite.

Tool outputs should be matched by tool name and normalized arguments so that the candidate encounters the original responses. New calls without recordings run live and are flagged as new behavior in the report. Repeat handling also needs an explicit rule. Where the tested execution retains non-determinism, a case counts as regressed only if it fails in most repeated replays, while fewer failures identify flakiness. This differs from fully recorded replay, which is bit-stable.

Must-pass cases and aggregate score floors have different roles. The gate needs both, because an improved average can conceal a broken safety-critical workflow, and must-pass cases block on any failure. Tens to low hundreds of cases is a practical pre-merge range, while very small suites make thresholds sensitive to individual flaky outputs. The model version, configuration, and evaluation settings must also be recorded and pinned so that each comparison uses the same baseline.

Each added case covers user behavior that was missing from the suite.

Sources

  1. Chronicle: Cut-Point Replay for Regression Testing of LLM Agents Tisha Chawla*
  2. AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  3. Published as a conference paper at COLM 2026

More in Eval & Validation