Structuring Historical Conversations as Deterministic Regression Fixtures for Agent Changes
Freezing the model's response lets you test a fix against the exact failure.

Testing agent changes against production traces requires freezing non-deterministic boundaries as immutable envelopes and replaying only the changed layers live. Historical conversations then become regression fixtures that test a candidate fix against the recorded conditions of a failure, including the model response. For deterministic harness fixes, this requires no model calls, and the assertion checks the changed decision directly.
Why recorded production failures are so hard to turn into repeatable tests
A production incident occurs, an engineer reruns the conversation, and the agent behaves differently. The failure has disappeared, but nothing has been fixed. LLM inference is not bitwise reproducible even at temperature zero, tools read external state that has changed since the first run, and a multi-step trajectory rarely takes the same path twice. Chawla & Koul’s REALM 2026 work identifies all three sources of non-determinism as obstacles to reproducing an incident.
These sources compound because agent behavior is a trajectory encompassing tool use, memory, and delegation. The failure lives in the sequence of decisions. The same code can therefore pass or fail based on the order in which it runs and on conditions outside the code under test. Recording that sequence makes it inspectable, but inspection alone does not make it executable against a candidate change. Observability and testability are different capabilities.
Why final-answer comparison is the wrong surface for regression testing
Comparing final answers does not resolve this problem because a correct answer can emerge from a broken path. A broken answer can also hide correct decisions made along the way. An agent selects tools, supplies arguments, interprets results, and routes work before producing its final response. An error at any of those boundaries can cascade through the execution without becoming visible in an assessment of the last string.
Even apparent task completion can be misleading. An agent in a goal-driven simulation can echo the user’s assumptions rather than complete the task, producing a false pass. Different answers may both be correct, so comparing them can produce a false failure. A changed final answer is therefore not evidence of regression, and an unchanged answer is not evidence of correctness.
Tool selection, argument-schema conformance, and routing decisions can be evaluated independently of the final wording. Those checks target the decision that a change is intended to correct, rather than a downstream response influenced by many other decisions. If neither re-execution nor output comparison provides a repeatable regression test, the path is to freeze the non-deterministic parts and test the deterministic parts live, which is exactly what cut-point replay does.
What cut-point replay is and how it converts a recorded incident into a CI fixture
Cut-point replay records non-deterministic boundaries as immutable envelopes, then chooses which boundaries to serve from that record and which to execute with new code. In full replay, every boundary is served from the record. It makes zero model calls, and its repeated runs are bit-stable, which shows that the recorded trajectory can be reproduced. A cut-point test then executes a selected boundary live and asserts against its output. Per-name call-count checks guard each stubbed boundary so that bypassing it cannot silently produce a pass.
Consider an agent that issued a refund it should not have issued. The fixture freezes the model envelope that requested the refund and the external tool reads that supplied the incident’s context. The fixed tool gate runs live with those recorded inputs, and the assertion checks that it blocks the unsafe action. The model does not get another chance to choose differently.
Chronicle adapts selective replay to agent semantic boundaries. Temporal’s workflow replay similarly executes changed code against recorded event history, but Chronicle evaluates a candidate fix after an agent run has finished. Recorded envelopes also replace hand-written mocks with the data encountered during the incident. Chronicle’s 2026 analysis reports recording overhead per boundary crossing of 0.008% of a typical model call, supporting always-on capture as a practical basis for this approach.
How to capture a production trace that is actually replayable
The first step is to record inputs and outputs at every non-deterministic boundary during the original execution. There is no reliable way to get those values back after the fact. Capture should select samples based on consequential wrong actions and recurring failure patterns. The purpose is to preserve the conditions needed to test a real failure.
The trace should form a span tree rooted at the user request, with child spans for prompts, retrieval, tool calls, and sub-agent steps. Span metadata must preserve model and prompt versions, tool inputs and outputs, and retrieval context at execution time. OpenInference defines named span kinds including LLM, CHAIN, RETRIEVER, TOOL, AGENT, and EMBEDDING. OpenTelemetry GenAI instead uses operation names such as chat, execute_tool, invoke_agent, retrieval, and embeddings. These conventions differ, but both give recorded envelopes a named boundary to attach to.
Multi-agent capture also needs a trace ID that is shared across agent boundaries. Without that link, a symptom visible in one agent’s trace can hide a cause further upstream. Envelope formats should be versioned because agent instrumentation conventions remain in flux. OpenTelemetry’s November 2026 status marks JavaScript traces, metrics, and logs Stable while its GenAI agent spans and GenAI metrics documents remain Development.
The envelopes must include the prompt version, tool schema version, and routing logic version so that changes can be found. Harness versions should be treated as code with git history, allowing two states to be diffed. The fixture’s anchor is therefore a commit reference.
Structuring the fixture — what to freeze, what to run live, and where to assert
With that anchor established, fixture design selects the changed layer and holds its surrounding conditions constant. The selection must be narrow enough for attribution. LLM inference unrelated to the fix and external reads whose state has changed should be served from recorded envelopes. The tool gate, router, or other component under test becomes the live set, with an assertion immediately downstream.
Tool schema drift requires freezing the LLM envelope that generated the malformed call and running the tool validation layer live. The assertion checks that the argument now passes schema before tool execution. This isolates the validation change from model variation. A fresh model call could give different arguments and miss the original defect without testing the fix.
When a prompt is ambiguous, the test must freeze tool outputs and run the prompt-to-tool-selection layer live with the revised prompt. The assertion checks the selected tool name and argument shape against the schema. This configuration has a different scope from guard testing. If the changed prompt requires fresh inference, that boundary is live, so the test does not inherit the zero-model-call determinism of a fixture that freezes every inference boundary.
Workflow loops require freezing all envelopes up to the third repeated invocation and running the loop-detection guard live. The assertion checks that the guard fires and terminates the run before the fourth invocation. The repetition is part of the test input. Serving the guard’s result from the record would bypass the very behavior the fixture is meant to exercise.
Chronicle’s mutation study demonstrates why this distinction is essential. Its cut-point tests catch every mutant that lets a recorded unsafe action through, while a baseline that stubs every boundary catches none using the same assertion. That baseline never exercises the fix under test. Prompts, tool schemas, routing logic, and memory reads are independently changeable, so each fixture should isolate one layer for unambiguous failure attribution.
Committing fixture libraries to CI and keeping them from going stale
A fixture outside CI remains a debugging artifact. Regression protection requires automatic execution against harness changes before release, while production monitoring detects failures that pre-release tests missed. At scale, teams need a strategy for selecting the relevant subset of fixtures because a harness change does not directly identify every affected historical trajectory. Test selection must preserve that connection between the changed layer and the incidents that exercise it.
Fixtures age in two distinct ways. Their anchored harness can diverge from the current version, or the captured production behavior can stop representing the failure patterns now occurring. Pinning the version to a commit allows a flag for review when divergence goes beyond a defined delta, so the test cannot silently pass. Continuous intake addresses the second problem by turning new consequential wrong actions and recurring patterns into fixtures.
Review should link each proposed harness change to its fixture, the assertion that passes, and the run that failed before. That evidence must remain inspectable by the reviewer. Intake quality is more valuable than volume, so a small library of well-structured production incidents can be more useful than a large synthetic library. Treating fixture maintenance with the same rigor as code review distinguishes a team that improves its harness from one that repeatedly rediscovers the same failures.
Where replay validation alone falls short and what fills the gap
Historical fixtures establish whether a fix handles the conditions of a known incident. Each fixture covers one trajectory through the decision tree. That evidence does not cover dynamic decisions made with different context, even when the recorded case passes deterministically. A growing library broadens coverage but does not turn historical examples into a complete account of agent behavior.
Richard et al.’s 2026 work on agentic testing and evaluation explains that evidence gathered under test conditions may no longer correspond to fielded behavior when a system accumulates state between test and deployment. An old fixture can thus pass without backing up the claim it was meant to test about deployment. Defined expiry conditions and assigned ownership for staleness review help govern this gap.
Dhage’s 2026 study finds that a first-pass harness using finite-state execution, forced tool selection, validation, and retry significantly improves exact-match reproducibility in one model-task combination, significantly degrades it in two others, and has no effect in the fourth. The study also reports notable latency increases in some configurations.
The testing strategy therefore needs complementary layers. Fixture-based CI handles known failure modes, multi-turn simulation probes unknown ones, and production trace monitoring catches what escapes both. Staged rollout gives monitoring time to catch new failures before they reach all traffic. Replay supplies precise evidence about selected historical conditions, while these other practices address the uncertainty that remains.
The engineering workflow that connects production failure to validated harness fix
The engineering loop starts before an incident is investigated, because instrumentation must already have preserved the boundary envelopes, span metadata, and harness commit. The fixture comes from that record. Attribution then identifies the responsible prompt, schema, router, memory read, or workflow guard before a fix is written. In a multi-agent run, this requires following the shared trace ID upstream from the symptom rather than assuming the last failing span is the cause.
Once the responsible layer is identified, the fixture freezes surrounding boundaries and places that component in the live set. Its assertion targets the decision immediately downstream. The candidate harness change is then tested against the captured conditions, with validation requiring failure on faulty code and a pass on the fix.
The fixture, fix, and passing assertion are committed together, with the recorded incident available as pull-request context. Review connects the code change to the evidence. After deployment, monitoring checks whether the fix holds and whether new failure patterns emerge, feeding recurring incidents back into the library. Each release decision can then rest on an inspectable chain from production failure to targeted assertion, while new traffic supplies the next cases the harness must learn to handle.
Sources
- Chronicle: Cut-Point Replay for Regression Testing of LLM Agents Tisha Chawla*
- Testing and Evaluation of Agentic AI Systems In Military Command and Control
- Chronicle: Cut-Point Replay for Regression Testing of LLM Agents - ACL Anthology
- Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints


