Shadow Testing Agent Changes Against Live Traffic Without a Rollout
Shadow testing reveals production failures before canary exposure risks real users.

Shadow testing agent changes against live traffic without a rollout closes the gap between offline eval confidence and production behavior. The candidate harness runs on mirrored production inputs while users get only the responses from production. For agents, replay-based evaluation also requires recorded tool outputs and sandboxed side effects, so prompt, tool, and workflow fixes can be tested against actual failure traces before earning canary exposure.
Why offline evals keep failing teams that have already passed them
The migration passes offline evals, traffic flips on Wednesday, and by noon function-call accuracy and refusal rates have diverged. The post-mortem finds an eval set that under-represented real traffic, no per-cohort monitor, and no canary gate. Six months earlier, the findings were the same. The repeat failure shows the limits of a fixed dataset in establishing how a candidate will behave across the long tail of production inputs.
That gap grows as traffic changes over days and weeks. Edge cases accumulate, cohorts submit different inputs, and the distribution moves beyond the fixtures that originally justified the release. Public benchmark performance cannot close that gap. LiveTradeBench examines agent decisions under live uncertainty and demonstrates the structural mismatch between strong static benchmark scores and real-world competence. Evidence for a release therefore has to come from private fixtures and sampled production traces, beyond what benchmark standing alone can show.
A passing score describes performance only on the cases included in the test. That distinction becomes decisive when a small but important cohort behaves differently from the average. Offline evaluation remains a necessary gate, but its confidence must be tested against the distribution the system actually serves.
Why agents break differently than single-turn models when a change ships
A single request to an agent fans out across retrieval, tool calls, memory writes, and planning steps. Each step changes the conditions for the next. An edit to the harness can therefore seem harmless in the response yet cause a failure several spans later. A dangerous candidate may return valid HTTP 200s even as it completes fewer tasks, breaks a JSON schema, chooses the wrong tool, or doubles the token cost of long prompts. Availability is not evidence of successful execution.
Even without a model change, six drift modes age the eval set. These are dataset drift, tool-API drift, prompt drift, retrieval-corpus drift, user-distribution drift, and agent-step compounding. The last creates a specifically sequential risk. Even if each step has a high success rate, the end-to-end success rate across a chain can be much lower because each added step is another chance for failure.
Tool schema drift shows how that risk escapes CI. A mocked call returns the old payload, while the real endpoint has changed its schema or error shape. The agent retries until the loop times out, then fabricates a plausible answer. CI stays green because the mock is still the same. In a related failure, the trace shows repeated calls with slightly different arguments or a planner that never emits a final action, increasing steps and token cost without improving completion.
What shadow testing is and how it differs from canary deployment
Shadow testing runs a candidate in parallel on real inputs while only production sends the response that the user sees. The candidate’s output distribution is scored against production’s. With candidate side effects isolated, the test has no user-facing blast radius and can cover the full traffic distribution. It asks whether the candidate behaves acceptably on real inputs before anyone depends on its output.
Canary deployment answers a different question by serving the candidate to a percentage of real users through rollback-ready routing. Its blast radius is non-zero by design. Retry rates, escalations, and thumbs-down responses become meaningful only when people interact with the candidate. A candidate can win shadow quality metrics and still lose on canary retry rate. The two gates are sequential and complementary, rather than competing ways to measure the same outcome.
Mirroring bounds the cost of shadow testing by duplicating a sampled subset of requests. Azure ML supports traffic mirroring up to 50 percent and distinguishes it from traffic routing through blue/green percentage splitting. Sampling changes coverage while still evaluating the candidate without exposing users to it. If shadow is skipped, the candidate’s failure modes first turn up when users are exposed to them, while skipping canary leaves how it behaves with users untested. A clean shadow run earns canary exposure.
How to test agent changes against real production traffic before deploying
The gateway duplicates a request across production and candidate paths, with a shared correlation ID connecting their responses. Both arms are scored against the same rubric. Request duplication, trace correlation, and pairwise evaluation are jointly necessary because an unpaired comparison can confuse differences in inputs with differences in behavior.
The join is more important than the global average. Linking primary and shadow outputs by trace ID makes it possible to compare the same request across a route, tenant, language, or tool path. Without that link, strong results elsewhere can conceal a localized regression. A groundedness or task-completion drop is a stop signal even though no user received the candidate response. A new candidate-only failure cluster is another reason to halt promotion.
For agents, the request alone is not enough. Replay uses recorded tool outputs and sandboxed side effects rather than re-executing calls against live endpoints. Otherwise, changes to external responses and production state make the before-and-after comparison less sound. The harness defines scenarios for coverage, golden tasks for expected behavior, tool-trace assertions for path constraints, and acceptance thresholds for shipping. It is versioned with the workflow, while prompt, router, and guardrail versions are recorded with experiment IDs so a failed run can be reproduced and attributed.
What to measure in the eval rubric for agent shadow runs
An agent shadow rubric scores execution spans across quality, schema correctness, tool path, cost, and latency. These dimensions reveal different failure mechanisms. A fluent final answer does not establish that downstream parsers can consume it or that the agent followed an acceptable execution path. Schema violations are therefore a hard stop even when quality scores are high. Tool-path assertions also check whether the candidate used the right tools in the right order.
The measurements cover computational, semantic, and agentic behavior. Computational signals include latency, tokens, and cost, while semantic signals include relevance, groundedness, and safety. Agentic signals show the tool calls, loops, and reasoning paths. Normal latency can accompany a wrong answer, so timing alone is insufficient. A spike in token usage without more successful completions can flag a runaway loop or prompt injection that a generic latency monitor misses.
Cohort segmentation prevents aggregate scores from hiding concentrated damage. Results are separated by tenant, route, prompt version, and intent, with attention to specific tool paths. An overall improvement cannot excuse a severe regression for enterprise tenants. Across the rubric’s dimensions, majority aggregation can keep one noisy metric from flipping a cohort’s verdict, but it does not override hard-stop schema failures. The resulting score identifies where execution deteriorated, creating the starting point for attribution.
Attributing a shadow regression to the right harness layer
Root-cause attribution starts at the first divergence from a correct execution. It assigns responsibility to the model, harness, or environment. Within the harness, the relevant layer may be the prompt, tool schema, workflow logic, or memory. This separation determines what kind of intervention follows. Knowing that completion fell does not establish whether the model misunderstood an instruction or the workflow allowed an invalid state to proceed.
The evidence may sit far from the outcome. An early tool response can explain a later retry loop, and clues spread across far-apart spans can explain why planning never stopped.
Failures in the model can help set post-training objectives, while failures in the harness guide changes to how execution is designed. When the environment or grader has problems, the benchmark needs repair. This keeps a prompt edit from being the go-to fix for every failure. Prompt ambiguity, tool schema drift, and workflow loop conditions are common actionable categories that do not require model changes. A closed debugging loop detects step-level errors, locates causes, proposes ranked fixes, and regenerates the trajectory around the failure point, producing a diagnosis engineers can review and audit.
Validating a harness fix before shipping through replay and promotion
For a team acting on that diagnosis, the next gate is replay against the traces that exposed the regression. A tweaked prompt passing new synthetic examples is not equivalent evidence. The original traces represent the actual failure distribution. They become regression fixtures so the proposed fix can be tested under the conditions that broke the previous harness, without building another validation set from scratch.
The replay must keep the same conditions for comparison that were set during diagnosis. Recorded outputs and isolated execution distinguish a harness improvement from a changed external response. Engineers can then inspect what changed in the prompt, tool schema, or workflow logic and whether the trajectory now satisfies the acceptance criteria. A successful result establishes that the change addresses the known failure, but it does not establish that other traffic remains unaffected.
The fix therefore returns to the production-traffic funnel under the same rubric that flagged the regression. Broader shadow coverage tests whether solving one cohort’s problem introduced another, while sampled mirroring bounds the ongoing comparison cost. Canary exposure then adds the human-response evidence needed before full rollout. Where the funnel is expressed as shadow, mirror, canary, and full rollout, mirror is the sampled form of shadow testing.
Turning the shadow loop into a continuous improvement discipline
Production traces stay useful when they become fixtures in the next version of the eval harness. Each deployment cycle can then represent more of the real-traffic distribution than the previous one. The improvement loop complements the agent, verification, and event-driven operating loops. Its output is better prompts, tools, policies, and test cases, built from evidence of actual execution.
Anomaly detection helps surface failing runs, new topics, prompt-injection trends, timeout spikes, and quality drift. Human review remains part of the loop. Reviewers inspect flagged traces, calibrate the metrics, and decide which cases belong in the evaluation dataset. That selection connects monitoring to test coverage, so an incident does not disappear from engineering attention when its immediate fix ships.
GEPA (Agrawal et al., 2025) demonstrates that reflecting on execution trajectories in natural language and then mutating prompts can use 35 times fewer rollouts than reinforcement learning while achieving better performance. This supports trace-driven prompt improvement as an engineering technique. It does not remove the need for validation. A proposed improvement still has to pass replay, cohort-level scoring, and the exposure gates that protect users.
For teams not retraining the base model, harness engineering offers a faster and more flexible path through prompts, tools, skills, evals, and memory. The TaoLive harness-aware training report notes that compact models trained on a fixed harness can overfit to its configuration and fail to evolve with harness updates without continuous retraining. That is a different intervention at a different layer. The practical discipline is to instrument, score, gate, simulate, sample, and optimize, then restart with the next trace batch after a successful rollout.


