The Trace-Driven Iteration Cycle for Agent Prompts and Tools
Use production traces to debug agents faster than guessing at prompt changes.

A production trace-driven harness engineering workflow collects real agent traces, attributes failures to a specific layer, generates a bounded candidate fix, and validates that fix against production traces before shipping. This workflow replaces speculative prompt editing with repeatable, evidence-backed improvement. Continuous monitoring closes the loop by turning new failures into the evidence for the next change.
Why the standard improvement workflow for agents is broken
A prompt change goes in, behavior changes downstream, and the team cannot establish whether the edit helped or merely moved the failure. Without a record connecting the original failure, its cause, the intervention, and its validation, an apparently better answer offers little evidence that the agent became more reliable. The change may have corrected one response while disrupting a tool sequence elsewhere.
Traditional APM tools report uptime, latency, and error codes, leaving uncertainty about whether an agent chose the right tool, supplied valid arguments, or followed its original plan. An early model decision can send execution down the wrong path while later steps rationalize the mistake. A final-output log exposes the conclusion while potentially leaving the inflection point hidden. That gap leaves teams editing the most visible symptom while its underlying cause remains unaddressed.
The five surfaces where agent runs break, and what each looks like in a trace
The five core failure surfaces are plan, tool, retrieve, reason, and respond. These diagnostic surfaces distinguish different kinds of failure. A planning failure appears as the wrong tool sequence, a missing step, or an out-of-order action in the span tree. Each call may succeed even when the branch as a whole is wrong. A workflow loop has a different signature, with repeated tool calls or reasoning steps that make no progress.
At the tool surface, schema drift exposes a mismatch between the interface the harness expects and the endpoint it actually calls. An external API may change its schema, error codes, or rate limits while the harness keeps building calls for the old version. CI can miss this entirely. Its mock may still return the old payload, leaving the production trace as the evidence of the mismatch. By contrast, a retrieval failure may show a near-duplicate document that lacks the key fact needed to answer correctly.
At the reasoning surface, intermediate decisions can reveal an early wrong turn that later steps accept. The response surface shows the resulting behavioral breach. Prompt ambiguity can underlie skipped verification, incorrectly constructed queries, dropped handoffs, or the wrong output format because the instruction leaves an expectation underdetermined. An especially damaging trace shows a tool’s 500 response followed by a made-up answer that sounds right. The error and the agent’s handling of it are separate evidence.
What a well-instrumented trace actually contains, and why structure is what makes attribution possible
A useful trace has the agent run as its root and typed child spans for retrieval, LLM calls, tools, evaluation, and guardrails. Planning usually appears as a CHAIN or LLM span. Tool calls should nest under the LLM call that requested them so that an investigator can identify which model decision produced an invocation. A flat list loses that causal connection.
Model and prompt versions show which configuration an LLM call used, while latency and finish reason show how it ran. The OpenTelemetry GenAI spec identifies tool name, call ID, arguments, and result as tool-span attributes, with arguments and results opt-in because of their sensitivity. Evaluation scores can then attach quality signals to spans without replacing the underlying execution record.
This structure separates failures that look identical in a final answer. A wrong-tool selection can occur even when its input is valid, while retrieval drift can ground an answer in the wrong document. Both may produce a superficially plausible response. Only the linked record exposes the selection or grounding step, and only the intermediate path reveals where reasoning collapsed. Attribution depends on preserving those distinctions before an incident occurs.
Why root cause attribution requires more than reading the trace — and what it demands instead
A structured trace provides evidence that requires interpretation to support a diagnosis. The failure may be visible in the trace several actions after its cause, with the information needed to explain it spread across spans far apart. Reading from top to bottom does not solve that problem. An investigator can anchor on the first plausible explanation and stop before finding evidence that contradicts it. Larger, more distributed execution records make that shortcut especially risky.
Sound attribution treats diagnosis as a search problem. It tests multiple plausible explanations, seeks contradicting evidence, and rules out alternatives rather than selecting whichever story best fits the final error. For example, an incorrect response does not by itself distinguish missing source information from an ambiguous instruction about handling a tool failure. The distinction determines which layer should change. Self-reflection helps detect errors, localize them, and revise reasoning, but localization is useful only when the evidence search supports it.
The result should be a typed diagnosis. It names the root cause, the evidence that supports it, the confidence level, and the proposed fix. That output gives the engineer something concrete to assess. It also establishes the boundary of the intervention, so a failure at a tool interface does not automatically trigger a rewrite of the system prompt.
The harness as the search space — prompts, tools, workflows, and memory as levers
Most production agent failures originate in the harness. The harness comprises the prompts, tool interfaces, workflow logic, and memory surrounding a fixed model. That makes it an accessible search space. Teams can change tool schemas, middleware, state handling, and recovery logic without a training run, and those changes remain reviewable, reversible, and testable. Attribution narrows which part of that space deserves attention.
Prompt ambiguity points to the system prompt as the place to look, while schema drift points to the tool boundary and its validation logic. A workflow loop instead directs attention to the planner or recovery path. These are guarded intercepts at specific execution boundaries. They change how the model meets a tool, or how a planner hands work to an executor, rather than rewriting agent behavior indiscriminately.
TrajTune illustrates how that approach can become an operational loop. It triggers multi-LLM feedback when error metrics exceed adaptive thresholds and iteratively refines prompts. Its reported results include significant reductions in hallucination rates and meaningful gains in tool success and domain-specific accuracy across finance, software engineering, and IT-ops. Prompt changes can change tool use and reasoning paths without any code change, so prompt quality needs to be part of operational control.
Generating a candidate fix — what makes a proposed change reviewable rather than a black box
A candidate fix must show exactly what would change and why that change would address the attributed failure. For a prompt failure, that means a concrete instruction addition or revision tied to the expectation left underdetermined. A wholesale rewrite obscures the connection. For a tool-boundary failure, the proposal instead targets schema validation, error handling, or argument mapping at the affected interface. The engineer should be able to connect the edit directly to the evidence.
A candidate should target the branching condition or recovery path responsible for a loop or skipped step, preserving the rest of the planner. Narrow changes make review and rollback easier. They also make validation more informative because fewer altered behaviors compete to explain an improvement. A broad rewrite may raise a score while leaving the team unable to tell what worked.
Trace-aware optimization gives the candidate more to work from by showing execution paths, tool calls, and intermediate reasoning as well as the final result. A scalar score says that performance was poor. The trajectory can show where it became poor, allowing the proposed edit to target that point. Reviewability therefore belongs within the engineering mechanism during optimization.
How do teams validate a fix against real production traces before shipping?
A prompt change can score positively on a golden set and negatively on shadow traffic. The contradiction exposes a distribution gap. The curated examples did not represent the requests actually reaching the agent, so their score could not establish production improvement. Replay against production traces closes that gap by testing changes against evidence of how the agent ran, rather than relying only on selected questions and answers.
One concrete technique records a known-good execution trace and deterministically replays its LLM responses while running the real tool chain. This fixes the model-side responses while exercising tool integration and real side effects. Its scope must remain explicit. Holding those responses fixed isolates tool-chain behavior, while assessing changed prompt behavior also requires evidence from the candidate’s outputs. A replay score should not stand in for a claim the replay did not test.
Failing live sessions can also become regression cases through “replay to dataset,” moving production incidents into the evaluation suite. Mature workflows combine offline and live evaluation. Historical replay blocks known regressions in CI before promotion, while live scoring evaluates real sessions, gates risky responses through an inline judge, and feeds failures back into the offline dataset. Neither mode substitutes for the other because a fixed collection cannot contain the full range of user behavior.
For multi-tool agents with interconnected tool ecosystems, this distinction matters for more than the quality of the answer. Failures can lie in step sequences, side effects, and state. Replay exercises those relationships under the candidate change, rather than treating the final response as the whole system.
How harness iteration compares to retraining — and when each approach is the right call
Harness iteration generally addresses production failures faster and with fewer resources than retraining, but it cannot supply a capability the base model lacks. The decision depends on the bottleneck. An ambiguous instruction, mismatched interface, or broken recovery path calls for a different intervention from a task the model cannot perform. A failed first harness edit does not establish that the model has reached its capability limit.
The comparison also depends on how optimization uses evidence. GRPO-style reinforcement learning updates policy weights through reward-driven objectives, while GEPA uses natural-language trajectory context to reflectively evolve prompts across a compound AI system. GEPA’s findings show the value of keeping rich feedback and that prompt evolution can be sample-efficient, without proving that it always does better than reinforcement learning. AutoSaddler’s 2026 paper extends the comparison to the full harness and reports strong sample efficiency with its search strategy. Neither result eliminates the need to diagnose the failure first.
The practical decision rule is to begin with trace-grounded harness iteration. Fine-tuning or reinforcement learning is a good fit when the record shows that the model lacks task capability or needs policy-level behavioral changes that prompt updates cannot maintain across a wide range of inputs. RL-centric optimization also requires a robust reward and evaluation pipeline. Otherwise, the optimization process can introduce instability of its own.
Catching silent drift — how continuous trace monitoring keeps the loop running after a fix ships
A validated fix does not freeze the environment around an agent. Model providers update weights, APIs change response formats, user behavior evolves, and retrieved knowledge becomes stale. These changes can shift agent behavior without a deployment. This silent drift differs from a regression introduced by the team, even when both produce similar symptoms. Continuous production traces provide the evidence needed to distinguish them.
One operational signal is consecutive days of declining task completion, overlaid with model-version changes, deployments, knowledge-base updates, and traffic shifts. Per-turn classifiers can label production behavior, route those labels into the evaluation set, and feed them back as training signal. Live requests show how the range of cases the agent must handle is changing in ways a held-out dataset cannot. Those cases prompt teams to reopen attribution and validation before a user report.
Earlier failures keep examples of broken sequences on record, while validated fixes become regression cases for later candidates. Teams can compare each new deviation with evidence they already have to ground subsequent prompt edits. The trace-driven cycle compounds through that record, making the next diagnosis more precise and the next change easier to verify.


