Reading a Production Agent Trace to Assign Blame to One Harness Layer
Trace backward from failure symptoms to find which harness layer caused the agent error.

AI agent failure attribution by harness layer is a structured diagnostic process. Reading a production agent trace means locating the failure, following its evidence backward, and identifying where in the harness the bad state originated. The engineer must commit to one accountable layer before changing the prompt, tool interface, workflow, or memory handling.
Why agent failures get misattributed before anyone reads the trace
The dominant post-mortem instinct is to ask whether the model made a mistake. That question starts at the wrong boundary. The harness includes everything the model does not do by itself, including prompt templates, context construction, tool interfaces, orchestration policy, state management, retry and recovery logic, and observability. Each mechanism mediates whether a capable model produces a reliable outcome.
A final score, success/fail label, or HTTP 200 describes an outcome. An engineer working from them can rewrite a prompt when the tool schema was wrong, or blame the model when the orchestration loop was stuck. The correction is to locate the evidence that separates those possibilities before proposing an intervention.
What a production agent trace actually contains and why that structure makes attribution possible
A trace is not merely confirmation that a request returned successfully. It exposes the sequence behind that response. Where a log might record an HTTP 200, a trace can show that the agent called the search tool three times, retrieved an outdated policy page, exceeded its latency budget, selected the wrong handoff route, and still returned a confident answer. Those details separate a superficially successful response from a failed execution.
HarnessFix’s Harness-aware Trace Intermediate Representation, or HTIR, introduced by Chen et al. in Jun 2026, compiles raw execution traces and harness code into a unified representation. It records each step’s request and response along with Role, Execution Status, and Artifact/state effect. These fields distinguish what a step was doing, whether it completed, and what it changed. A response alone cannot make those distinctions.
HTIR adds temporal, input provenance, and control flow links between steps. Temporal links keep track of the order in which steps ran, and input provenance links show how later requests reuse earlier content and harness logic. Control flow links show why the harness moved from one step to another. Together, they turn a flat event list into a graph that supports backtracking from a failure to an earlier cause.
The seven ETCLOVG layers provide the destinations for that attribution. They are Execution Environment, Tool Interface, the layer for context and memory handling, Lifecycle & Orchestration, Observability, Verification, and Governance. Their names establish the diagnostic boundaries. Because agent frameworks, observability tools, and benchmarks lack a unified trace schema, teams often have to normalize their records before those boundaries can be examined consistently.
The seven harness layers and the failure signals that fingerprint each one
Execution Environment includes the prompt / execution evidence relevant to this walk. A characteristic signal is an instruction that tells the agent to call a tool with parameters it cannot yet know and skips clarification first. The failed call is only the symptom. The originating defect is visible in the instruction that demanded it. Span-level evidence can therefore distinguish a bad prompt from a bad inference.
Tool Interface failures appear when the model emits invalid or incomplete arguments and receives an error, then loops on retries or fabricates a result. The interface itself must enter the investigation. Harness-Bench (2026) reports that Goose makes unavailable-tool calls on tasks where OpenCode makes none because Goose does not give the model access to the tool interfaces that OpenCode provides. That comparison points to tool exposure without requiring a narrower claim about the kind of interface missing.
Tool schema drift is another Tool Interface fingerprint. The real endpoint changes its schema, error shape, or rate-limit headers, while the mock still returns the old payload. The agent retries, the loop times out, and a reasonable-sounding answer can conceal the failure. CI remains green because it exercises the stale contract. The raw response is therefore essential evidence, rather than an incidental detail beside the parsed result.
The layer for context and memory handling has a different signature. An output remains coherent but omits a constraint or document that appeared earlier in the trace. The key check is to compare what the earlier artifacts held with the prompt the final step actually received. Critical material may have been silently dropped at the context window limit. Stale memory instead appears as prior-session content or an outdated policy page retrieved without a freshness check.
Lifecycle & Orchestration failures appear as repeated calls to the same tool with identical or near-identical arguments and no convergence signal. Another fingerprint is execution continuing after a terminal error. These signals concern how the harness proceeds. A bad tool argument can originate elsewhere, while an orchestration defect allows the resulting failure to spread through subsequent steps.
Observability failures hide the evidence needed to tell those cases apart. A trace shows a transition without the execution error, and downstream steps proceed as though nothing happened. Verification failures occur when a claim that the work is complete is accepted without checking that the state in question changed. No assertion, diff, or checksum supports the claim. Governance remains a separate attribution boundary, but it should not receive blame merely because a failure is serious rather than because the trace establishes its responsibility.
Why step-compounding makes layer attribution urgent before the trace gets long
A wrong retrieval changes the plan, a bad tool argument corrupts the result, and a sub-agent handoff can lose context. Later steps inherit those defects and amplify them. As execution continues, the terminal symptom grows more distant from the originating step, so reading backward from the final answer becomes increasingly demanding.
Scale AI’s Continual Search framework, described by Raj et al. in Sep 2026, shows how LLM judges settle early on a diagnosis that seems right as execution logs grow larger. Critical evidence elsewhere in the trace remains unexamined. The same work cites a security incident whose reconstruction required recovering and analyzing more than 70,000 agent messages and files, with repeated passes uncovering missed evidence. Attribution is therefore a search problem. A structured walk forces the backtracking that an intuitive reading can omit.
How do you pinpoint the responsible layer with a four-step trace walk?
HarnessFix defines four diagnostic steps in a fixed order. The method begins with symptom localization, proceeds through evidence backtracking and candidate adjudication, and ends with layer assignment. Each stage answers a different question. The Harness-Bench comparison provides a worked illustration, moving from a failed task to unavailable-tool calls and then to the tool exposure boundary.
Step 1. Symptom localization. The engineer reads the external evaluation result together with the final TraceStep’s diagnostic evidence to establish the observable failure and where it appears as an output. Backtracking has not begun at this stage. In the Goose illustration, the starting point is the task failure.
Step 2. Evidence backtracking. The engineer follows data-flow and control-flow links backward from the TraceStep already found, stopping at each earlier step that touched the state in question. For Goose, this moves the investigation from the task outcome to the tool-call steps and the interfaces exposed by the harness. When those links are missing, the engineer must use timestamps and what the artifacts contain to rebuild them by hand. Chronology alone does not explain which earlier content shaped a later request.
Step 3. Candidate adjudication. At each candidate step, the engineer identifies its causal role: forming the bad state, propagating it, failing to expose it, or failing to validate it. In the worked comparison, the unavailable-tool calls must be assessed against the interfaces Goose exposes and those OpenCode provides. The exposure gap identifies the originating defect, rather than treating the failed call as proof of faulty reasoning. Any later propagation or missing validation would require its own evidence before being labeled a secondary defect.
AgentTrace, presented at an ICLR 2026 workshop, addresses the common mistake of misidentifying the error manifestation node as the root cause. Its causal graphs supplement text-based reasoning with structural and positional signals to rank candidates. The distinction supports the adjudication step directly. A node can show the clearest error message while an earlier node shows why the error happened.
Step 4. Layer assignment. The engineer maps the originating defect to its ETCLOVG layer and names any other implicated layers as consequences or secondary defects. In the Goose example, that assignment is Tool Interface. The proposed repair belongs in tool exposure. This commitment follows the evidence walk rather than substituting for it, and it establishes the boundary within which a fix can be tested.
When output-only traces make the walk impossible and what to demand from your instrumentation
Output-only traces block evidence backtracking because they omit the inputs needed to establish provenance. Raj et al. demonstrate this limitation using the Who&When benchmark, where developers cannot reliably attribute failures from output-only logs in a meaningful share of cases. Adding the missing inputs back makes step-level attribution much more accurate.
Teams should require full inputs and outputs at every step. Each step also needs an execution status that distinguishes success, error type, and partial completion. These records allow an engineer to compare what an action received with what it returned and whether it actually completed.
Artifact and state effects must show what changed in the environment or memory because of each step. Control-flow transitions must explain why execution continued, including a condition, timeout, error handler, or policy. Tool records need arguments and raw responses, including error payloads, alongside parsed results. Otherwise, interface failures and swallowed errors can disappear from view. These requirements belong in instrumentation before an incident, when the evidence can still be captured.
Committing to one layer before touching anything, and why premature multi-layer fixes invalidate the trace
The walk is complete only when the engineer can name one originating layer and state the evidence supporting that assignment. Until then, the proposed cause has yet to be proved. Several layers may be defective, and naming one originating layer distinguishes formation of the bad state from propagation and failures to expose or validate it, assigning each role to a layer so the repair has a testable scope.
Changing the prompt, tool schema, and retry logic together breaks that test. An improved outcome cannot show which change fixed the failure. The original trace still records what happened, but it no longer supports a clean comparison between the diagnosed defect and the intervention. The next failure in the same region then requires disentangling several changed mechanisms before the earlier diagnosis can be reused.
HarnessFix formalizes the repair discipline as regression-awareness. It generates bounded, layer-specific repair operators and validates patches against flaw-specific repair specifications and regression checks before recommending what to ship. OpsHarness applies a related discipline by contrasting successful and failed trajectories, turning evidence into atomic proposals, and admitting updates through dual-gate verification designed to prevent overfitting and regression. Each gate concerns a specific layer’s behavior.
If changing only the attributed layer resolves the original failure without breaking passing traces, the result supports the diagnosis. If other traces degrade, the attribution was incomplete or wrong. The practical endpoint is therefore not a persuasive account of what failed, but a bounded change whose effect can be separated from every layer left untouched.
Sources
- From Failed Trajectories to Reliable LLM Agents:
- From General Agents to RCA Experts: A Self-Evolving Harness for Root Cause Analysis
- Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
- Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
- From Failed Trajectories to Reliable LLM Agents: Diagnosing and Repairing Harness Flaws
- Published at ICLR 2026 Workshop on Agents in the Wild.
- Beyond Fault Localization: A Trajectory-Level Study of LLM Agents for Microservice Root Cause Analysis


