Layer Attribution in Multi-Step Agent Failures

Multi-step agent failures almost never become visible at the step where they start. The error that breaks a workflow at step nineteen usually got planted at step three, and by the time anyone notices, the trail has gone cold under a dozen layers of downstream reasoning. This piece lays out a method for tracing a failure back to the layer that actually caused it: prompt, tool, workflow, memory, model, or product logic. That's the whole job. Not finding that something broke, but finding which of six places broke it.

Take a common case: a search agent reports "no results found," and the obvious fix is to patch the agent that delivered the null result. But that agent did its job correctly. The actual defect sits one hop upstream, in an earlier agent that wrote a query so narrow it could never return anything. Fix the reporting agent, and the bad query survives untouched, waiting to misfire again on the next run. This isn't a rare misdiagnosis, either. Cemri et al.'s MAST taxonomy, built from 1,642 execution traces across five multi-agent frameworks, found failure rates in popular frameworks like OpenHands and MetaGPT running as high as 86.7%. At that frequency, the gap between where a failure occurs and where it originates is the default shape of the problem, and every hour a team spends fixing the wrong layer is an hour the real bug spends surviving.

Attribution as a search problem rather than a lookup

Attribution isn't a lookup, because the evidence you need is scattered, sparse, and rarely sits next to the outcome it caused. Long-horizon agent logs can stretch into the millions of tokens, and the clue that explains a failure at step forty might live in the input to step six, three thousand tokens back, with nothing in between pointing to it.

One-shot LLM judges don't handle this well. They tend to land on a plausible-sounding diagnosis early and stop looking, which works fine on a short trajectory and falls apart on a long one. Scale AI's Continual Search paper (arXiv 2609.13463) documents exactly this pattern, showing that as trajectories grow, the judge's ability to find evidence buried mid-log degrades, a pattern where the judge's ability to find evidence buried mid-log degrades as context grows longer.

The benchmark numbers make the difficulty concrete. On Who&When (Zhang et al., ICML 2025), a set of 184 annotated failure logs, the strongest attribution method identified the responsible agent only 53.5% of the time, and the exact decisive error step in just 14.2% of cases. LLM-based prompting approaches scored 51.1% to 54.3% at the agent level but dropped to 12.5% to 13.5% at the step level. Run frontier reasoning models like OpenAI's o1 or DeepSeek-R1 on GAIA trajectories for the same task, and accuracy falls below 10%.

The problem is missing inputs. Transcripts capture what an agent said and did, but rarely what it was actually given at each step, so there's often no way to check whether a decision was reasonable given its inputs. Multi-agent systems make this worse. When an orchestrator omits a constraint and an executor ignores it anyway, both agents contributed to the failure, but not in the same way, and no attribution method resolves "different kinds of fault" if it can only assign one label per step. A layer taxonomy doesn't by itself determine which evidence to trust, but it bounds the search space before any judge starts looking, which is most of the battle.

The six layers where agent failures originate

Six layers cover the failure surface: prompt, tool, workflow, memory, model, and product logic. Each has its own failure signature, and knowing the signature narrows the search before you've read a single trace.

The prompt layer covers ambiguous instructions, missing constraints, underspecified output formats. System prompt text populates every request an agent sends, so a small drift in wording compounds silently across thousands of runs. The telltale sign is an agent that behaved correctly last week and is quietly worse this week, with no code change anywhere near it.

Tool layer failures show schema drift as the classic case: a tool's description or argument types change, but nothing tells the agent's prompt, so it keeps calling the tool the old way. Hallucinated arguments are the other common failure, where the agent builds a call that matches no real schema and either fails silently or throws a generic error. Overlapping or redundant tools create selection ambiguity; past a certain menu size, the model can't hold all the options reliably. Watch for silent retry loops, malformed arguments, or a tool call that "succeeds" but returns stale data.

Workflow layer failures come from bad task decomposition, handing a subtask to an agent that lacks the context or capability to do it, or role disobedience, where an executor ignores a constraint the orchestrator set. The CHIEF paper's constraint-violation example is a clean illustration: a constraint set at step 14 gets dropped by step 16, and nothing catches it. Missing phase gates or retry policies let a bad intermediate result travel downstream unchallenged. The signature is task drift, infinite loops, or a final output that technically satisfies a subtask while violating the actual goal.

Everything an agent stores in memory traces back to something, a retrieved document, a tool output, an environment observation, or a prior message. That provenance matters, because a corrupted or stale memory item contaminates every decision that reads it afterward. Context compression adds another failure path. Anthropic's server-side compaction cut token consumption by 84% in a 100-turn evaluation, a real efficiency win, but summarization can quietly drop the one constraint-relevant detail that mattered. The signature: an agent "forgets" something established early in a session, or treats a stale value as current.

Genuine model-layer failures, meaning hallucination or reasoning errors that can't be traced to a bad prompt or missing context, are rarer than they first appear, and this is where most teams get the diagnosis backwards. Field reports from practitioners put the figure at 65% of enterprise AI project failures tracing back to harness-level defects rather than the model's own reasoning. So the signature worth chasing is narrow, consisting of consistent wrong reasoning on a class of inputs, even after the prompt, tools, and memory have all been checked and cleared.

Sometimes the agent does what it was asked, and the ask was wrong. That's the product logic layer. Mis-specified success criteria fall here, along with missing permission boundaries or human-review gates before high-risk actions, which is a product defect, not a model one. The signature: the agent completes the task as defined, but the task definition doesn't match what the product actually needed.

MAST's fourteen failure modes, drawn from those 1,642 traces, map onto these six layers without much friction. The taxonomy here is built for the person debugging the system, not for benchmark scoring, but the correspondence holds.

What complete traces must capture to make attribution possible

The core observability failure is missing inputs. A trace can show you an agent's full reasoning text while leaving out the one input that actually explains why it reasoned that way, and a trace like that is close to useless for attribution.

TraceElephant, presented at ACL 2026, found that in its static-agentic setting, full execution traces raised step-level attribution accuracy from 17% to 30%, a 76% relative gain over traces that only capture outputs. That's what separates recording what happened from recording why.

Three span types carry the weight. Tool spans need the tool name, the arguments passed, the raw output, duration, retry count, and error state; without that, hallucinated arguments and silent retries look identical to normal traffic. Reasoning spans need to capture the plan, the action chosen, the observation received, and the next decision, because a single collapsed LLM span hides plan drift and wrong-branch selection that only show up when you can see the sequence. State transition spans need the context before and after each step, including memory reads and writes and whatever payload got handed off between agents, since that's the only way to catch context loss at a handoff.

Classical distributed tracing runs on an assumption that doesn't hold here: same input, same output, same latency. LLM systems break that assumption by design; the same prompt can produce a different output on the next run. A LangChain survey of 1,340 teams in late 2025 found 89% running some form of agent observability and 71% with detailed tracing in place, yet having traces that were actually useful for attribution remained a practical gap. Having traces isn't the same as having traces you can actually attribute failures from. Three properties separate the two: coverage, meaning which layers are instrumented at all; fidelity, meaning whether inputs get recorded and not just outputs; and linkage, meaning whether each action can be tied back to whatever caused it. Every major APM vendor has added LLM observability features, and OpenTelemetry has standardized span structure across the industry, but a standard span shape doesn't guarantee the inputs needed for attribution are actually in it.

Tracing a failure backward from symptom to originating layer

Start with the failure event itself, which can be a wrong output, a halted task, a violated constraint, or a silent wrong answer. That wrong output, halted task, violated constraint, or silent wrong answer is the symptom. The cause sits somewhere upstream, and finding it takes four steps.

Step 1: find the first unrecoverable step. Work backward through the trace looking for where execution first diverged from a correct path, before it eventually produced the visible wrong result. Those are different points, and conflating them is the single most common attribution error. AgentRx, a Microsoft Research project from 2026, synthesizes executable constraints from tool schemas and domain policy, logs violations with evidence attached at each step, and pinpoints the first "critical failure" step directly. Against prompting baselines, it improved failure localization by 23.6% and root-cause attribution by 22.9%. The distinction driving that gain is simple: a step that propagates an error is a symptom, and the step that introduced the error is the actual candidate.

Step 2: map the candidate step to its layer. Ask whether the input to that step was correct. If it was, the failure lives in how the step executed, which points to tool or model. If the input was wrong, keep walking backward. When responsibility is split between an orchestrator and an executor, as in a constraint-violation case like this, check both sides: did the orchestrator's output actually contain the constraint, and did the executor's input include it? The answer to both questions determines where the fault actually sits. CausalFlow, a 2026 approach, combines causal graph construction with counterfactual attribution to separate root causes from steps that just passed the error along.

Step 3: rule out confounding layers before committing. Model-layer attribution should be the last thing checked, not the first, which is exactly backwards from how most teams instinctively debug. Prompt ambiguity, tool schema drift, missing memory context, and workflow decomposition errors all need to be ruled out before blaming the model's reasoning. DuoTrace's detect-before-attribute pattern applies this directly: filter the trace down to anomalous spans first, then hand only the failure-relevant evidence to the attribution step. That filtering cuts the noise that otherwise leads a judge to settle on the wrong layer, and it improved agent-level attribution accuracy by 8.7% and step-level accuracy by 7.0%. The counterfactual test that produces this distinction is straightforward: if this step had received a correct input, would the failure still have happened? If yes, the step is a symptom, and the real cause is whatever fed it instead.

Step 4, validate before acting, means replaying the corrected layer against historical traces before shipping anything. Misattribution isn't a neutral mistake. It triggers a change to the wrong layer, leaves the real defect in place, and can introduce a new failure mode on top of the old one. Scale AI's arXiv 2609.13463 paper makes the stakes explicit: at production scale, misattributions carry real cost and can trigger interventions that were never needed in the first place.

None of this works as a single pass. Scale AI's Continual Search work from 2026 found that iteratively nudging the diagnostic process to keep searching, rather than stopping at the first plausible answer, improved GPT-5.5's attribution F1 score by more than 40%, from 0.349 to 0.498, on MegaRCA-Mix, a set of 50 human-annotated long-horizon failure trials.

Where attribution breaks down in practice and how teams work around it

Long traces degrade every attribution method available today. Evidence stays sparse and disconnected, and judges keep settling early. Current step-level accuracy runs 12.5% to 13.5% for LLM-based prompting, and even the best causal inference method on Who&When only reaches 36.2%. That's real progress over prompting alone, but still well short of what production diagnosis actually needs.

Part of the difficulty is structural: correctness is only observable at the final outcome, and there's no intermediate ground truth to anchor a search against along the way. It's a needle-in-a-haystack problem without a metal detector.

Partial trace coverage, which is the normal real-world condition rather than the exception, makes it worse. If tool inputs never get recorded, tool-layer failures are invisible by default. If memory reads go unlogged, there's no way to tell memory corruption apart from model hallucination, because both look the same from outside. If handoff payloads between agents go missing, a workflow-layer boundary failure looks exactly like a model failure in whichever agent received the bad handoff.

Teams working around this in practice tend to converge on the same few moves, and the most effective one is the least glamorous: scope agents into bounded, phase-gated workflows instead of open-ended multi-agent swarms. Shorter, more structured traces attribute more accurately, full stop, and that alone does more for debuggability than any tooling purchase. Instrumenting incrementally, adding full input recording to whichever layer is currently generating the most unexplained failures before expanding coverage elsewhere, gets more return per engineering hour than instrumenting everything evenly. Treating attribution confidence as a signal in its own right helps too: a high-confidence attribution to a layer that changed recently is far more actionable than a low-confidence attribution to a layer that's been stable for months. Running the detect-before-attribute pattern to strip noise from a trace before any LLM judge sees it reduces the odds that the judge locks onto the wrong layer early.

The broader trend cuts against comfort here. "Towards a Science of AI Agent Reliability" (ICML 2026), which decomposes reliability across twelve metrics and fifteen models, found that recent jumps in agent capability produced only small gains in reliability. Agents are getting better at doing things and not much better at doing them dependably. Attribution tooling has not kept pace with how fast these systems are getting deployed.

Turning a correct attribution into a validated fix

Attribution tells you which layer failed. It does not write the fix, and treating those as the same step is how teams end up shipping changes that don't resolve anything.

Fix generation has to stay scoped to the layer the attribution actually pointed to. A prompt-layer finding calls for revising the specific instruction or constraint that was ambiguous. A tool-layer finding calls for fixing the schema mismatch or trimming an oversized tool menu, not adjusting the model's reasoning around it. A workflow-layer finding calls for fixing the decomposition or adding the phase gate that was missing.

The discipline that matters most here is restraint: fix the layer the evidence points to, and only that layer. Every fix applied outside the attributed layer is, in effect, a second bet placed without the odds. Layer attribution turns agent debugging from a search problem into an engineering discipline precisely because it forces that restraint. Skip it, and every fix is a guess dressed up as a diagnosis.