Prioritizing Agent Improvement Backlogs by Failure Impact
Root-cause attribution must precede backlog ranking to fix failures.

Prioritizing agent improvement backlogs by failure impact starts with attributing each failure to its responsible harness layer, then ranking it by frequency, downstream consequence, and recurrence potential. Production traces become a compounding improvement loop when teams use that attribution to ship targeted fixes and turn resolved failures into regression coverage. Ranking terminal errors before finding their causes leads engineers to work on symptoms that come back.
Why agent failure backlogs grow faster than teams can clear them
The backlog never clears when each shipped fix is followed by another version of the same failure. A tool call breaks, a prompt changes, and a later run fails differently because the original defect remains. These are not necessarily separate problems requiring separate fixes. Agent failures often become visible at a different step from the one that caused them. Triage based on whatever broke last therefore selects the wrong intervention.
HPFA at COLM 2026 formalizes this non-locality problem through failed web-search tasks. Existing attribution methods either require expensive step-by-step counterfactual testing or flatten reasoning traces into sequences that hide non-linear dependencies. Scale AI’s paper “Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures” identifies a related obstacle in logs that can reach millions of tokens. Relevant evidence is sparse and distributed across the record. One-shot LLM judges can settle on a plausible explanation before examining the evidence needed to establish it.
LongRCA Bench illustrates the distance between cause and symptom across 1,140 human-annotated failed trajectories. Reference root causes precede terminal failures by a median of 48 recorded steps, while the strongest baseline finds the correct root-cause step only a fraction of the time. This makes attribution a substantial part of the repair work. When teams skip it, downstream patches leave upstream causes active, and the backlog refills.
What makes agent failures hard to attribute in the first place
A single early error can produce several observable deviations, each of which looks like a plausible cause. Their correlation with failure does not establish their causal role. Later conclusions may also depend on a mix of earlier parallel steps that are far apart in the trace. HPFA models these relationships as hypergraphs rather than sequences because a linear reading can obscure how an error propagated. The visible order is not the whole explanation.
TRAJDEBUG from Tsinghua/Tencent adds another distinction by tracing the lifecycle of local errors. Some errors are repaired during execution, some remain harmless, and only some contribute to the terminal failure. The critical error is therefore not necessarily the first mistake or the last one. Attribution requires following what happened after each candidate error, including whether later steps corrected it. Finding a bad step is only the beginning.
TelemetrySuffBench shows why more logging alone may not solve this problem. Detection and attribution need different evidence, especially clear links between decisions and their provenance. Observability remains necessary, but a trace that reveals failure may not explain its origin. A backlog ranked only by terminal error counts therefore ranks the signs of failure before their causes are known.
The five failure categories that structure any agent backlog
Tool schema drift is the canonical silent failure. The tool accepts a call and returns something, but the output is corrupted because the argument shape no longer matches the interface contract. Trying again with the same malformed arguments can look like model confusion. The defect nevertheless belongs to the contract between harness and tool.
Prompt ambiguity can look much the same, but it has a different cause. The model returns plausible output that satisfies the wrong interpretation of an instruction. The answer may look like a capability failure. But if the instruction itself allows that interpretation, the ambiguity remains after a model upgrade. This category points first to the prompt layer.
Workflow loops expose the limits of classification by symptom. An agent may continue because the harness exit condition is never satisfied, or because the model cannot recognize task completion. Both appear as a run that does not terminate. Their fixes are entirely different, while their downstream costs can include token consumption, degraded latency, and blocked agents.
Multi-agent coordination failures extend attribution across handoffs. A syntactically valid message can propagate an incorrect assumption, trigger unsafe delegation, or create a circular dependency even when its originating agent succeeds by its own criteria. The METR and Redwood Research investigation report describes a Hugging Face security breach whose reconstruction required repeated passes over more than 70,000 agent messages and files.
Reasoning errors complete the taxonomy, but the label does not establish model responsibility. Incomplete context, missing memory, or absent preconditions in a prompt can all make a run look capability-limited. The visible reasoning failure may have an upstream source. These five categories give the backlog a shared vocabulary, while layer attribution identifies the actual repair target within each category.
Why layer-specific attribution must come before ranking
Each backlog item needs a layer label before it gets a priority score. Identifying responsibility in the prompt, tool, workflow, memory, model, or product logic determines ownership and the kind of change required. A category alone cannot make that decision. A workflow loop caused by control flow needs a harness correction, while one caused by failure to recognize completion presents a different target.
Scale AI’s framework separates root causes into model, harness, environment, and grader components because each calls for a different intervention. A harness fix can be a code diff, a model fix may require post-training, and an environment fix may sit outside the team’s direct control. The boundary determines what engineering can actually change. Ranking before establishing that boundary attaches urgency to a symptom without establishing a path to resolution.
The evidence must support the label. TelemetrySuffBench finds that when decision-content fields are removed, origin-step accuracy falls to zero for every model tested. TRAJDEBUG separately handles error trigger detection, error state classification, and causal attribution because multiple local errors can coexist in a failed run. These distinctions prevent an observable deviation from becoming an unsupported diagnosis. They also explain why a larger collection of inputs and outputs cannot substitute for missing causal evidence.
The practical cost of reversing this sequence is concrete. A prompt patch for schema drift leaves the interface defect unfixed, while a model upgrade for ambiguous instructions spends budget without repairing the instructions. A model-side workaround for a control-flow bug also leaves the exit condition broken. Frequency and impact estimates become useful only after attribution identifies which layer needs changing.
How to rank failures once each has a layer attribution
Frequency is how often the same failure pattern, once attributed, appears in production traces. It does not count every terminal error as an independent defect. A schema problem triggered on every invocation of a particular tool outranks a prompt ambiguity appearing once a week when their other consequences are comparable. Intelligent sampling makes this analysis tractable at scale by focusing collection on high latency, errors, or low confidence. The sampling must still retain signals from infrequent but severe patterns.
Downstream consequence measures how far the defect propagates and what that propagation costs. An error that starts during planning or memory use can affect later tool calls and reasoning steps before a later step exposes it. The originating step may look locally minor. A workflow loop shows why the full path is important, since its costs accumulate with each iteration until a timeout or resource limit stops execution. Summaries of terminal errors hide much of that cost.
Recurrence potential is how likely the same defect is to trigger again unless its layer changes. An outdated schema or a prompt that lacks a required constraint is still broken between runs. The trigger is repeatable because the defect is static. Coordination failures can have the same persistence when an incorrect assumption is encoded in a message format or delegation rule. By contrast, errors repaired during execution should not inflate counts of unresolved failure patterns.
These criteria let engineers spell out their judgment without making up numerical weights. Failures that happen often, spread far downstream, and have structural causes that cannot correct themselves should come first, even when their final symptoms look minor. Failures that are rare, have limited effects, and are reliably fixed mid-run can usually wait. Safety or policy implications are the exception, since low frequency does not make those consequences acceptable.
Why most harness-layer failures should be fixed before considering model changes
Most failure categories in a production backlog point to changes in prompts, tools, workflow, memory, or recovery logic before changes to model weights. Their fixes can be inspectable, reviewable, and reversible diffs. That makes harness iteration the first intervention to test. It also keeps the proposed change close to the evidence, since a schema correction addresses a schema defect directly rather than asking a different model to work around it.
HarnessX shows how much can improve while model weights stay the same. In tests across five benchmarks, it reports gains averaging 14.5 percent across fifteen configurations through trace-driven harness evolution. Co-evolution adds incremental gains beyond those changes, particularly for capability-limited agents. Google Research’s open-sourced Harness², released October 9, 2026, makes improvements at two levels. One improves the harness for the current task, while the other improves how harnesses are improved across tasks.
Weight updates are appropriate when attribution establishes a genuine capability gap rather than a configuration gap, with the notes identifying conditions such as dense step-level rewards, training stability as the binding constraint, or multi-step tool use where harness-level gains have demonstrably been exhausted. SIA applies harness iteration first and switches to weight updates when progress stalls.
For backlog management, the lesson is to test the cheaper, more reviewable correction before escalating to a model change. Historical traces give teams a way to validate whether a harness diff resolves the attributed pattern without introducing regressions. Prompt ambiguity and tool schema drift do not become capability problems because a stronger model sometimes tolerates them. Their underlying contracts still need repair. Harness-first improvement is therefore a sequence of interventions that allows for model capability to limit reliability.
How to close the loop between agent monitoring and regression coverage
A triage process remains incomplete when it stops at “fix shipped.” Each resolved failure should become a regression test that detects future changes reintroducing the same pattern. The production trace supplies the case’s essential context, including tool arguments, memory state, inter-agent messages, and prompt version. That context is difficult to reconstruct after the fact. Synthetic cases can describe the defect, but they cannot reproduce its original circumstances with the same fidelity.
Once a failure has been attributed and resolved, its trace should be labeled with the expected behavior and the failure rubric. It then belongs in the regression layer of the evaluation set. A versioned golden dataset, refreshed regularly from production failures, records the conditions the harness has learned to handle. Pass thresholds should block release when a change reintroduces a covered failure.
Shadow evaluation can run scorers on live production traces without replaying the request, giving teams a view of quality as the production distribution drifts. Historical regression cases and live evaluation serve different purposes. The former protect confirmed fixes, while the latter expose conditions that the existing dataset may not represent. New failures can then enter the same attribution and prioritization process.
The compounding improvement loop from production agent observability is this accumulation of causal evidence and regression coverage. Each resolved case adds a confirmed failure pattern at a known layer, narrowing the search when similar symptoms appear again. Triage becomes faster because the team has more than a history of alerts. It has a library of attributed defects, reviewed fixes, and tests that make those fixes durable.
Sources
- Published as a conference paper at COLM 2026
- TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?
- LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories
- TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
- Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures


