The Agentic Harness

Why Harness-Unaware Observability Platforms Cannot Assign Root Cause Across Prompt Tool Workflow and Memory Components

Span-level replay shows what failed but not which harness component caused it.

· 8 min read
Cover illustration for “Why Harness-Unaware Observability Platforms Cannot Assign Root Cause Across Prompt Tool Workflow and Memory Components”
Agent Failure Diagnosis · November 13, 2026 · 8 min read · 1,896 words

AI agent failure attribution by harness layer requires tracing a visible failure back to the prompt, tool, workflow, or memory component that caused it. Span-level replay alone cannot do this because a state in one component can cause a later failure in another. Harness-aware attribution connects the full trajectory to component-specific evidence, then validates a scoped repair against historical cases.

Why span-level tracing cannot locate harness failures

Span-level trace replay assumes that the evidence for why a failure happened is in the failed span or its direct parent. Agent trajectories violate that assumption because intermediate states accumulate and shape later decisions. A tool response can be misinterpreted without producing an exception, while the resulting belief persists through several otherwise reasonable actions. Inspecting the failed span therefore exposes the consequence without necessarily exposing its origin.

For a deterministic service failure with a local cause, replay can surface the relevant operation directly. That model falls short when the task is to find how an earlier interpretation changed what happened later. More logging does not, by itself, close this gap. A replay system must also connect the recorded evidence across states and harness components to explain the failure. AgentDebugX makes this distinction, saying that existing observability tools provide execution replay but do little to help with root-cause identification or recovery.

This is a collision between two architectural models. One organizes evidence around individual operations, while the other must explain decisions conditioned by an accumulating history. A trace remains useful as the record of what happened. It becomes insufficient when its boundaries are treated as the boundaries of causality.

What "non-local causality" means in a multi-step agent run

Consider a corrupted tool response that is accepted as valid at step three. The agent writes that response into its working state and references it at steps five and seven. Each later action can make sense given the state at that point, even though the state itself is wrong. In this example, the failure at step seven follows from what step three established.

At step seven, replay might show a failed tool call, an inconsistent decision, or an unsupported final response. None necessarily identifies the earlier corruption. Attribution must reconstruct how the accepted response conditioned the intervening decisions, rather than inspect those decisions as independent events. That requires the full execution history in scope. Without it, a developer can patch the final response or tool invocation while leaving the state-changing defect untouched, allowing the same origin to produce a different symptom later.

The Who&When benchmark tests this difficulty through exact agent-and-step attribution. AgentDebugX reports that even the strongest single-pass baseline achieves only 21.7% accuracy on that benchmark. The result establishes how difficult attribution remains even when a method is explicitly tasked with it. It also explains why replaying a failure should not be mistaken for diagnosing one.

What a harness actually contains and why each component has its own failure surface

A harness is a multi-component system. Its system prompt specifies behavior, so ambiguity there can propagate across thousands of agent steps and appear to be a limit in model capability. A tool description instead shapes the agent’s understanding of a tool’s interface. When that understanding diverges from the actual schema, a seemingly correct call can produce unexpected outputs. These defects can look alike but need different fixes.

Tool implementation has its own failure surface because a tool can return corrupted or missing fields without raising an exception. Middleware can silently transform parameters between the agent and the tool in ways neither side detects. Skills and sub-agent configuration add workflow and configuration dependencies, including invocation at the wrong point or stale parameters. Failures caused by skills and sub-agent configuration can occur far from their definitions. Long-term memory extends that separation further, as a poisoned read can produce a reasoning failure steps after the write that introduced it.

A generic “wrong answer” signal cannot distinguish an ambiguous prompt from a faulty tool implementation, a poisoned memory read, or a broken workflow loop. Even finding the first faulty step leaves another attribution task. The diagnosis must map that step to concrete implementation artifacts, including source code, prompt templates, tool specifications, configuration files, adapters, and instrumentation hooks. Structured component tagging supplies the connection between a trajectory event and the part of the harness that can actually be changed.

The three failure categories that harness-unaware platforms cannot distinguish

Tool schema drift is difficult to detect because execution can appear normal. Memory-Induced Tool-Drift in LLM Agents documents schema drift as a failure category that produces corrupted outputs without an exception. AgentCheck formalizes it as fault type A4, with the pass criterion “detect field mismatch,” distinguishing it from eleven other fault categories. The interface mismatch provides the evidence needed to distinguish schema drift. Without that distinction, teams investigate downstream behavior rather than the upstream schema change that altered what the agent received.

Prompt ambiguity is easier to observe but easy to misattribute. An agent system prompt functions more like a software specification than a chatbot persona, because its instructions govern behavior across many steps. If the instructions are unclear, behavior that is hard to predict can look like a limit in what the model can do. A model upgrade then becomes an attractive but misplaced response. The defect is in the specification, so attributing it to the model redirects effort away from the instruction that needs repair.

Workflow loops have visible symptoms, including the same or similar tool calls three or more times in a row, repeated edits to one resource, or verbose reasoning without progress. A standard trace can expose repetition and latency, but not necessarily the missing exit condition or skill that repeatedly re-enters a branch. Visibility does not make the repair obvious. Misdiagnosed loops can provoke harness rewrites or model changes, making them the costliest of these three categories when the required change is instead confined to control flow.

These categories progress from silent corruption to ambiguous behavior to visible repetition. Yet each can end in a recorded wrong answer or failed tool call. The component that created the conditions for that outcome distinguishes these failures. Without that distinction, a monitor provides only a symptom class.

What misattribution actually costs engineering teams

Misattribution redirects engineering work to the wrong layer. An ambiguous prompt that triggers a harness rewrite can turn a short debugging session into a multi-week project. The opposite mismatch is prompt-only repair applied to a defect that requires runtime harness modification. Such a change can improve observed performance without correcting the responsible trace step and component. A better result on the observed case does not establish that the underlying flaw has been removed.

A team may revise instructions repeatedly while a workflow defect remains, or change model behavior around an unresolved tool mismatch. Each intervention adds work without clarifying responsibility. Component-specific attribution changes the unit of repair from the whole agent to the harness element that produced the faulty state. That narrower scope is what makes the diagnosis useful to engineering.

Missing attribution also weakens the harness’s operational feedback. A failed tool call can supply context for a memory write, repeated failures can flag a skill for revision, and a latency spike can prompt a switch between protocol paths. These responses depend on structured evidence about what happened and where it originated. Raw failure counts cannot supply that causal distinction. Without that link, the harness stays a fixed scaffold, with no way to use what happens as it runs to guide changes to its parts.

How do you pinpoint which harness layer caused a production failure?

Trajectory-level attribution begins with evidence that ordinary span inputs and outputs omit. Every intermediate belief state that conditions a later decision must be captured as a snapshot, so the diagnosis can follow state changes using evidence beyond boundary events. Failure evidence also needs component tags that name the prompt template, tool schema version, workflow branch, or memory read tied to the failure. Step identity alone is not enough for attribution. The system must connect the first divergence in the trajectory to the harness implementation responsible for it.

Re-running execution with cached tool outputs can reproduce a failure, but reproduction does not establish where the causal chain broke. Attribution needs a reasoning process that holds the entire history in scope and investigates how evidence passed between components. That takes model-driven analysis beyond what rule-based heuristics alone can do. A loop indicator can flag repeated calls, for example, but the diagnosis must still explain which control-flow defect caused them.

Riddell et al. describe the adjacent problem of multi-hop fault propagation in complex distributed systems at FORGE’26, where “symptoms appear far from their true causes.” Local causality is therefore not universal even in cloud systems. The same diagnostic obstacle appears in agent harnesses when state propagates a fault across several decisions. Diagnosis requires cross-step causal reconstruction.

AgentDebugX’s DeepDebug approach meets that need with multi-turn diagnosis that uses global trajectory understanding, structure-guided investigation, and cross-examination. AgentDebugX reports 28.8% exact agent-and-step attribution accuracy for that approach in its 2026 Who&When results. That improves on the single-pass result without making attribution a solved problem. The distinction is important when assessing an observability platform. Its logs must allow component mapping and trajectory-level reasoning, and its diagnosis must be checked before it is accepted.

How a harness-aware attribution loop works from detection to validated fix

A harness-aware loop starts when step-level anomalies trigger attribution. Tool call failures, schema mismatches, and loop indicators give an engineer evidence to investigate as well as alerts that point to a trace. The attribution process then examines the full trajectory and connects the failure to a prompt, tool schema, workflow branch, or memory read. Its output is a structured diagnosis explaining what changed and why and identifying a repair target.

AgentDebugX organizes that process as Detect, Attribute, Recover, and Rerun. AgentDebugX reports that on GAIA, this closed loop repaired substantially more failed tasks in a single rerun than three decoupled self-correction baselines. Recovery was grounded in the attributed cause rather than applied generically where the failure was visible. That distinction determines what the recovery phase changes. A prompt defect calls for a targeted prompt revision, a tool-description defect for a corrected description, and a missing workflow exit for a control-flow patch.

The candidate repair then faces historical evidence. Each replay anchor holds the raw input and success criterion from a case seen before, so that case can run under both the deployed and candidate harnesses. Testing only the original failure is insufficient because a modification can reduce one recurring problem while introducing another. Regression-aware commit gates run the full trace set against the candidate before it can be recommended. Validation therefore tests both the proposed causal explanation and the repair’s effects beyond the incident that prompted it.

Patterns that recur in live traffic feed the next attribution cycle, so the process goes beyond single incidents. The TaoLive / Taobao Live deployment illustrates this separation between model and harness change through a Harness Agent with a slowly updated policy model and rapidly evolving harness state. Operators can change business behavior without retraining, and the approach was tested against a production ReAct system in a user-level online A/B test. The Nex/NexAU0 framework identifies observability, rather than agent capability, as the bottleneck for agent-driven harness evolution. Once evidence maps failures to editable components, those components can be revised independently and validated in isolation.

Sources

  1. AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  2. Stalled, Biased, and Confused: Uncovering Reasoning Failures in LLMs for Cloud-Based Root Cause Analysis

More in Agent Failure Diagnosis