Distinguishing Model Errors From Harness Errors in Agent Traces

Two agent runs can display the identical failure, the same dropped instruction, the same wrong tool call, and require completely different repairs. One needs a change to the model's weights. The other needs a change to how the harness manages context. Mistake one for the other, and the fix does nothing, or worse, papers over a harness bug with a model patch that degrades performance somewhere else.

Why the same visible failure can demand opposite fixes

Most engineers debugging an agent trace know something failed. They rarely know, with any confidence, which layer of the system is responsible. That gap, between detecting a failure and attributing it, is the actual problem, and it's a harder one than it sounds.

Take a long-running coding session, the kind Raj et al. use as their canonical example: a user gives an instruction early on, and dozens of turns later, the agent acts as though the instruction never existed. There are two ways this happens. In the first, the harness's context compaction routine trimmed the instruction out of the working context to save tokens, and the model never had a chance to obey it because it was never shown to the model at the relevant step. In the second, the instruction survived compaction just fine, sat right there in context, and the model looked past it. The transcript reads the same in both cases. The agent ignores an instruction it was supposedly given. But scenario one calls for a harness fix, probably a change to what compaction is allowed to discard. Scenario two calls for something closer to post-training or a reasoning-level intervention on the model itself. Applying the wrong one means you've spent engineering time solving a problem that doesn't exist while the actual problem persists into the next run.

This is the problem of deciding who should fix a failure, and it recurs across nearly every category of agent failure. Depending on where the fault actually sits, the correct fix might be model post-training, harness engineering, environment redesign, or, in cases where the evaluation itself is broken, benchmark repair. A pass/fail label tells you none of this. Agent behavior emerges from the interaction of model, harness, user, tools, memory, and environment, and once you collapse all of that into a binary outcome, you've discarded the one piece of information that would have told you where to look.

The harness's contents and why it is the more common fault site

The harness is everything non-parametric wrapped around the model, including the prompts, the tool definitions, the control flow, and the coordination logic between agents. It's the scaffolding that determines how a fixed, unchanging model actually executes a task. Nothing about the model's weights changes when the harness changes, yet the harness governs almost everything about how those weights get expressed in a given run.

The scope of what counts as "harness" has grown substantially. It used to mean prompt templates and maybe a retry wrapper. Now it covers workflow design (sometimes called orchestration engineering), evaluation logic, permission and access controls, and persistent state management across sessions. This is closer to runtime and systems engineering than prompt writing, and the skills it demands look more like distributed systems design than natural language processing.

Externalization is the name for the broader trend driving this. Agents increasingly get built by rearranging the runtime around a model that never changes, rather than by retraining the model itself. State, procedural knowledge, and interaction structure increasingly live outside the model itself, managed by the runtime rather than encoded in weights. The harness coordinates and governs execution, and it underlies all three.

This matters for diagnosis because a large share of agent failures aren't failures of model capability at the level of a single action. They're failures of process stability: a step gets skipped, two operations happen out of order, the agent terminates before finishing. None of that requires the model to be "wrong" about anything. It requires the harness to have failed at sequencing, and sequencing is a harness responsibility, not a model one.

How the interaction-edge model turns a trace into an attribution problem

Raj et al., in "Model or Harness? An Interaction-Centric Taxonomy for Localizing Agent Failures" (arXiv:2607.28802), propose treating interactions as the unit of analysis, not components in isolation and not outcomes. A failure isn't "the agent failed." A failure is assigned to a specific edge, the connection between two components, plus a fault side that says which end of that edge the repair belongs to.

If the fault lands on the model side of an edge, the target is post-training. If it lands on the harness side of the same edge, the fix is scaffolding or tool-integration work. If the fault is in the environment or the grader itself, the evaluation setup needs to be redesigned before anyone trusts it to measure capability. Same edge, three possible fault assignments, three different engineering responses.

Their taxonomy visualizes this as a radial map: the model sits at the hub, with User, Harness, and Environment as an inner ring of families, and the specific components of each family arranged on an outer ring. Model-to-model interactions get grouped under Harness, alongside Context and Memory, because a model only ever talks to another model through its own harness. There's no direct model-to-model edge that bypasses scaffolding. Across this graph, the taxonomy catalogs 41 named failure modes as a structured set of edges, each with its own diagnostic signature.

Reading tool-call edges: malformed arguments, schema drift, and mistranslation

Start at the model-to-tool edge. Malformed Arguments means the call itself doesn't follow the required format, a model-side fault. Suboptimal Arguments means the call is technically valid but poorly chosen, also model-side. Incorrect Tool Selection means the agent picked the wrong tool from what was available, and Tool Hallucination means it tried to invoke a tool that doesn't exist. All four sit on the model side of the ledger.

The return edge, tool back to model, has its own set. Tool Feedback Neglect is when the model gets a response and doesn't act on it. Tool Recovery Failure is when a call fails and the model doesn't adjust. Both are model-side. But there's a third mode here that isn't: Mistranslation, where the integration layer garbles an otherwise correct observation or action as it crosses the model-environment boundary. That one's harness-side, and it's easy to mistake for a model reasoning failure if you're not looking closely at where the corruption actually occurred.

Schema drift is probably the most common version of this in production systems. Agents depend on exact shapes: exact field names, exact examples, exact validator rules. Changing one field name without updating every downstream consumer produces drift, which produces retries, which produces compensating logic bolted on to paper over the drift, which eventually produces a support ticket from someone who has no idea any of this happened upstream. What makes schema mismatch particularly dangerous is that the tool does return data, but it's data that no longer matches the contract the agent was promised. A single malformed argument at an early step can silently corrupt every step downstream that depends on its output, and nothing in the trace screams "error" until several steps later when the damage is already compounded. The 2025 paper "Butterfly Effects in Toolchains: A Comprehensive Analysis of Failed Parameter Filling in LLM Tool-Agent Systems" (arXiv:2507.15296) is dedicated to exactly this propagation mechanism.

Reading a trace at this edge comes down to one diagnostic question: was the tool call correctly formed, but the payload that came back violated the contract, or was the call itself malformed going in? The former is Mistranslation, a harness fix. The latter is Malformed Arguments, a model fix. Check three things side by side: the schema the model was actually given, what the tool actually returned, and what the model actually passed in the call. A harness-side Mistranslation leaves the model's reasoning perfectly intact; it just poisons everything the model touches after that point.

Reading workflow edges: retry loops, goal drift, and orchestration failures

Retry loops are one of the more expensive and more diagnosable failure patterns in agent traces. An agent misreads a response, or gets back a malformed payload, and it retries the same action. It hits the same wall. It retries again. Without a hard ceiling on how many times this can happen, the loop keeps going until the token budget runs out or the orchestration layer times out on its own. Each iteration eats into the context window, and as earlier reasoning gets pushed out of scope to make room, the agent can end up, by the time it finally stops, with no coherent memory of what it was even trying to do. The fix here isn't subtle: explicit retry limits, enforced at the orchestration layer, not left to the model to decide when enough is enough. Some agent workflows fail close to 41% of the time, and retry loops contribute to failures compounding once they begin.

A few named modes live specifically on orchestration edges. Goal Drift is when the objective the agent is actually pursuing diverges from the task it was originally given, usually over a long horizon where small deviations compound. State Tracking Failure is losing an accurate account of what's already happened in the run. Context Rationale Erosion is when reasoning that was perfectly sound several steps earlier becomes disconnected from the agent's current state, so the agent keeps acting on logic that no longer applies.

A formal model called "Binding Drift in Multi-Step Tool-Augmented Agents" uses four scripted personas to make this diagnosable rather than impressionistic. The Oracle stays correct the whole way through, zero drift. The Drifter is correct at step one but switches to a distractor at the final action step, producing a drift score of exactly one wrong action. The Propagator binds to a distractor right at step one and carries that error through every subsequent step, compounding it the whole way. The Guesser acts on ambiguous instructions instead of stopping to ask for clarification. Two behaviors matter for diagnosis here: whether the agent holds its initial binding through subsequent steps, and whether it stops to seek clarification when its state becomes ambiguous rather than proceeding on uncertain footing.

Reading the trace means checking whether the agent's stated goal, or the target of its tool calls, changed across steps without any user instruction accounting for the change. If the goal shifted and nothing in the input explains why, that's Goal Drift, and it's an orchestration or memory fault. If the same tool call repeats identically across several consecutive steps with no change in input, that's a retry loop, and it belongs to the orchestration layer, not to a failure of the model's reasoning.

Reading context and memory edges: compaction, staleness, and instruction loss

Context management deserves its own edge in the taxonomy because it isn't really a modeling decision. That grouping is doing real work: decisions about what stays in context and what gets dropped are harness decisions, full stop, regardless of how much they end up looking like a model choosing to ignore something.

Three named modes sit on these edges. State Staleness is the agent acting on stored state that's no longer accurate. Missed Write is an update that should have been saved but wasn't. Context Rationale Erosion, which also appeared on the orchestration edges above, occurs when valid earlier reasoning becomes disconnected from whatever the agent is looking at now.

Return to the compaction example from the opening. Context compaction is a harness design choice, and it's the harness, not the model, that decides what survives when the context window gets compressed to save space. If an instruction was correctly registered by the model at the time it was given, and later silently dropped by compaction, the resulting failure looks exactly like an instruction-following failure. It reads as the model ignoring the user. Because the instruction was delivered to the model at the time it was given and later silently dropped by compaction, the resulting failure looks exactly like an instruction-following failure and reads as the model ignoring the user, but it is not an instruction-following failure. The diagnostic here is mechanical: check whether the instruction is actually present in context at the step where the model fails to act on it. If it's gone, the fault sits on the harness-context edge, and no amount of model post-training will fix a harness that keeps deleting the thing the model needs to see.

Schema drift and context staleness are, structurally, the same failure wearing different clothes. Both involve the agent acting on a contract or a piece of state that no longer matches reality. The only real difference is where the stale information came from, a tool's response schema in one case, the memory store in the other.

Reading model-side faults: what the trace looks like when the model is the problem

None of this means the model is off the hook. There's a clean positive test for a genuine model-side fault: the instruction was present in context, the tool schema was correct, orchestration behaved as it should have, and the model still produced the wrong output or took the wrong action. When all the surrounding infrastructure checks out and the failure persists anyway, the fault has nowhere left to hide.

The named model-side modes cover a fair amount of ground. Instruction-Following Failure is what it sounds like: the instruction was there, it was unambiguous, and the model didn't follow it. Reasoning Failure is when the chain of reasoning is internally invalid given the evidence actually available to the model. Domain Knowledge Deficit is a gap in what the model knows, not a process failure. Over-initiative is the model acting beyond the scope of what it was asked to do. Unauthorized Irreversible Action is a model taking an action it had no authorization to take, which is a distinct and more serious category than ordinary overreach. Indirect Prompt Injection is adversarial content sitting in the environment that hijacks the model's behavior, arguably more of a security failure than a competence one, though the trace often looks similar either way.

Among reasoning failures specifically, fabricated evidence and evidential insufficiency are the most common patterns in production traces, with temporal misordering, spurious causal attribution, anchoring bias, straightforward logical fallacies, and internal contradictions also cataloged as recurring types.

Distinguishing these from context or harness problems in long trajectories is exactly the problem the TRAJDEBUG framework was built to solve. The insight driving it is that in a long trajectory, a failed step might conflict with a constraint the task stated many steps earlier, and judging that step as a genuine model error requires tracing it back to that distant piece of evidence, not just inspecting the step in isolation. TRAJDEBUG detects error triggers by identifying wrong commitments, points where the agent commits to something that conflicts with task instructions, trajectory history, environment feedback, or its own prior reasoning. That reframes error discovery from a holistic, gut-feel diagnosis into something closer to an auditable evidence-verification exercise, where every claim of "this was a model error" has to be backed by a specific conflict, traceable to a specific earlier point in the run. It's evaluated on TRAJERRORBENCH, a set of 486 manually annotated failed trajectories drawn from τ²-Bench and SWE-Bench Pro.

Why critical errors are rarely the most visible ones in the trace

The failure that draws attention in a trace review is usually the final wrong answer, the crashed tool call, or the obviously broken output, the one closest to the visible outcome. But the edge-based model of failure makes clear that the actual fault often sits several steps upstream of that visible moment, quietly, in a compaction decision or a schema mismatch or a goal that drifted three turns earlier without anyone noticing at the time.

This is why outcome-level review keeps missing root causes even when engineers are looking directly at the transcript. A malformed argument at step four doesn't announce itself; it just quietly corrupts step nine, and step nine is where everyone's attention lands because that's where things visibly broke. Fixing step nine changes nothing, because step nine was never broken on its own terms. The fault lived at the edge between the tool and the model at an earlier step, and until the review process is built to trace failures back along that interaction graph rather than stopping at the first visible symptom, the same category of failure will keep resurfacing, get patched at the wrong layer, and reappear in the next run wearing a slightly different face.