The Agentic Harness

Mapping the Observability Gaps Where Multi-Agent Orchestrators Absorb Tool Errors Without Surfacing Them

Orchestrators quietly swallow tool failures, and this shows where.

· 8 min read
Cover illustration for “Mapping the Observability Gaps Where Multi-Agent Orchestrators Absorb Tool Errors Without Surfacing Them”
Agent Failure Diagnosis · November 14, 2026 · 8 min read · 1,882 words

Detecting silent tool failures in production multi-agent pipelines means tracing how errors change as they pass through the orchestrator as well as checking whether tool calls succeed. Orchestrators absorb failures through context rewriting, silent fallbacks, retry reclassification, and missing cross-agent verification. Detecting these mechanisms requires semantic handoff checks, schema comparisons, typed retry failures, and explicit verifier spans. The corrective work belongs in the harness that coordinates agents, tools, and checks.

Why multi-agent pipelines return wrong answers but no errors

ADE-PRF, Dexing Liu’s July 2026 reliability framework, calls the condition “false prosperity [1][2][3].” Every agent can report success, every API call can return HTTP 200, and the final response can be confidently formatted yet factually wrong. External metrics remain normal while output quality degrades. By the time someone detects that degradation manually, substantial technical debt may have accumulated.

This is a mismatch between monitoring and correctness. Infrastructure monitoring tracks CPU, memory, and API success rates, while LLM-specific platforms such as LangSmith record prompts, token counts, and tool invocations. ADE-PRF distinguishes those observations from assessing whether an agent’s output is semantically correct. A successful request establishes that an exchange completed. In a multi-agent workflow, that distinction extends across every handoff, because downstream agents can accept and reformat an upstream error without producing a new exception.

Single-agent failures are more often local and legible. A wrong answer, timeout, or failed call points to a limited set of steps in the execution path. Multi-agent orchestration distributes the cause across agents and spans, so the visible symptom may appear well after the operation that introduced it.

The MAST taxonomy as the right starting frame for absorption mechanisms

Cemri et al.’s 2025 MAST taxonomy gives names to these structural failures so they can be told apart. Its categories are Specification Issues, Inter-Agent Misalignment, and Task Verification. For the absorption mechanisms examined here, those categories frame specification ambiguity, coordination breakdowns, and verification gaps. These are related problems, but they require different evidence. Treating them all as poor model output obscures where the workflow lost correctness.

Specification ambiguity arises when agents read a shared task or context in different ways. Neither agent necessarily violates its own prompt. The disagreement lives between their instructions or interpretations, allowing a confidently wrong report to emerge without an erroneous individual span. Coordination breakdowns instead involve dependencies between actions, including an agent assuming state that another agent has not yet written. A broker dropping a task under high load without any agent retrying it is another coordination failure, one that may not appear in pre-production tests.

Task Verification directs attention to who checks the combined result. The quality of each prompt, task completion rate, and latency cannot show whether anyone has that responsibility. Schema fallbacks and retry reclassification must still be examined at the tool and harness layers where they occur.

How specification ambiguity at agent boundaries rewrites intent without leaving a trace

Specification ambiguity becomes an absorption mechanism when an orchestrator turns a bad upstream output into a plausible downstream input. The change can keep the syntax intact while changing the meaning. An agent receiving the rewritten input can then complete its assigned task correctly relative to that input, even though the workflow has departed from the original intent. The error comes from the handoff, while each agent on its own may be working correctly.

Context rewriting makes this loss difficult to reconstruct. When a tool returns partial or malformed output, the orchestrator may reformat it into syntactically valid input for the next agent. A later span then contains acceptable input rather than the malformed material that prompted the transformation. Confident reformatting has made the workflow easier to continue and harder to diagnose.

Structural tracing shows that agent B got input from agent A. Whether B understood what A intended to communicate is a separate semantic question. Type checking can establish that the handoff has the required shape, but shape alone cannot establish agreement about the task or shared context.

How tool schema drift produces silent fallbacks that look like successful completions

Tool schema drift moves the same problem from shared intent to the contract governing a tool call. When that contract changes, agents may default silently, adapt incorrectly, or describe an action that only partially happened as complete. Prompts that emphasize helpfulness, evaluations that emphasize task finish rate, and wrappers that smooth over errors all favor continuation over safe abstention.

SilentProbe’s 2026 audit of OpenAPI documents found that a large share of parameters lacked machine-checkable constraints. Many documents also expressed constraints in prose that their schemas did not encode. Agents must interpret those requirements. Consequently, even a schema that has not changed can leave room for inconsistent behavior, while a changed schema adds disagreement between the contract the agent expects and the one the tool exposes.

AliveMCP’s Q2 2026 audit found schema hash changes within days among a meaningful share of healthy MCP servers. Infrastructure health did not establish contract stability. A server could return tools/list successfully while downstream agents retained stale definitions. Calls could then fail loudly or fall back to a “no tools match” path that produced well-formed but wrong output.

How the retry harness turns transient-error handling into an absorption amplifier

A schema mismatch can generate an exception and still disappear inside the retry harness. The decisive step is how the harness classifies it. If a structural mismatch enters the same bucket as a network timeout, retrying treats an incompatible response as a temporary interruption. Once the retry budget is exhausted, routing around the tool can convert that unresolved defect into a degraded answer.

The documented fourteen-day case followed that sequence. A changed response shape caused a tool-execution exception, which the harness classified as transient. After several retries, the agent reported the tool as unavailable, and the orchestrator chose another path. The user received a worse answer with no error reported. The workflow continued in that state for two weeks because the resilience path prevented the structural failure from surfacing.

Retry logic is appropriate for genuinely transient failures. Its absorptive behavior begins when classification erases the distinction between temporary unavailability and a broken contract. ADE-PRF identifies a related attribution problem in multi-agent calls, where efficiency monitoring records tool activity but loses its connection to the originating agent. A retry count alone explains neither the initiating defect nor the consequence of the fallback. Tracing only the orchestrator compounds that loss when specialist agents and their tool calls remain outside the recorded execution path.

How verification gaps let orchestrators absorb errors by never assigning ownership of cross-agent checking

Context rewriting, stale schemas, and misclassified retries can all produce outputs that appear acceptable to the next agent. Without a designated cross-agent verifier, no component is responsible for challenging that appearance. Responsibility is distributed, but the check is absent.

In a single-agent system, verification can be attached directly to that agent’s work. A multi-agent pipeline needs a clear assignment of who checks the combined result across boundaries. Otherwise, an intermediate hallucination can corrupt subsequent steps while the final output gives little indication of where the fabrication began. Production runs can span dozens to over a hundred turns. An early mistake may therefore become visible only after a long chain of otherwise successful operations.

The security implications overlap with this operational gap. The Crew Scaler MAS security study in 2026 found that Non-Determinism and Data Leakage were the risk categories least addressed by the sixteen security frameworks it reviewed. That finding supports attention to monitoring gaps without making security the primary explanation for ordinary correctness failures.

Why root cause attribution collapses in multi-agent traces and what that costs in practice

Even a detailed trace can leave attribution unresolved. The initiating error may be sparse, distributed across handoffs, and far removed from the visible symptom. Investigators must distinguish the operation that introduced the defect from the later operations that reformatted, retried, or propagated it.

TraceTraj results show particularly weak step attribution with both Claude and GPT-OSS when the root cause can precede the final symptom by a long interval. The practical cost is uncertainty about whether a harness change would address the cause or its last visible consequence.

MAST, ADE-PRF, and AgentDebugX collectively reflect attention to classification, runtime evidence, and attribution, without being interchangeable frameworks. AgentDebugX specifies five requirements for closed-loop attribution. They include low-friction capture, portable OpenTelemetry GenAI export, typed diagnoses containing root cause, evidence, confidence, and fix, local-first storage with scrubbing, and cost-aware analysis with free deterministic triage and opt-in LLM depth. These requirements connect evidence collection to a diagnosis that can guide intervention.

Instrumenting against each absorption mechanism: what to measure at which layer

Against specification ambiguity, every agent-to-agent boundary needs semantic validation as well as type checking. The relevant comparison is between what the upstream agent intended to communicate and what the downstream agent receives. Terminal-output evaluation can detect a wrong answer without locating the handoff that changed its meaning.

Against schema drift, the tool-call log should record the schema version cached by the agent at invocation time. Recording only the server’s current version loses the comparison. A hash comparison between cached and live schemas at the point of invocation exposes drift before fallback behavior can conceal it. This distinguishes a healthy server from a compatible tool contract, which are separate conditions.

Against retry absorption, retry spans need a failure-class attribute distinguishing transient from structural failures. Retry count is not enough. The classification preserves whether the harness is recovering from temporary unavailability or repeatedly encountering an incompatible response. Without it, a prolonged schema mismatch can resemble a resolved network interruption.

Against verification gaps, the workflow must assign cross-agent verification explicitly and record its output as a named span. Its absence should itself produce a signal. OpenTelemetry GenAI semantic conventions provide a vendor-neutral default for exporting this evidence into existing observability stacks. Portability supports analysis, but it does not replace the mechanism-specific checks that make the exported trace useful.

Harness engineering as the intervention layer once absorption mechanisms are visible

Once traces locate the absorption mechanism, the intervention belongs in the harness. Prompts, tool schemas, workflows, memory, and verification assignments govern the behavior exposed by these failures. Model retraining does not directly repair a stale contract or assign an absent verifier. OpenAI, Anthropic, and LangChain’s engineering guidance converges on explicit harness loops, tool contracts, trace replay, evaluation suites, context budgets, and controlled execution boundaries.

The appropriate change follows the diagnosed cause. Encoding vocabulary and constraints in a tool schema removes ambiguity at the contract and the need for a prompt to compensate for it. Assigning verification in the workflow likewise gives someone the job of checking, instead of relying on agents to check their own work. These changes address the coordination structure.

Replay provides the bridge from diagnosis to validation. With recorded tool outputs, sub-agent decisions, and intermediate state, a proposed harness change can be tested with sandboxed side effects before it is deployed. The LangFuse case illustrates that process. A tool-schema breakage on a code-gen agent was caught on canary tenants, and the on-call engineer validated the candidate fix against twenty failed traces before deploy. Harness engineering should make that loop based on evidence as thorough and repeatable as code review and testing, so structural failures that keep coming back lead to a systematic review as well as fixes for each case.

Sources

  1. Agent Delivery Engineering Predictive Reliability Framework
  2. Security Considerations for Multi-agent Systems
  3. AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  4. Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures

More in Agent Failure Diagnosis