The Agentic Harness

Instrumenting Tool Call Return Paths to Surface Silent Failures Before Orchestrators Propagate Bad Output

Validate tool outputs at the return boundary before orchestrators trust them.

· 9 min read
Cover illustration for “Instrumenting Tool Call Return Paths to Surface Silent Failures Before Orchestrators Propagate Bad Output”
Agent Failure Diagnosis · October 24, 2026 · 9 min read · 2,051 words

Silent tool failure detection in production multi-agent pipelines requires validating tool outputs before the orchestrator consumes them, rather than treating HTTP success as proof of correctness. Instrumenting tool call return paths means asserting schema, semantic validity, and side-effect confirmation at that boundary. Linked traces then make failures attributable and replayable, so a bad return value becomes evidence for a fix and is intercepted before reaching the next agent.

Why tool call return paths are the blind spot that standard agent observability leaves open

A Salesforce API accepts a hallucinated field mapping and returns a 200 OK at normal latency. Datadog records a successful transaction, with no stack trace or 500 error. In this scenario, the monitoring system correctly reports transport success but says nothing about whether the accepted mapping was valid. The unmonitored seam is between the tool completing and the orchestrator trusting its output.

Traditional APM is built around deterministic code paths, where 5xx errors and latency spikes provide useful failure signals. A semantically wrong return value can come back without either. Standard LLM tracing leaves a related gap when it watches a prompt go in and text come out without covering the surrounding execution. Missing domains include reasoning before action, exact tool arguments and outputs, external state changes, and retry or fallback paths. Those paths can hide systematic failure when a run succeeds in the end, so a correct run and an incorrect run look alike on an operational dashboard.

How do you know if a tool call failed silently inside a multi-agent workflow?

Four distinct mechanisms explain why a clean response cannot establish that a tool call worked. The first is tool-call drift and schema violation. A function expecting an integer user_id and an ISO8601 date_range instead receives userId set to "last_week". The tool may misinterpret the arguments or return null while the orchestrator proceeds. Detection therefore requires inspecting the contract and payload.

The second mechanism is a workflow loop. An agent that receives a rate-limit error as text may immediately repeat the identical call, skipping backoff. The symptom is elevated tool volume, which can occur without an operational error. The third mechanism is semantic omission caused by prompt ambiguity, where the returned result exposes something the instruction failed to specify. Retry frequency can therefore implicate the prompt rather than tool instability, even though the symptom appears at the tool boundary.

The fourth mechanism is fail-plausible output. Wei Wu’s 2026 study of 22 fully documented production incidents describes an HTTP 400 error page entering a cache and becoming a confident industry analysis about a “Hugging Face platform crisis.” The downstream LLM delivered it as a routine digest. No detector fired, and every test stayed green. Wu calls this “fail-plausible” because the failure does not merely withhold a useful signal from a detector. It gives the human a counterfeit account instead.

Multi-agent handoffs add a semantic seam across these mechanisms. A span can establish that agent B received input from agent A without establishing that B interpreted it as A intended. Receipt is not evidence of shared meaning. Structural tracing shows that the transfer took place, but explicit semantic checks are needed to find the mismatch.

What the consequences look like once a bad return value propagates unchecked

Diagram: How a Silent Tool Failure Propagates Unchecked. Visualizes: Show the blast-radius timeline of a silent failure moving through a multi-agent pipeline.

The propagation window determines the blast radius of a silent failure. By the time anyone notices, several irreversible actions may already have executed. In the deprecated-SKU scenario, an agent recommended a deprecated SKU to two thousand customers over three weeks. No exception fired, no latency alarm sounded, and no log marked the output as suspicious. An internal audit discovered the failure after the recommendations had already reached customers.

Wu found incident latency ranging from hours to months of silence in the production runtime studied. The failures that lasted longest were at seams between components where no test was designed to run. Roughly 70% were ultimately caught by human observation of system output rather than unit tests, health checks, or governance audits. Detection depended on someone noticing an implausible result.

Multi-step chains make that gap more costly because later calls use the return values from earlier ones. A single wrong result can become the premise for subsequent decisions, while further errors accumulate before someone notices an implausible result in the system output. The orchestrator’s trust is the link between them. Once an upstream value has been accepted, downstream execution can be locally consistent with an already incorrect premise, so interception after the chain finishes leaves the propagation window open.

Treating the return path as a validation layer — what to assert before the orchestrator consumes a value

The engineering response is to make the return boundary an active validation layer. This is the point where schema, semantic validity, and side-effect confirmation can be checked together before the orchestrator receives the result. Validate each return value against its expected schema. If a function returns null or an unexpected type where a string is required, the contract violation should fail loudly there. Logging records the value, while assertion prevents an invalid value from advancing.

Schema conformance is necessary but insufficient for acceptance. A date can parse correctly yet fall outside a plausible range, and a correctly formatted user ID can refer to a deleted record. These need semantic checks to see whether the values mean what they should. Faithfulness evaluators also compare the agent’s downstream response with what the tool actually returned, exposing drift between evidence and narrative. Boundary checks belong before consumption, while response-level comparisons can move from a daily batch to near-real-time sampling.

For state-changing tools, validate the outcome as well as the response. Database writes, API POST requests, and calendar entries need side-effect confirmation that the expected change occurred. The invocation record should include the tool name and version, exact input arguments, raw output, latency, call status, and a side-effect marker. A hash of the environment fixture is also required for replay.

Loop detection needs evidence that shows how repeated decisions are linked as well as a count of failed calls. Record the tool name, arguments hash, retry index, and parent decision id as span attributes for every tool decision. This makes it possible to see where the same arguments recur along the retry path. Set a threshold for tool-loop depth for each task class and alert when it is exceeded, so a sequence that seems to succeed cannot hide harmful repetition.

How to emit return-path spans that are actually useful for debugging and regression detection

Use OpenTelemetry as the foundation for the span model, with GenAI semantic conventions for shared attributes. As of 2026, those conventions remain at Development status and have yet to reach Stable status. Their standard gen_ai.* attributes cover model name, token counts, finish reason, and tool calls, with telemetry routable to Datadog, Honeycomb, or Grafana Tempo. Backend choice need not dictate the instrumentation model. With the common format, traces can follow a run through the edge, LLM, tool, and database.

Each tool interaction needs its own span with invocation metrics, parameter payloads, return values, status, and latency data. Make that span a child of the parent reasoning or planning span, preserving the chain from the LLM reasoning step to tool selection and execution. This hierarchy helps show whether the tool gave a bad response, the wrong tool was chosen, or an argument was malformed. A shared trace ID must extend across agent boundaries so the execution tree remains reconstructable when the symptom appears elsewhere.

Semantic alerts should supplement error rate, latency, and saturation. Refusal rate above baseline by route can indicate a prompt regression or model behavior change after a vendor update. Tool-call entropy above a per-session threshold shows that the agent is exploring rather than executing, and a shift in the output-length distribution from a rolling seven-day baseline can be a sign of a context-window regression. These are different signals from transport failure. Run semantic evaluators over a sampled stream of one to ten percent of traffic off the hot path to detect population-level shifts within minutes.

That sampled monitoring serves a different purpose from blocking invalid individual returns. It detects changes across traffic that a single invocation may not reveal. Latency also needs this finer separation. Split alarms by planning, retrieval, tool, and generation stages, as well as by route, because tail latency in agent systems is dominated by retries, tool calls, and reasoning loops, with the base model call contributing less. An aggregate alarm conceals which stage is consuming the time.

Using return-path traces to run replay validation before shipping harness changes

Use the return-path traces that were captured to test harness changes against tool sequences that worked before. Replaying only the LLM call leaves step sequences, side effects, and state outside the test. The boundary record supplies the missing execution context. Its payloads, version, status, latency, side-effect marker, and environment fixture hash form the minimum basis for deterministic replay. Without that record, a replay cannot show whether the tool’s past response still matches what the live tool returns.

A weekly sample of recent production traces can be rerun against the same inputs in a sandbox, with new outputs compared against historical outputs. A shift above a threshold should trigger an alert and go into a triage queue before the proposed change reaches users. Distribution changes deserve attention even without HTTP errors. A meaningful week-over-week shift in tool calls, or a small increase in schema failures, can indicate that tool behavior has changed beneath an apparently stable interface.

Manus rewrote its harness four times in less than a year, using the same underlying models across five architectures. Each rewrite improved reliability and task completion. Harness design was the variable that changed. GPT-5.5 also scored differently on functionality tests in OpenAI’s native Codex harness and Cursor’s harness in the same week. Claude Opus 4.7 improved when moved from its native harness to Cursor’s, further illustrating why a failure should not automatically be assigned to model capability.

Notion rebuilt its agent harness several times, kept infrastructure uptime high, and cut the cost of Custom Agent runs by a meaningful amount in testing. That production example locates reliability work in the execution environment as well as in model selection. Replay makes those changes testable before release. The return path still needs continuous monitoring after release because a tool’s schema, behavior, or third-party dependency can change without an agent-side deployment. Prelaunch evaluation cannot detect drift that has not happened yet.

Attributing a return-path failure to the right layer so the fix addresses the actual cause

Attribution should distinguish the prompt, tool schema, workflow logic, external dependency, and model interpretation. Ambiguous instructions can produce malformed arguments, while an unannounced schema change can invalidate previously correct calls. A retry path may neglect to check for prior success, a third-party API may change its contract, or the model may hallucinate an argument value. These causes require fixes in different places. A trace that says only “tool call failed” creates a triage item without identifying what the responsible engineer should change.

Microsoft’s AI Red Team published an updated agentic failure taxonomy in June 2026, adding seven categories including supply chain compromise, goal hijacking, and MCP/plugin abuse. Memory poisoning was already present in v1.0. A successful injection can propagate bad instructions into subsequent sessions, where each failure appears to arise from normal operation. Finding where it began therefore requires tracing the injection point across session boundaries, since the defect may have entered before the current tool call.

Cursor 3.6, shipped May 29, 2026, has a gate-based architecture that shows where calls can be intercepted. Allowlisted calls run automatically, sandboxable calls execute in isolation, and remaining calls go to a classifier subagent that decides whether to allow, retry, or ask for approval. Each stage produces an attribution record identifying the gate and reason for interception. The decision is part of the evidence.

Wu’s fail-plausible incident demonstrates the complementary need to preserve evidence before the final narrative replaces it. The cached error and the confident digest belonged to the same failure chain, yet passing tests did not expose that relationship. The return-path record makes the discrepancy inspectable. Treating that evidence with the rigor of code review and testing turns harness changes into specific, testable repairs. Return-path instrumentation is therefore not an observability add-on, but the foundation of a repeatable process for improving agent reliability.

Sources

  1. When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime

More in Agent Failure Diagnosis