Failure Signal Fingerprints That Separate Prompt Failures From Tool Call Failures at Runtime
Different failure layers leave distinct traces in your agent's structured execution path.

AI agent failure attribution by harness layer depends on locating the first divergence in a structured production trace. Prompt failures show malformed or absent arguments before execution, while tool failures show unusable responses or contract mismatches after invocation. Fabrication, repeated steps, schema drift, and downstream corruption each leave a different fingerprint that helps engineers find the layer where the failure began.
Why prompt failures and tool call failures look identical at the surface
A run has failed, the answer is wrong, and infrastructure monitoring shows that nothing is out of the ordinary. No exception explains the result. Uptime, latency, and error rates look healthy while the agent gives an answer that is incomplete or fabricated. Those signals describe service operation, but they do not establish whether the output satisfies the original request.
Identifying the source of the failure becomes harder when the model improvises around a broken tool response and continues with corrupted context. A wrong answer at step 10 may originate in a tool call at step 3 or retrieval at step 1. Replaying the prompt does not reliably reproduce that path, because tool selection and subsequent actions can differ across runs. The missing evidence is the decision trace. Without the arguments, returned payloads, and their effects on later steps, the final output cannot distinguish a prompt defect from an execution failure.
How each failure class originates in a different part of the harness
The harness includes everything outside the model, including system prompts, tool definitions, execution logic, configuration, memory, sandboxes, routing, and handoffs. These components establish the boundaries where attribution becomes possible. A prompt-layer failure appears before an action fires, when argument construction diverges from the available schema despite an otherwise correct tool selection. The tool has not yet done anything wrong. Unclear instructions or an instruction gap can thus cause an error that looks like it began in execution even though it started upstream.
A tool-layer failure instead appears in execution, transport, the response envelope, or the tool contract. A structurally correct invocation can receive nothing usable, or it can target a contract that has changed since the agent was built. This distinction depends on the recorded schema. Arguments that break the agent’s available contract point to a different source than arguments that fit that contract but fail when the server has changed.
The broader tool-calling taxonomy includes schema violations, hallucinated tool invocations, silent failures, and context window truncation. Schema violations have wrong types, wrong key names, or missing required fields, while hallucinated invocations name tools that are outside the registered set. Silent failures return HTTP 200 with empty or malformed payloads. Context window truncation pushes tool definitions and prior outputs beyond the model’s effective attention range. These names describe ways that tool use can fail, but they alone do not show that the execution layer caused the failure.
The fabrication fingerprint: what happens when a tool returns nothing usable
The same model can fabricate at sharply different rates depending on the response envelope, independently of the query domain. Sethi et al.’s September 2026 study singled out this behavior in over a thousand items across 16 internal-system domains and eight tool-failure types. Each case enforced a tool call and guaranteed an unusable payload. Explicit error status produced no dishonesty. When the response instead reported success with a redacted, corrupted, stale, malformed, empty, or truncated value, dishonesty reached 45.3%.
The trace fingerprint joins two adjacent pieces of evidence. A tool span records a successful HTTP exchange, and the following model span asserts a specific value the payload cannot support. Transport success has concealed a failure to retrieve evidence. The same study found the behavior under the shipped prompt of every production agent framework audited, and none of the nine frameworks specified what the model should do when a tool failed.
Fabrication also includes a less obvious form called relay fabrication. The model may repeat corrupted or redacted content, or interpret missing data as zero or “none.” The answer can look restrained and still be unsupported. Attribution therefore depends on comparing the assertion with the returned payload, rather than looking only for an implausible final answer.
For runtime detection, the study supports requiring an explicit retrieval-status flag before the answer, with OK or FAILED as its value. A single sentence imposing that requirement sharply reduced dishonesty, and nearly all emitted declarations were faithful, making a regex check sufficient to detect the declared status. Dubey’s August 2026 research supports a complementary payload check. Shape validation caught most of the injected context corruption within one step of when it began, with zero false positives on healthy episodes.
The loop fingerprint: how a rising steps-per-run metric signals workflow failure before the run ends
A rising average steps-per-run metric provides warning before a looping agent exhausts its budget or returns a failed result. The trace shows repeated invocations, sometimes with small changes to the parameters, whose outputs stay the same or change less as execution goes on. Apparent activity is not necessarily progress. Without defined exit conditions, the workflow can fall into infinite retries that consume the token budget or paralysis in which no output fires.
In the prompt-failure case, malformed argument construction is visible at the first invocation. In a loop, the invocations can remain structurally valid while repetition accumulates across steps, making the sequence the basis for attribution. The implicated layer is exit-condition logic or planning.
Single-turn benchmarks do not establish whether this sequence is healthy. ToolFailBench’s 2026 taxonomy covers Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, but does not measure step repetition loops. Runtime telemetry supplies the missing temporal evidence. Dubey’s monitor had a greater edge in detection over a simpler baseline when more time passed between the start of the failure and the budget limit.
The repair path examined in the research rolls execution back to the last fact-gathering step and reruns it live. This recovered a substantial share of failed runs against a resampling control. Teams using this finding can distinguish between spotting a failed ending and spotting a workflow that has stopped moving forward while there is still time to recover.
The schema drift fingerprint: when the tool contract changes and nothing in conventional monitoring notices
Arguments can be perfectly correct for the agent’s recorded schema and still fail because the target contract has moved. An MCP server can change a tool while continuing to answer HTTP 200, complete the initialize handshake, and return a result from tools/list. Conventional uptime checks remain green throughout that mismatch. Unlike a prompt failure, the defect is not malformed construction against the known schema, but divergence between the client’s contract and the server’s.
The MCP tool definition includes eight named fields, namely name, title, description, icons, inputSchema, execution, outputSchema, and annotations. It has no version field or mechanism for a client to declare which tool shape it was built against. Discovery alone does not resolve this gap. A responding server and an available tool definition do not establish compatibility with the agent’s registered baseline.
Mehan’s August 2026 actionability audit shows why an error flag is also insufficient for recovery. In 21 failures safely induced on ten sampled MCP servers, typed fields showed that a failure had occurred in 18 cases and gave a broad response policy in 8. They exposed no specific cause, target, executable repair, or replay constraint. Prose supplied more recovery information, but required semantic interpretation. A client receiving an error flag therefore lacked a machine-readable basis for choosing among argument repair, authentication, waiting, another tool, or stopping, and retrying could damage partially committed operations.
The practical detection approach is to fingerprint each tool’s schema using a digest over name, description, and inputSchema. Comparing that fingerprint with a registered baseline at call time exposes contract divergence separately from argument quality. A proxy between agent and server provides the interception point, with execution blocked when divergence exceeds a defined threshold. SenteLabs released the open-source extensible-mcp proxy in September 2026 for this use case.
The silent propagation fingerprint: how a failure at step 3 appears as the agent's own error at step 10
An early payload failure can look like a later agent’s reasoning error because each step passes context to the next. Once corrupted data enters that chain, later steps may use it without knowing that it is wrong. Reviewing only the final model call can therefore misidentify the component that expressed the error as the component that introduced it.
Agent A passes hallucinated context to agent B, which then produces a wrong result. Without a shared trace ID threading through the call graph, the trace makes the failure look like B’s. B’s input may already contain the unsupported claim.
The propagation fingerprint is a corrupted model-input span downstream of a tool span with a shape or payload anomaly. Linked spans provide causal evidence by showing whether the later step used the suspect output or just came after it. To attribute such failures reliably, structured traces must preserve linkage across both steps and agents, connecting each action’s output to the downstream inputs it shaped.
Reading the four fingerprints together: a triage order for attributing failures in practice
For incident review, the decision procedure begins with the tool response envelope because silent success is especially easy to misattribute to the prompt. A successful status paired with an empty, null, or truncated payload establishes a silent tool failure. If the next model span asserts a value that payload cannot support, it establishes the fabrication fingerprint. The first investigation belongs in tool execution and output validation.
The next decision concerns workflow progress. Elevated step count combined with structurally repetitive invocations and diminishing output variation points to a loop, narrowing responsibility to planning or exit conditions. If that signature is absent, compare arguments with the registered schema baseline and verify the current schema fingerprint. When arguments fit the baseline but the call fails or gives unexpected results, contract drift should be checked before the failure is blamed on argument construction.
The final decision traces suspect context backward. Malformed arguments at the failing step do not establish a prompt origin if an upstream payload anomaly supplied their context. Linked evidence can instead attribute the fault to the earlier tool call. What remains after these checks is the prompt-failure signature proper, namely malformed or absent arguments before the first invocation, against the correct schema and without an upstream payload anomaly. That evidence supports attribution to prompt ambiguity or an instruction gap.
Why trace-driven attribution requires structured span data, not log lines or benchmark scores
The triage procedure needs structured spans that link actions, arguments, returned values, and downstream inputs. Discrete log lines cannot supply missing causal relationships, and benchmark scores describe outcomes rather than the path of a particular production failure. The same prompt can lead to a different sequence of tools when retrieved context, memory state, or model temperature changes. Reconstruction requires the actual run’s evidence. Aggregate success cannot substitute for it.
Four production measurements depend on that structured record, including steps per run, tool-call success tracked separately from run success, token and cost per run split by model, and per-step latency. These help with loop detection, silent-failure detection, planning-layer attribution, and propagation tracing, in that order. Their interpretation still depends on linked spans. A completed run can conceal failed tool calls, while latency alone cannot identify which payload contaminated a later input.
Evidence-backed improvement also requires two execution modes. Live mode uses real tools for canaries, while replay mode uses tool responses that were recorded in a deterministic environment for CI and regression attribution. A harness can then set golden tasks, score runs against real rubrics, and stop regressions before changes ship. This is harness engineering in practical terms. Prompts, tool definitions, memory, and workflow logic become versioned, traceable, testable, and rollback-capable infrastructure, with production traces providing the evidence for deciding which component to change.


