Degraded Agent Output With No Exception Trail Is a Harness Layer Attribution Problem
Silent agent failures trace to prompts, tools, and workflows, not infrastructure.

Silent tool failure detection in production multi-agent pipelines is a harness-layer attribution problem. Degraded output with no exception trail usually points to the prompt, tool, workflow, or memory layer, where behavior can drift while infrastructure stays healthy. Finding the responsible layer requires a full execution trace and controlled replay that tests whether a targeted change corrects the failure.
Why degraded agent output with no exception trail is a different class of problem
The agent runs, the job completes, and the output looks plausible. Users begin complaining days later about answers that are subtly wrong. Health checks remain green, and the exception logs offer no explanation. The application has failed without failing in the way its monitors recognize.
Wu’s June 2026 analysis found this pattern in 22 production incidents that had each gone through a full postmortem in a runtime that ran continuously. The recurring pattern, a failure whose error signal never reaches a human in actionable form, appeared at least 28 times. Thousands of unit tests and declarative governance checks stayed green. The gray failure literature calls this differential observability. The application suffers while the observer designed to detect the failure does not.
Wu calls the LLM-specific escalation “fail-plausible,” because the model can turn an internal error into coherent, contextually appropriate, false output. In one incident, a logging bug captured an HTTP 400 error page into a cache. The downstream LLM treated error strings as input signals, fabricated a confident industry analysis, and delivered it as a routine insight digest. No detector fired, and every test remained green. The absence of an exception was therefore diagnostic evidence about the failure path.
How the error compounds before anyone notices
A silent failure becomes harder to attribute as later steps consume its results. An agent can reason correctly, select the wrong context, and then act confidently on that context. The final output may look reasonable on its own. Its defect lies in the relationship between steps, so inspecting only the final response or the last tool call misses the source.
Wu found that roughly 70% of silent failures were caught in the end by people looking at the output as users saw it. Detection took hours to months, with latency linked to the failure mechanism. The longest-lived failures occupied seams between components, including deployment topology, cross-script contracts, and observer–observed coupling. Those seams had no tests covering their interactions. By the time a user noticed the result, the originating error could be several steps or agents removed from the visible defect.
TRAJDEBUG, introduced by Tsinghua / Tencent in August 2026, explains why simply finding an earlier mistake is insufficient. Failed trajectories contain local errors with different downstream effects. Some are repaired, some remain harmless, and others contribute to the terminal failure. The nearest error is not necessarily the cause. Attribution must distinguish an error’s presence from its role in the outcome.
Liu’s June 2026 study describes compounding through the Entropy Principle, under which disorder increases with interaction rounds when the relevant intrinsic properties of agent systems coexist. The study found the pattern in different architectures, models, and task domains. This makes interaction history part of the diagnostic evidence. Waiting for a complaint leaves the engineer reconstructing that history after the effect is visible in the output, whereas trace-level detection preserves the connections needed to identify where it began.
Why the failure almost always originates in a harness layer, not the model
The harness is the environment that governs what the agent can do with the model’s output. It includes available tools, visible files, retrieved context, stopping conditions, and the response when a tool returns unusable results. These controls shape behavior apart from the model weights. They can therefore cause failures in specific contexts while the infrastructure metrics used to judge system health stay the same.
Liu found that silent failures persisted in controlled environments with static prompts, identical models, fixed databases, and no external inputs. External drift is not a necessary cause. An unchanged model can participate in a failing sequence because the surrounding architecture governs how information is transmitted, retained, acted on, and checked. That finding shifts the initial diagnostic target from model replacement to the execution environment.
Liu organizes these failures across lifecycle layers that include inter-agent transmission, memory, execution, coordination, and verification. Each describes a different place where intended behavior can diverge from actual behavior without a weight change. In practice, the diagnostic signatures are often narrower than a general model-quality problem. A mismatched tool call or irrelevant retrieval result points to a particular execution boundary. The model remains a possible cause, but it should not displace a layer-specific explanation supported by the trace.
The three harness-layer failure categories that account for most silent degradation
Tool schema drift occurs when the interface an agent expects no longer matches the tool it calls. An external API may add a required field or deprecate an endpoint while the model continues using the old signature. The mismatch can precede a downstream reasoning failure by several steps. Tool-call patterns therefore show more than whether a request completed, and a schema anomaly can be visible in those patterns before the workflow stops producing output that serves the task. CI-side schema snapshot testing, which records schemas and compares them on each run, can provide warning before the change becomes a production incident.
Prompt ambiguity and prompt drift can change behavior while operational health stays the same. A provider’s serving infrastructure can change while the system prompt stays the same and latency, error counts, and token counts stay within expected ranges. A change intended to improve tone can widen the gap between intended and actual behavior, while an eval suite calibrated to the previous prompt configuration becomes an unreliable signal.
Workflow loops and false-success reporting break the relationship between an agent’s completion claim and the environment’s actual state. An agent may assert success while constraints remain violated or side effects invalidate another part of the task. Passing tests do not settle that discrepancy. TRAJDEBUG illustrates it with a CSV boolean export task in which an agent edits a forbidden file, then compensates with a patch that breaks JSON export, while all tests pass. The diagnostic target is the constraint-violating decision and its consequences, which may begin well before the final action.
Memory failures can participate in the mechanisms these categories identify when retrieval supplies context that does not fit the task, just as coordination failures can propagate a tool-layer defect between agents. The categories give engineers recognizable starting hypotheses that the execution trace must then distinguish.
What trace-level attribution requires that infrastructure observability cannot provide
Attribution-grade instrumentation must capture the full agent call tree, including inputs, intermediate decisions, tool arguments and results, retrieval queries, and document IDs. Latency and error counts cannot show whether the agent used the right information or met a task constraint. The relationships between spans are essential evidence. Without them, a valid-looking tool result cannot be connected to the decision that requested it or the later action that relied on it.
To keep detecting failures, three measurement layers need to work together: output quality scores per span, tool-call distribution baselines, and trajectory coherence checks. Quality scores kept over time show distribution shifts as well as single bad answers. Tool-call baselines expose departures from stable-period patterns. Coherence checks test whether the steps in the reasoning chain agree with one another, catching failures that look plausible when each span is viewed on its own.
REFLECT, presented by UCLA / Cisco at the ICML 2026 Workshop, formalizes four requirements for faithful attribution, namely execution grounding, prefix-preserving replay, targeted intervention, and inference-time computation. Execution grounding ties a diagnosis to what actually happened. Prefix-preserving replay keeps the earlier trajectory the same while a targeted intervention tests the suspect step. Inference-time computation supports the attribution process rather than reducing it to a stored label or a dashboard reading.
Long trajectories also need evidence from across instructions, observations, and earlier context. TRAJDEBUG uses multi-granularity history compression and evidence-based error identification to find that evidence. A point-in-time span inspection is not enough. Wu’s retrospective governance audit had a 0% ex-ante prevention rate but a high ex-post regression-blocking rate, illustrating a different limit. Periodic audits can block recurrence, while early detection needs evidence from the trace during or immediately after execution.
How do you debug wrong answers with no errors and attribute the responsible harness layer?
The practical method begins with the failed trajectory. An engineer should locate candidate divergence points and judge them against the task’s instructions, observations, and environment state. TRAJDEBUG divides this work into error trigger detection, error state classification, and critical attribution. Each stage answers a different causal question. The aim is to identify the earliest error linked to the final failure that was not subsequently repaired.
First, examine where execution departed from correct behavior, then follow what happened to that error. A local mistake that was repaired is not equivalent to one that shaped every later decision. An error near the final response may also have had no part in causing the failure. This lifecycle analysis prevents the first suspicious span from becoming the diagnosis merely because it is easy to find.
Next, map the causally relevant step to its responsible layer. A tool call with mismatched arguments or results points toward the tool layer, while retrieval of documents that do not fit the task points toward memory. Constraint violations or instruction drift implicate the prompt or workflow layer. Incorrect loop termination and unsupported completion assertions implicate workflow control. These signatures narrow the hypothesis without yet proving it.
REFLECT supplies the next step by testing a diagnosis-specific patch through controlled replay and using the outcome to refine attribution. The replay must preserve the execution prefix. The intervention begins at the diagnosed step, and execution continues from there. This limits the chance that unrelated earlier changes explain the result. An outcome change then becomes evidence for or against the proposed cause.
For an agent pipeline returning wrong answers without errors, the same method makes the call tree the unit of investigation. The proposed fix should then be replayed against traces of past failures before it ships. Recorded tool outputs and sandboxed side effects make replay more stable, while scheduled replay of a golden trace set against current production provides early warning. A synthetic benchmark calibrated to an earlier configuration cannot substitute for the execution history that exposed the defect.
Why harness-layer fixes, not model changes, are the right response to silent degradation
Once attribution identifies a harness defect, the corrective change belongs at that layer. Changing the model does not change a mismatched tool schema, a retrieval configuration, or an incorrect loop termination condition. A targeted harness patch addresses the mechanism supported by the evidence. Because it requires no weight update, it can also be evaluated and released on a shorter cycle than a new base or specialist model revision.
A genuine model capability gap remains possible. If the model cannot perform the reasoning the task requires, harness iteration alone will not resolve the failure. Trace evidence and controlled intervention distinguish a configuration gap from a capability gap, allowing the corrective path to follow the diagnosis instead of a preference for either harness changes or model changes.
Replay validation must extend beyond the single diagnosed case. A patch can correct one failed trajectory while introducing silent degradation in another trace subset that synthetic evals do not cover. Traces of past runs give engineers a way to test that risk. The engineering discipline is therefore to subject prompt changes, tool schema updates, retrieval changes, and workflow modifications to the same review and validation rigor as code changes. Silent degradation becomes a first-class reliability problem when completion is no longer accepted as proof of correctness.
Sources
- Silent Failure in LLM Agent Systems:The Entropy Principle and the InevitableDisorder of Autonomous Agents
- TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
- When Errors Become Narratives: A Longitudinal Taxonomy of Silent Failures in a Production LLM Agent Runtime
- TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?
- Silent Failure in LLM Agent Systems: The Entropy Principle and the Inevitable
- REFLECT: Intervention-Supported Error Attribution for Silent Failures in LLM Agent Traces


