CrewAI Tool Call Autopsies Using Harness Layer Traces When No Exception Is Ever Raised
Harness-layer traces reveal which layer failed when CrewAI tools never raise exceptions.

To detect silent tool failures in production multi-agent pipelines, harness-layer traces must show whether tools succeeded as well as whether they returned. In CrewAI, a valid-looking answer does not establish that a tool ran, worked, or passed its result intact to another agent. The autopsy must link provider responses, tool execution, handoff payloads, and task termination to find which layer caused the failure.
Why CrewAI tool failures are invisible by design, not by accident
An ordinary HTTP failure can leave a stack trace or a non-2xx status code. A wrapped CrewAI tool failure can leave a sentence. When a tool raises, the wrapper catches the exception and returns a string such as “Error: connection timed out”. That string becomes an observation, which the model reasons over just as it would a successful result.
The agent is built to handle ambiguity and produce a plausible answer, so routing around a failed tool can look like fulfilling its contract. The run ends with well-formatted output even though the work was never done, and there may be no outward exception or sign in the logs. This is a structural property of the execution path. Ordinary service debugging follows propagated errors, while this autopsy must follow errors that became input.
The four distinct shapes a silent failure takes in a CrewAI harness
Wrapper absorption, also called write-around, occurs when a tool raises, its wrapper returns an error string, and the model uses that string to produce a confident answer. Execution returned control, but the work failed. Its diagnostic signature is the gap between a recorded return and a successful operation. A typed result that crew logic inspects before the model sees it allows failure to stop the task.
Fabricated observation, or hallucinated tool use, occurs when the model generates a valid-looking sequence of thought, action, observation, and final answer without invoking the tool. The observation has no runtime source. CrewAI community reports describe cases with no tool usage in logs or the Phoenix interface, no execution of run(), no side effects, and no memory trace. Reports cover both Qwen2.5-Instruct and GPT-4, so this is not confined to a custom-hosted model.
Silent repair occurs when malformed or truncated JSON is accepted and repaired at the schema boundary. A usable answer can conceal this defect. The repair runs whenever there is a value to salvage. Cases K3 and K4 land on the right value but undergo the same silent repair. The relevant evidence is the discrepancy between submitted arguments and schema-required arguments.
Policy-to-event-bus gap occurs when failure handling bypasses both an explicit policy and its expected event path. A flow tool action raised ToolExecutionFailedError on every detected ToolFailure, even for tools configured with WARN or IGNORE. Because the raise bypassed handle_tool_failure, no ToolFailureDetectedEvent appeared. The flow could visibly abort while the event bus remained silent. PR #8426, filed in October 2026, introduced routing through that handler for explicit policies, allowing WARN to continue and emit the event, IGNORE to remain silent, and RAISE to abort.
Why the Responses API provider mismatch creates a fifth silent failure class engineers miss
A missing tool execution can also originate below agent reasoning. The model may issue a real tool call that the provider integration fails to recognize. This is provider-shape mismatch, a fifth failure class. Its output-layer evidence can resemble fabricated observation because the tool never runs, but changing the agent’s instructions does not address the same cause.
A July 2026 CrewAI community bug report describes agents using the GPT-5.6 family with tools failing on the first LLM call through the native OpenAI provider in crewai v1.15.1. The affected family includes gpt-5.6-sol, gpt-5.6-terra, and gpt-5.6-luna. The api="responses" flag already exists in crewai/llms/providers/openai/completion.py, but it is not automatically selected for models that require it. Endpoint selection is therefore part of the autopsy. It precedes any conclusion about whether the agent chose to use its tools.
Making the Responses API path work exposed three compounding bugs, including a silent recognition failure. The functions is_tool_call_list() and extract_tool_call_info() in crewai/utilities/agent_utils.py recognize several providers’ tool-call formats but not the Responses API shape containing id, name, and arguments. A genuine call is consequently stringified and returned as the agent’s final answer without execution or an error. The other two bugs are in the Responses API tool-calling loop. For attribution, the decisive comparison is between the provider’s actual response and the harness’s interpretation of it.
What the trace record actually contains and where the critical attributes are hidden by default
OpenTelemetry’s GenAI conventions, as of v1.41, define agent, workflow, tool, and inference spans. Their operations include invoke_agent, invoke_workflow, execute_tool, and chat or generate_content, with latency and token-usage metrics. Delegation can appear as its own span, distinguishing a single handoff from repeated delegation. These records describe the run’s execution structure. However, the conventions have Development status, are not marked required, and their gen_ai.* attribute names can change without a major version bump.
The most diagnostic tool-boundary fields are not automatically available. The arguments the model selected and the raw result it received are Opt-In attributes. Tracing can therefore be enabled while the evidence needed for an autopsy is absent. A tool span establishes a boundary in the record, but without its payload it cannot explain what crossed that boundary. Turning on recording for those fields after a failure cannot bring back the original exchange.
At a minimum, the record must keep each invocation’s arguments, raw return, and success status separate. It also preserves the exact payload crossing each agent handoff, crew-wide provider call and retry counts, and each task’s termination reason, whether completed, max iterations, or error. Parentage needs attention as well. When context fails to propagate across delegation or framework boundaries, the receiving side starts a new root span and severs the visible chain. A rerun cannot repair this historical gap because it may follow a different valid route.
How do you know if a CrewAI tool call failed silently, and which harness layer caused it?
The first step is to check whether the tool span exists. A missing span requires comparison with provider and runtime evidence. If the provider issued a real tool call that the integration stringified, the signature is provider-shape mismatch. If the apparent observation exists only in generated text and run() never executed, the signature is fabricated observation. The absence of execution alone cannot distinguish them.
Next, establish whether run() was called and a real return was recorded, then inspect success separately. A stringified error is not a successful result. When execution raised and the wrapper converted that exception into text, wrapper absorption points to the wrapper’s result contract.
Wrong or incomplete arguments usually show that the tool description or docstring needs to spell out the requirements more clearly and include example arguments. Accepted malformed input points elsewhere. Silent repair shows a failure at the schema boundary because the harness accepted input it should reject. Comparing submitted arguments with schema requirements, and then checking whether the return matches its expected schema, prevents a plausible result from obscuring the earlier defect.
The next check establishes whether the result and required context crossed the handoff intact. If Agent B lacks a constraint that Agent A should have passed, the exact boundary payload places the failure in the memory or handoff layer. Confident downstream prose does not change that attribution. The termination reason then identifies a different orchestration defect when max iterations coincides with an expected success state that no available tool can verify.
Policy evidence completes the decision sequence when the flow aborts. When a WARN tool aborts unexpectedly without ToolFailureDetectedEvent, the policy-to-event-bus gap may be the cause. IGNORE requires a different reading because silence is its intended behavior after the fix, while an abort is not. Otherwise, an autopsy can mistake correct silence for broken observability.
Each failed tool call contributes to a failure counter, and reaching the limit opens the breaker so subsequent calls fail fast with a clear error. This turns silent retry loops into attributable failures and signals upstream that the tool is broken. For distributed crews, the recommended design uses a singleton or class variable to share breaker state. Separate counters for each agent do not show tool health across the crew.
Why harness-layer fixes must be validated against historical traces before shipping
A locally passing prompt, schema, or workflow change does not validate the production failure it claims to fix. The original conditions must be exercised again. Synthetic tests often check structure rather than the conditions that caused the failure, while checks that compare the final text can reject successful runs and accept broken ones. Validation therefore needs the recorded tool outputs and sandboxed side effects from the relevant traces. When the crew is simply rerun, nondeterminism can send it down a different path from the one being investigated.
Replay makes the comparison about the failure mechanism. A wrapper change must prevent an unsuccessful return from becoming ordinary narrative input, while a schema change must reject the malformed arguments that previously underwent silent repair. A handoff fix must be evaluated against the context that was missing.
The PROTEA system in 2026 formalizes a repeated validation loop for prompt revisions. After a revision is accepted, the system reruns the full workflow on an offline test suite and evaluates the new outputs. Automatic iteration repeats evaluation, revision, and re-evaluation for a fixed number of iterations, retaining a proposal only when repeated checks show improvement and stable behavior. The fixed suite is not sufficient indefinitely. Practitioners add CI gates for regressions on critical metrics and nightly evaluation against a sample of yesterday’s production traces to detect drift.
Turning recurring silent failure patterns into an ongoing engineering discipline
One autopsy finds one defect, while patterns that keep coming back show where the harness needs broader work. Research on production agentic systems found that no standard metric detects more than two of seven documented failure modes. In one documented case, accuracy stayed the same across evaluation windows while output diversity fell sharply. A stable dashboard can therefore coexist with a changing failure pattern.
The harness is also where skill-level continual learning becomes agent-system learning that includes parameter learning. Pioneer Agent in 2026 routes judged inference failures into diagnosis, taxonomy construction, targeted curriculum synthesis, retraining with regression gating, and conditional deployment. On AdaptFT-Bench, it improves or preserves its deployed checkpoint across all seven scenarios while naive retraining degrades. That result does not make retraining the first response. The practitioner rule is to exhaust prompt, retrieval, decoding, and model-swap variants on a stable evaluation set before fine-tuning.
For silent failures, this engineering work starts before deployment. It includes opting into tool-boundary payloads, recording returns separately from successes, and validating schemas at registration. Explicit termination conditions end retry loops when the agent cannot verify the success state, and shared breaker state lets the whole crew see that a tool is broken. Historical replay then tests whether each change corrects its attributed failure without regression. Harness engineering belongs under the same continuing rigor as code review and testing, with evidence preserved before failure and fixes judged against the execution that exposed them.


