Structuring Trace Spans for Multi-Agent Orchestration
Preserve causality across agent boundaries by structuring spans hierarchically.

The observability contract built for a single LLM call was always simple: inspect the prompt, inspect the response. That contract fails in a multi-agent system because causality now runs across agents, handoffs, and shared memory reads that a flat per-call log has no way to link together. Four groups of people feel this gap differently. Backend engineers lose the ability to reproduce intermittent failures without the message context that passed between agents. SREs cannot attribute latency spikes to the specific agent responsible for them. Compliance teams cannot prove which agent in the chain actually emitted a regulated output. Product leads see a user-facing failure with no way to tell which layer of the system caused it.
This is not a niche concern confined to research labs experimenting with exotic topologies. Production stacks built on LangGraph, CrewAI, and the OpenAI Agents SDK, alongside AutoGen (now in maintenance mode), have made researcher-writer-critic teams, planner-plus-N-executor arrangements, and supervisor-with-subagents patterns the default architecture for serious agentic work in 2026. Once a system has more than one agent making decisions, the question "why did this run fail" stops having a single answer embedded in a single log line. A sequence of decisions, handoffs, and reads holds the answer, and structuring the trace so that sequence survives is the only way to recover it.
What multi-agent tracing produces
Multi-agent tracing is the discipline of capturing every LLM call, tool call, handoff, and shared-memory read across every agent in a system, all tied to a single root trace ID, so the result is one continuous, replayable tree instead of several disconnected flat logs. The mechanism underneath this is OpenTelemetry, and each individual step in a run, whether it is a model call, a tool invocation, or a memory lookup, gets recorded as a span, carrying the prompt, the response, the latency, the token counts, and the cost. String those spans together in the right order and the result is a trace: the complete account of one run from start to finish.
A span is more than a log entry with a timestamp. Each span carries a parent span ID, and that parent-child link is what preserves the causal chain, the planner's decision that triggered a tool call, the tool call that returned a flawed result, the flawed result that produced a hallucination several steps later. Without that chain, a team is left looking at a pile of events that happened to occur in the same time window, with no structural reason to believe one caused another.
Academic and industry work has already begun formalizing this: llmmas-otel, from Delft University of Technology and JetBrains Research, builds OpenTelemetry-based distributed tracing combined with fault injection specifically for LLM-based multi-agent systems in software engineering contexts, giving teams a lightweight way to capture and inspect agent behavior under controlled conditions. A trace built this way can support attribution, rather than just visualization, only if it holds two structural invariants. The first is explicit context propagation: the trace ID and parent span ID have to be passed across agent boundaries, which matters most when agents run as separate processes rather than as calls within one program. If that propagation is dropped anywhere in the chain, the result is not one tree but several disconnected flat traces, each describing a fragment of the run with no way to stitch them back together. The second invariant, covered next, concerns how spans are arranged relative to one another once propagation is working.
The span hierarchy decisions that determine whether failure attribution is possible
The parent-child arrangement of spans dictates whether a broken run can be walked backward to the specific agent, tool, or handoff responsible. Hierarchy functions as attribution architecture. A hierarchy that mirrors the system's actual causal structure puts the user-facing task at the root, gives each contributing agent its own child span, and nests that agent's tool calls, memory reads, and model invocations as grandchildren beneath it.
The placement choices inside that structure carry real consequences. A planner agent's model invocation belongs as a child of the planner's own span. So sibling placement erases the link between the decision and the action it caused ⦅c8⦆. The resulting trace shows not just that a call happened, but which phase it belonged to, which agent was responsible for it, and how it relates to the steps that followed.
Consider the same run represented two different ways: a flat trace versus a properly nested one, in a flat trace every span, the planner's reasoning, the tool call, the executor's response, the final output, sits at the same level directly under the root. Causality does not survive: nothing in that flat arrangement distinguishes "the tool failed, and the agent looped as a result" from "the agent looped on its own, and one of the calls it made during the loop happened to fail." In a properly nested trace, the tool call sits as a child of the specific agent turn that invoked it, and the loop (if there is one) shows up as repeated sibling spans under that same agent's span, with the tool failure sitting inside the first iteration. The shape of the tree itself becomes diagnostic.
The hardest design call in building that tree is deciding exactly where one agent's span ends and a sub-agent's span begins, because that boundary decision determines which attribution bucket a given failure falls into. A handoff span has to carry the sending agent's ID, the receiving agent's ID, and the message payload itself, because without these attributes the trace has no way to answer the question that matters most after a failure: did the planner send the wrong task, or did the executor receive a correct task and misinterpret it? Depth discipline matters on both sides of this decision. A workable heuristic is to give a span to any step where an engineer would plausibly want to set a breakpoint or ask "was this step correct?" and to leave out internal implementation details that cannot fail in a way anyone would need to diagnose separately.
Attribute schemas that carry the evidence needed for root-cause search
A well-structured hierarchy tells a team where in the execution a failure occurred. It does not, by itself, tell them why. The attributes attached to each span carry that answer, so a trace can be perfectly nested and still be useless for root-cause search if its attributes are thin.
The OpenTelemetry GenAI semantic conventions set the baseline for this schema: standardized gen_ai.* attributes cover model name, token counts, and finish reason. Prompt and response content belongs in span events rather than in attributes, because attributes are indexed and size-limited, and storing full prompt text there is a known anti-pattern. But beyond that baseline, different span kinds need different evidence attached to them. Without that "used" flag, a bad retrieval that genuinely influenced a wrong answer is indistinguishable from a bad retrieval the model simply ignored.
Turning each memory.retrieve call into its own OpenTelemetry span is the pattern that makes memory reads attributable instead of invisible. The obvious objection to all of this is that such a schema is verbose and adds overhead to every single call. Tail-based sampling is the answer: every failed, expensive, or anomalous trace gets kept in full detail, while happy-path traces get sampled aggressively. The overhead lands on the runs worth investigating, instead of being spread evenly across traffic that will never be opened.
Handoff span design separates planner failures from executor failures
The handoff boundary is where attribution most commonly breaks down. Without explicit handoff spans carrying the task payload in both directions, a trace has no way to answer whether a planner delegated the wrong task in the first place, or whether a correct task was delegated and an executor carried it out incorrectly.
Two failure modes produce identical symptoms without properly designed handoff spans. The fix for this failure lives in the planner's prompt or its workflow logic. Treating these as the same bug wastes engineering time fixing the wrong layer.
A support team's eval-fail-rate-by-cohort metric spiking for one particular routing path illustrates how this plays out in practice. Practical handoff span design treats a2a_send and a2a_receive as two distinct span kinds rather than collapsing them into a single generic "handoff" span, because the gap between the send timestamp and the receive timestamp is itself a diagnostic signal worth preserving. Handoff span density, the count of agent-handoff events per trace, functions as a useful dashboard metric in its own right, since spikes in that density tend to correlate with planner regressions specifically.
This structure is also what makes a broader multi-agent root-cause pattern possible: an orchestrator identifies which agents sit on the critical path of a failure, spawns a focused investigation into each one, and then synthesizes the findings into a single diagnosis. That synthesis step only works if the handoff spans have preserved which agent called which, and in what order. Without the handoff metadata, the orchestrator has nothing to synthesize beyond a list of events it cannot relate to one another.
Span-level and trace-level scoring as the evaluation complement to structure
Structural correctness in a trace is necessary, but it is not sufficient on its own. A trace can be perfectly nested and richly attributed and still leave a team unable to say which spans represent genuine failures, because structure shows what happened, but scoring says whether what happened was correct.
Two distinct levels of scoring cover two different categories of failure. Span-level scores evaluate things like tool-use correctness, retrieval faithfulness, and grounding, and they catch failures that stay invisible at the trace level because the final output still looks acceptable even though one intermediate step along the way was wrong. Trace-level scores evaluate task completion, plan adherence, and context maintenance across turns, and they catch a different category of failure where every individual step passes its own check but the overall trajectory still misses the user's actual goal.
Agent observability earns the label "attribution instrument" when it runs evaluations on tool calls, sub-agent handoffs, retrievals, and individual spans, not only on the final answer a system produces. This has a direct consequence for span design: a span has to carry the inputs and the expected behavior that a score will need to run against it. A tool call span with no argument schema version attached cannot be scored for schema-drift correctness. If a memory read span has no retrieval score attached, you cannot score it for grounding quality.
Multi-agent regressions tend to hide inside sub-agents, so scoring every span catches failures that a final-output score would miss. If a later agent compensates for a sub-agent failure, the end result still looks fine and will not show up in a final-output score, but it will appear clearly in that sub-agent's own span score. Trace completeness rate, the percentage of traces in which every agent that was expected to run actually emitted at least one span, is a prerequisite metric for all of this. A score computed on an incomplete trace is a score computed on incomplete evidence; no amount of downstream analysis corrects for spans that were never captured.
The three span-detectable failure categories
Span design is the precondition for diagnosing failures; without it, a team can see that something failed but cannot say what, specifically, went wrong.
Tool schema drift appears in a tool call span when the argument schema version recorded on the span does not match the schema version the tool's response was actually generated against. The model emits a call that is valid under the old schema, gets back an error or a mismatched response, and from there either loops on the same failing call or silently produces a wrong answer built on bad data. The schema-version attribute on the tool call span is what makes this drift visible instead of looking like an ordinary tool error.
Result-utilization failures occur at the boundary between a memory read or tool call span and the model invocation that follows it. The failure here sits in the handling of a correct result once it arrived, not in the retrieval or the tool call, and only a span schema that records whether a retrieved item was used can separate this from a genuine retrieval failure.
Workflow loops appear as a repeated pattern of sibling spans under the same agent: often a tool call, then a model invocation, then another call to the same tool with slightly altered arguments, repeating several times before either succeeding or exhausting a retry budget. Properly nested spans turn what would otherwise look like a vague timeout into a clearly bounded, repeating structure, pointing directly at the agent and the specific call where the loop originates.
Sources
- Multi-Agent Tracing 2026: traceAI, OTel, Span Hierarchy
- Multi-Agent AI Systems in 2026: Frameworks, Patterns, Production
- Observability and Fault Injection for LLM-Based Multi-Agent Systems in Software Engineering
- OpenTelemetry for AI Systems: LLM and Agent Observability (2026) | Uptrace
- Multi-Agent Tracing: Components & Use Cases (2026)
- The 2026 Guide To Agent Observability Tools
- Multi-Agent Systems for Distributed Tracing Root Cause Analysis: 2026 and Beyond
- Span vs Trace in LLM Observability: What's the Difference?


