Prompt Ambiguity Root Cause Analysis

When an agent gives the wrong answer, the reflex is to blame the model, re-run the same prompt, and hope for a cleaner draw next time. That reflex fails in production for a simple reason: LLM failures rarely throw an exception. The pipeline runs to completion, the output is wrong, and there's no stack trace pointing at the fault. In a five-step agent, if one step misfires, the other four still execute and still produce something that looks like a normal result. Without visibility into which step went bad, a team is stuck staring at a wrong answer with no way back to the cause.

A taxonomy published on arXiv in September 2025 (2509.14404) offers a way out of that stuck position. It treats prompt defects as a structured failure class within LLM-based systems, and ambiguity inside them as a defect category that can be traced and addressed systematically. That distinction matters because prompt defects behave differently than model errors: they're structural, they repeat under the same conditions, and they can be traced back to a specific layer in the system. This piece treats prompt ambiguity as an engineering problem with a real investigation workflow.

Where prompt ambiguity lives in the agent failure landscape

The MAST taxonomy, presented at NeurIPS 2025 and built on more than 1,600 execution traces across seven multi-agent frameworks, sorts agent failures into three buckets. Specification problems, meaning role ambiguity, unclear task definitions, and missing constraints, account for 41.77% of failures and form the single largest category. Coordination failures, covering communication breakdowns and conflicting objectives between agents, make up 36.94%. Verification gaps, where nobody checks the output quality before it moves downstream, are 21.30%.

Prompt ambiguity sits inside that first bucket, the 41.77% slice. That makes it the single highest-leverage target for anyone trying to raise reliability, because it's the biggest lever available before you even start touching model choice or infrastructure. Separate reporting from Augment Code, drawing on the same MAST data in 2026, puts overall multi-agent failure rates in production between 41% and 86.7%, with specification ambiguity and coordination failures together responsible for 79% of all breakdowns.

Specification failures don't always start in the system prompt. Ambiguity failures appear wherever instructions are underspecified, which can mean a role definition, a tool description, or a task handoff message passed between two agents mid-run. Knowing that specification ambiguity is the biggest category only helps if a team can point at one failure and say, with evidence, that this is where it came from. That requires knowing what the defect actually looks like on the inside.

What prompt ambiguity looks like as a defect (a working taxonomy)

The same arXiv paper, 2509.14404, lays out six dimensions of prompt defects, the first systematic attempt at cataloguing this failure class. Three of the six appear directly in ambiguity failures.

Specification and intent defects happen when an instruction doesn't fully pin down the task, leaving the model to pick among several plausible readings instead of the one intended. Context and memory defects happen when an instruction assumes context the model doesn't actually have at that step, or contradicts something still sitting earlier in the context window. Structure and formatting defects happen when the expected output format is implied rather than spelled out, so the model defaults to some reasonable-looking format that the next tool in the pipeline can't parse.

All three share a nasty property: they pass dev testing. Under clean test inputs, the model happens to land on the interpretation the team intended, so the prompt looks fine. Then production traffic arrives with messier phrasing or a slightly different framing, and the model picks a different, equally defensible reading of the same instruction.

The FutureAGI 2026 RCA guide describes exactly this shape of failure in a planner prompt listing multiple tools without clear boundaries on when each one applies. Under clean inputs, the planner picks correctly every time. Adding an urgency qualifier to the user's message makes a different tool's description suddenly look like the better match, so the planner routes wrong, silently.

Conflicting instructions and missing constraints are two sub-types named separately. Conflicting instructions appear in long system prompts where two rules quietly contradict each other; the model resolves the conflict on its own, and which rule "wins" isn't stable across runs. Missing constraints appear when production throws an edge case the prompt never addressed; the model improvises, and that improvisation is indistinguishable, in the raw output, from behavior the team actually intended. The defect type tells you where in the trace to go looking, which is the actual starting point for root cause work.

Tracing prompt ambiguity to its layer using trajectory data instead of prompt re-runs

The instinct to re-run the failing prompt and see if it reproduces is understandable and wrong. LLM output is non-deterministic, so a passing re-run tells you nothing about whether the original failure will recur; per the FutureAGI guide, one clean pass doesn't invalidate the earlier bad one, it just means you got a different roll of the dice on a non-deterministic system.

A second common mistake is reading only the final response. In a multi-step agent, the ambiguity failure usually happens somewhere in the middle of the run. The final answer is the symptom. The cause is back at whichever step made the wrong call and passed it forward.

Trajectory data gives you what a re-run can't: the exact prompt content active at the step where things went wrong, not a guess at what it probably was; the model's raw output at that step, before later steps touched it; the tool call or handoff that followed, showing whether the model's interpretation actually got acted on; and a baseline from a successful run at the same step with a similar input, so the comparison has something to stand on.

Research published on arXiv (2511.19933) names a related pattern called multi-step reasoning drift, where output diverges by more than 20 to 30% across repeated runs of the same task. That kind of drift is invisible if you're only scoring final answers. It only becomes visible with per-step evaluator scores, things like tool selection accuracy, groundedness, and reasoning quality at each step, which are what turn a vague "the agent failed" into something specific: step 3 failed, and here's the exact prompt content that was live when it did.

A separate study on arXiv (2602.09937v2), covering 1,675 agent runs against the OpenRCA benchmark, found that the dominant failure modes, including hallucinated data interpretation, appeared across every model capability tier tested. That's a meaningful finding: if the failure doesn't go away when you swap in a stronger model, the defect isn't in the model at all, it's sitting in the prompt or the harness around it. The paper's conclusion is blunt on this point, prompt-level fixes alone didn't resolve these failures; they needed structural changes to the framework. Trajectory data is the only evidence that can confirm which layer actually owns the problem.

The step-by-step RCA workflow for a prompt ambiguity failure

The entry point is usually one of three things: a spike in eval-fail-rate for some cohort, a user complaint, or a downstream system choking on wrong output. From there, the workflow has five steps.

Step one is isolating the failing trajectory. Pull the full OpenTelemetry span tree for the run in question to find which spans carry low per-step evaluator scores; those are your candidates. Start at the first span where the score drops, not at the final response.

Step two is inspecting the prompt content at that failing span, and this means reading the whole assembled prompt, including the static system prompt sitting in a config file somewhere. That includes any context pulled in dynamically, memory chunks injected at runtime, and the tool descriptions active at that moment. The question to ask is plain: does this prompt actually pin down what the model should do here, or does it leave real room for interpretation? Look for missing task constraints, rules that quietly conflict, tool descriptions with overlapping scope, and output formats that are implied but never stated.

Step three is diffing against a successful run of the same task type. Find a trace where the same kind of task went right, then compare the prompt content at the identical step. What was different in the input that pushed the model toward a different reading of the same ambiguous instruction? Usually the diff points straight at the actual defect: a qualifier buried in the user's message that the prompt never accounted for, or a tool description that overlaps with another tool's under one specific input pattern.

Step four is attributing the failure to a specific defect type. Is it a specification or intent defect, where the task itself was underspecified? A context or memory defect, where the instruction assumed context that wasn't there? Or a structure defect, where the output format was implied rather than stated? The attribution has to be concrete: "the planner prompt doesn't distinguish Tool A from Tool B when the input contains urgency language" is something you can act on. "The prompt is ambiguous" is not. The OpenRCA findings in 2602.09937v2 show that prompt-level fixes alone didn't resolve these failures; they needed structural changes to the framework, suggesting the defect layer matters as much as the fix itself.

Step five is generating a candidate fix at the layer the attribution actually points to. That might mean adding the missing constraint, resolving the conflicting rule, spelling out tool selection criteria explicitly, or moving a critical instruction somewhere that survives context compression instead of getting dropped. The fix should be something a reviewer can read and reason about, a targeted change rather than a blanket rewrite of the whole prompt that changes ten things at once.

The FutureAGI 2026 guide walks through a real shape of this: a customer note reading "urgent please" sitting inside the prompt at span 3, overlapping in language with an unrelated tool's signature. The trace diff surfaced it. The fix was narrowing the planner's tool descriptions so urgency language stopped triggering the wrong route. After replay validation, eval-fail-rate-by-cohort dropped from 8% to 1.2%.

Prompt ambiguity failures hidden in the harness (the layers engineers miss)

The harness is everything in the runtime that isn't the model itself: prompt templates, tool descriptions, context construction logic, memory reads, handoff messages between agents, retry logic. Any one of those can carry an ambiguous instruction, and most RCA efforts stop looking well before they reach all of them.

Tool descriptions are a hidden prompt layer that gets overlooked constantly. A tool schema tells the model what a tool does and when to reach for it, and a vaguely written schema causes the exact same kind of mis-selection as a vague task instruction does. Schema drift makes this worse over time: when a tool's interface changes across versions but the description the model relies on doesn't get updated to match, the model is now working off an instruction that's factually wrong about the tool it's describing. An n8n production incident on v2.6.3 showed this pattern directly: a tool schema change broke production workflows for both OpenAI and Anthropic API users at the same time, with exceptions raised at the API level by both providers.

Context construction is another quiet vector. Dynamically assembled context can inject instructions that conflict with the system prompt without anyone noticing, a retrieved memory chunk that contradicts current guidance leaves the model to resolve a conflict it was never told existed. Context compaction in long-running agents tends to preserve the most recent and locally relevant content, while earlier instructions and intermediate decisions may not survive to the step where they were supposed to matter. A constraint buried in the middle of a long context may simply not make it to the step where it was supposed to matter.

Handoff messages in multi-agent setups are prompts too, even though nobody writes them thinking of them that way. When one agent hands a task to another, an underspecified handoff message leaves the receiving agent to fill the gap with its own guess. That's technically filed under coordination failure in the MAST breakdown, the 36.94% category, but the root cause tracing back through it is very often prompt ambiguity sitting in the handoff text itself. The practical rule: check the full assembled context at each span, including whatever lies beyond the system prompt file.

Validating the fix before shipping it (why replay against production traces is non-negotiable)

RCA that stops the moment a fix looks right is RCA that hasn't finished. A team finds the root cause, writes a tighter prompt, tests it against three examples that look like the failure, and ships it. Three weeks later, the same failure pattern resurfaces, except now it's coming from a different slice of production traffic that nobody thought to check.

The fix worked for the inputs the team had in mind while writing it. Production input distribution is almost always wider than that. Replay validation closes that gap: run the candidate fix against the full historical cohort of traces that previously failed, not new examples written to match the theory, and separately run it against the cohort that previously passed, to confirm nothing that used to work just broke. That's the only evidence that the fix holds against the real distribution of production input rather than a hand-picked sample.

The arXiv:2511.19933 taxonomy has a name for the failure this guards against: regression introduced by prompt changes, where a fix to one behavior quietly breaks another, invisible because testing only checked the case that was repaired. The OpenRCA paper (2602.09937v2) offers a useful counterpoint too, enriching inter-agent communication protocols cut communication-related failures by as much as 15 percentage points, but that gain was only visible because the evaluation setup measured the whole process, not just whether the final answer matched.

RCA works best as a loop, not a one-time fix. Once a fix is validated, the failed cohort becomes a standing regression dataset, and the same root cause gets watched for in new production traffic going forward, per the "recurring failure pattern" workflow in the FutureAGI 2026 guide. Shipping a prompt fix without replay validation is the same mistake as shipping code without running the test suite: the change can be correct in isolation and still take production down.

Building RCA for prompt ambiguity into the team's regular engineering workflow

Treating this as reliability-team work alone misreads where the failure actually lives. The FutureAGI guide states that prompt ambiguity failures sit inside prompts and the retrieval or configuration layers around them, artifacts owned by the engineers building the agent, not by the reliability team watching dashboards. Handing this exclusively to incident response means the fix always arrives after the damage, from people who didn't write the prompt and don't know its history.

A regular cadence looks less like a fire drill and more like code review: engineers checking eval scores per step the way they'd check test coverage, treating a drop in a specific cohort's pass rate as a bug report pointing at a specific span, and running replay validation before merging a prompt change the same way they'd run a test suite before merging code. The tooling that makes that cadence possible instead of aspirational, trace capture, per-step scoring, replay infrastructure, produces this outcome directly rather than aspirationally. Without it, RCA stays a one-off exercise reserved for the worst incidents, and the same specification failures, the 41.77% category driving nearly half of all agent breakdowns, keep resurfacing under new names.