Memory Retrieval Failures in Long-Horizon Agents

Memory retrieval failure in a long-horizon agent is a collection of distinct problems. It's four distinct breakdowns, each rooted in a different layer of the memory pipeline, and each requiring its own diagnostic before anyone can claim to have fixed it. Treat them as interchangeable and the wrong layer gets patched, the benchmark stays flat, and someone concludes memory is just hard. It is straightforward. It's misdiagnosed, and that distinction is the whole argument of this piece.

Long-horizon agents differ from single-turn systems in a structural way: success depends on carrying information across dozens or hundreds of observations, actions, and partial decisions, not on answering one prompt correctly in isolation. So the memory pipeline has to be understood as a sequence of stages, not a monolith. There's bank maintenance (what gets stored and how it's labeled), retrieval (what gets pulled for a given query), and answer-time resolution (what the model actually uses once it has the retrieved material in hand). A failure can originate at any one of these stages, and by the time it reaches the final output, it looks identical to a failure at any other stage. The agent just gives the wrong answer, and nothing about that wrong answer tells you where the fault line sits.

Agent trajectories compound the problem, because they're made of natural language, not structured logs. A code defect throws a stack trace. An agent failure sits buried in reasoning steps, tool calls, and summaries that read as plausible even when they're wrong. A 2026 TUM survey covering 55 papers published between early 2025 and April 2026 gives the field a systematic map of these failure types. What it doesn't give the field, and what remains unsolved, is a reliable way to localize which layer produced a given failure and attribute it correctly across a full execution trace.

The stakes are underappreciated. An agent can complete its assigned task, pass the evaluator's check, and still fail to leave behind a memory artifact that the next session can recover correctly. Final task success and memory-lifecycle integrity are not the same measurement, and conflating them is how engineers end up tuning retrieval when the actual defect sits somewhere in consolidation. No amount of retrieval tuning touches a consolidation bug. That's a direct statement. It's the pattern this piece traces four separate times.

The four distinct failure modes and which layer each belongs to

Ghost memory lives at the bank maintenance layer. Old facts, current facts, and the transitional facts in between all coexist in the memory store, and they stay mixed at retrieval time. Research on this problem, including work on the A-TMA system out of NUS in 2026, frames the core issue directly: deleting old facts destroys history the agent might still need, but timestamps or relevance scores alone don't tell the system which version of a fact is the one that should answer the current query. An agent asked about a user's current job might surface last year's job with the same confidence as this year's, because nothing in the storage layer marked one as superseded.

Behavioral state decay lives at the retrieval and injection layer. The relevant fact was seen. It may even technically remain within the context window. But it stops exerting influence on the decision loop at the moment it matters. This is a live-agent problem specifically: an agent diagnoses a failed command, or identifies a requirement early in a task, and later violates or repeats it anyway. Not because the memory disappeared, but because nothing injected it back into working attention when the decision came up.

Memory pollution lives at the write and ingestion layer. As an agent accumulates episodes over a long task stream, failed attempts, redundant entries, and plain noise pile up in the memory bank alongside anything useful. The dangerous part: this pollution doesn't look like noise to a similarity-based retriever. Work on TraceRetain in 2026 describes the mechanism as "high similarity, low precision." Failed distractors sit close to the query in embedding space, so the retrieval mechanism surfaces them with confidence, not hesitation.

Memory drift, or consolidation failure, lives at the consolidation layer. Systems that periodically summarize or compress stored memory introduce small distortions with each pass, and those distortions become the input for the next summarization cycle. Work on SSGM in 2026 frames this as a compounding loop across three interfaces: input ingestion, where poisoning enters; consolidation, where drift accumulates; and retrieval, where the now-drifted memory gets served up and treated as fact. Nobody audited the compression step, so the pipeline ends up manufacturing hallucination downstream of it.

These four aren't four flavors of the same problem. They start at different stages of the pipeline, and a fix aimed at one does nothing for the others. A better retrieval index doesn't fix a labeling problem in the bank. A better bank doesn't fix an injection-timing problem. Diagnosis has to be layer-specific before any fix can be validated. Most teams skip that step, and it shows up later as a benchmark score that won't move no matter what gets rebuilt.

Ghost memory: diagnosing state coordination failures at the bank maintenance layer

The diagnostic question for ghost memory is narrow. For a given query, which version of a fact should be live right now: the current one, a historical one, or one caught in transition? Standard QA accuracy can't answer that, because it aggregates across all of these state types at once. A system can score respectably on an end-to-end benchmark while consistently returning the wrong temporal version of a fact, and the aggregate score never exposes it.

A-TMA's response is architectural: evaluate at three decoupled levels rather than one. Bank maintenance asks what's stored and in what role. Retrieval asks which records get fetched for a given state-view request. Answer-time resolution asks which record the model actually uses once it has multiple candidates in hand. Failure at any one of the three levels produces the identical symptom at output, a wrong answer, so without decoupling the three, there's no way to tell which one broke.

Practically, A-TMA keeps superseded and transition records rather than deleting them, builds evidence packets scoped to the specific state view a query is asking about, and exposes current, historical, and transition labels directly to the QA step, rather than leaving the model to infer temporal status from context alone.

The measured effect is significant. On LTP, a conflict-heavy benchmark built specifically to stress-test ghost memory, adding A-TMA on top of Graphiti and Zep improves conflict accuracy by 0.240 absolute over the baseline. Temporal F1 on LoCoMo rises from 0.0295 to 0.1705, roughly a six-fold jump. What that gain reveals matters more than the number itself: the underlying systems weren't wrong about facts. They had the facts. They were wrong about state roles, about which fact was supposed to be live. The fix was labeling and separation, not sharper retrieval.

For engineers running production agents, the tell is specific. If an agent gives answers that were correct at some earlier point in a user's history but are stale now, that's ghost memory at the bank maintenance layer, not a retrieval bug. Tuning the embedding model won't touch it, and spending a sprint on it is close to wasted effort.

Behavioral state decay: diagnosing active-context failures at the retrieval/injection layer

Behavioral state decay shows up in long tasks specifically. An agent identifies a requirement, diagnoses an error pattern, or watches a command fail early in a session, and later on, it violates the requirement or repeats the failed command anyway. The fact was set aside on purpose. It just wasn't injected back into the decision loop at the point where it needed to intervene.

Separate this carefully from ghost memory. The memory bank may be storing the fact correctly, with no conflicting versions and no staleness problem. It may even be clean of pollution. The failure here is timing: something relevant failed to surface at the moment a decision depended on it. Passive, on-demand retrieval systems are vulnerable to this even when everything else in the pipeline works, because retrieving "on demand" assumes the agent knows to ask. Long-horizon agents often don't know what they've forgotten to check.

A proactive memory agent architecture from Meta AI, described in July 2026, tests this directly. The setup runs a separate memory agent alongside an unmodified action agent, and that memory agent's sole job is deciding whether to inject a memory-grounded reminder into the loop or stay silent. On Terminal-Bench 2.0, Claude Sonnet 4.5 improves from 37.6% to 45.9%, a gain of 8.3 percentage points. On τ²-Bench, it moves from 55.0% to 61.8%, up 6.8 points. The gain doesn't vanish with a stronger action agent, either: Opus 4.6 still picks up 2.4 and 2.5 points respectively on the same two benchmarks, ruling out the easy explanation that only weaker models need the extra nudge.

Ablation results reinforce the point. Selective, decision-aware intervention beats passive bank exposure, beats always-on injection of every stored memory, and beats general-purpose retrieval. Whether and when to inject a memory matters as much as what's stored in the first place. Treating the two as the same problem gets the causality backwards.

For production diagnosis, the signal is behavioral repetition. An agent that repeats a mistake it already diagnosed, or breaks a constraint it correctly identified earlier in the same session, is failing at the injection layer. Treat that as a storage or retrieval-index problem and the fix will look reasonable on paper. It won't hold up once the agent runs long enough to need a well-timed reminder rather than a bigger memory bank.

Memory pollution: diagnosing write-layer failures that standard retrieval metrics cannot surface

Pollution accumulates from many small writes acting together. It's a failure of accumulation: as an agent operates over a long stream of episodes, failed trajectories, redundant entries, and outright noise build up in the store right alongside the memories that are actually useful.

The paradox that makes this dangerous is embedding similarity. Failed distractors often sit close to a query in vector space, precisely because they were generated by an agent attempting the same or a similar task. A similarity-based retriever surfaces them with high confidence rather than filtering them out. TraceRetain's 2026 findings show that unbounded memory retention produces the highest mean similarity score, 0.87, and simultaneously the lowest retrieval precision of the retention strategies tested. Similarity and precision move in opposite directions as pollution accumulates. The metric engineers usually trust to signal "good retrieval" is exactly the metric pollution corrupts. That makes standard evals actively misleading here, not just insufficient.

This also explains why naive eviction policies fail quietly. FIFO eviction removes the oldest entries in the bank, which has nothing to do with whether those entries are noise. An old, high-quality memory gets evicted at the same rate as an old, useless one. Treating age as a proxy for uselessness is the actual bug here, not some downstream retrieval quirk.

The masking effect compounds the problem. Under clean conditions, using ALFWorld as a reference environment, most bounded retention policies land within a similar confidence range of unbounded memory. Pollution simply doesn't manifest at moderate task horizons on standard benchmarks. It takes an artificial stress test to expose it: under a 75% synthetic distractor injection, unbounded memory's Precision@5 drops from 20.2% to 12.4%, and FIFO-K50 collapses further, from 15.8% to 3.8%. A retention approach designed around content-aware eviction, TraceRetain-CEM, holds nearly flat, moving from 16.9% to 16.6%, and preserves 97 of 100 task completions under the same stress.

The implication for anyone running agents in production is direct. Pollution shows up in real, noisy write streams long before it shows up on any clean benchmark, so the diagnostic signal has to come from production traces, not test suites. If retrieval quality degrades as sessions get longer, or the agent surfaces memories that look topically relevant but turn out factually wrong, look at the write stream first. The retention and eviction policy is the more likely culprit, not the retrieval index sitting downstream of it.

Memory drift: diagnosing consolidation failures that compound across the pipeline

Drift doesn't originate in the bank and it doesn't originate in retrieval. It originates in the act of compression itself. Systems that periodically summarize accumulated memory to keep storage manageable are, by construction, doing something lossy, and each summarization pass introduces small distortions that get carried forward as the input to the next cycle.

Work on SSGM in 2026 frames the resulting failure as a loop running across three interfaces in sequence. Bad input enters at ingestion. Drift accumulates during consolidation, where the stored representation diverges from ground truth. The drifted memory then gets retrieved later and presented to the model as fact, at which point it functions indistinguishably from hallucination. Except the model didn't invent it. The pipeline manufactured it across the consolidation stage.

This differs meaningfully from a standard retrieval-augmented generation setup, where a bad retrieval is a one-off event scoped to a single query. In a consolidating memory system, a distorted summary becomes the raw material for the next write, and the error compounds rather than resetting with each new session.

Drift is also the hardest of the four to catch by inspection, because the retrieved memory it produces is often internally coherent and confidently phrased. It's a plausible, well-formed distillation that happens to be wrong, sitting alongside a missing fact or two conflicting versions competing for the model's attention. That's precisely why it survives most manual review.

The diagnostic tell is gradual. Agent behavior that degrades slowly across sessions, where retrieved memories look reasonable but are systematically off in ways that echo the agent's own earlier summaries, points to the consolidation layer as the origin. That calls for an audit of the compression step itself: which algorithm is doing the summarizing, how many compression passes an entry has been through, and whether outputs get validated against source material at any point. Retuning the retrieval query does nothing here. The query was never the problem.

Why end-to-end pass/fail scores miss every one of these failure modes

Final-accuracy masking is a common occurrence. It's the default condition of evaluating agents this way. An agent can satisfy its local task, pass the simulator's check, and still fail to leave behind a memory artifact that the next session can recover correctly. Memory has to be evaluated as a lifecycle spanning writes, consolidation, retrieval, and injection, not judged by a single pass/fail number at the end of a trajectory. That single-number habit is what most teams still default to, and it's the wrong default.

Even strong models struggle to self-diagnose without structural support. GPT-5.1 reaches only 18.15% accuracy on failure attribution datasets when it isn't given structured behavioral abstractions to reason over, according to multi-institutional research spanning Tsinghua University, Microsoft Research, Microsoft, and UIUC in 2026. That's a design incentive problem. It's a diagnostic framework problem: the model has no scaffolding telling it where to look, and no amount of scale fixes a missing scaffold.

Turn-level tracing work under the TRAIL framework shows a related pattern: even long-context models that perform well on other tasks struggle at trace debugging once traces are broken down turn by turn. The failure sits in the diagnostic tooling available to the model, not solely in the agent being diagnosed.

The MAST taxonomy adds a further wrinkle. Existing attribution methods are reasonably good at locating the step where a trajectory went wrong. They fall short of explaining why. Knowing a run failed at step 14 doesn't tell anyone whether the cause was a bad prompt, a corrupted write, a drifted consolidation, or a reminder that should have fired and didn't.

This isn't unique to research settings, either. A two-year production postmortem study at a major retailer found a persistent attribution error rate of roughly 10%, where the diagnostic model blamed a technology simply because it happened to be mentioned somewhere in the incident thread. Surface correlation stands in for causal attribution whenever the underlying framework can't actually trace cause and effect.

The consequence for anyone building or maintaining these systems is blunt. Investing in better retrieval without first knowing which layer broke is roughly a coin flip, often worse than doing nothing, because a fix aimed at the wrong layer can shift behavior without addressing the actual defect, burying the real problem under a change that looks like progress. Knowing whether the failure originated in the bank, the write stream, the consolidation step, or the injection timing is the prerequisite to attempting a fix at all, not an optional refinement on top of one.

What layer-specific attribution requires in practice

Attribution has to be specific to be useful. Saying a run failed is not the same as saying which of the four modes caused it, and each of the four calls for a different intervention. Relabeling in the bank, a retention policy change at write time, an audit of the compression pipeline, or a decision-aware injection mechanism running alongside the action agent may be needed.

MemTrace, published in 2026, converts a memory pipeline into an executable memory evolution graph, allowing fine-grained tracing of exactly how information moved through the system at each step. Its accompanying benchmark, MemTraceBench, covers representative systems across Long-Context approaches, RAG, Mem0, and EverMemOS. The core finding echoes what shows up across the other work cited here. When a final answer comes out wrong, the actual cause usually sits earlier in the pipeline: a fact that got missed, a memory that got silently overwritten, or the wrong memory retrieved and never corrected. Feeding these fine-grained attribution signals back into prompt optimization closes the loop and, per the 2026 findings, boosts end-task performance by up to 7.62%.

A separate line of work, presented as an ICML 2025 Spotlight paper titled "Which Agent Causes Task Failures and When?", pushes attribution further using counterfactual replay. A case study included in that work is instructive: given the same failed trajectory, QWEN3-8B attributes the failure superficially to a code execution error. CLAUDE-4-SONNET digs somewhat deeper. AgenTracer-8B, the method proposed in the paper, traces the root cause all the way back to Step 2, where a Web Surfer subagent retrieved the wrong file, one carrying an incorrect date. That error stayed hidden behind seemingly correct outputs all the way through to Step 11, where its consequences finally became visible.

That gap, between where an error originates and where it finally becomes visible, is the entire argument for layer-specific attribution. A defect nine steps upstream of its symptom will never get fixed by someone staring at the symptom. It gets fixed by someone with a trace, a taxonomy, and the discipline to ask which layer actually broke before touching anything downstream of it.