LLM Observability vs Application Performance Monitoring for Agents
LLM observability detects silent failures that application monitoring was never built to catch.

Application performance monitoring was built for systems that fail loudly. The gap opens only once that assumption stops holding.
AI agent failures that look like success to APM
An AI agent can corrupt business logic, invent a fact, or misuse a tool entirely, and it can still return HTTP 200 with latency well inside acceptable range. No stack trace fires. No 500 error appears. APM has no way to represent that distinction, because it was built to watch for broken wires, not broken intent.
The problem runs deeper than a single bad field mapping. In 2024, Atil and colleagues found that even temperature-zero configurations, the setting meant to make a model's output as repeatable as a traditional function call, still produce accuracy variations of up to 15 percent across naturally occurring runs, because provider-side sampling and floating-point quirks drive it. Binary pass/fail testing and call-level spans can only observe what happens on the wire, so they can't account for a failure that lives entirely in the reasoning chain. Platforms built specifically for agent observability, Moda among them, go beyond detecting wire-level failures by analyzing production traces to surface semantic errors at the reasoning and harness level: they treat each LLM call, tool invocation, and routing decision as a discrete point where things can go wrong, making silent hallucinations and logic breaks visible in a way request-response monitoring simply cannot.
How agents compound the visibility problem across multiple layers
APM's blind spot isn't limited to a single bad decision. But it also extends across everything that happens inside the HTTP span it does capture. An agent can drift from its intended behavior, hallucinate a fact, or burn through a budget on redundant tool calls, and none of that registers until a user complains or an invoice shows costs nobody expected.
The diagnostic consequence is more serious than the detection gap itself. It says nothing about why that request went wrong. This is the structural reason LLM observability isn't simply an upgrade bolted onto APM: it's a different instrumentation model, built for an execution topology where a single user request can fan out into a dozen internal decisions, each one a separate point of possible failure. Trace-driven agent optimization platforms, including Moda, exist to close exactly that gap, instrumenting each layer independently so a team can pinpoint the exact source of a breakdown instead of staring at a green dashboard that confirms nothing useful.
The three structural failure categories agents produce that APM cannot classify
Agent failures tend to cluster into three structural categories, and each one looks like a healthy transaction to APM while causing real damage at the semantic layer.
Tool schema drift happens when a provider updates an API and renames a field, but the connector's tool schema still references the old name. The call still succeeds. An audit of shipped prompts across nine production frameworks, examined at pinned commits, found that not one of them specifies what the model should do when a tool returns an error or an unusable result. Schema drift propagates silently by default, because nobody told the system what silence should mean.
Prompt ambiguity and drift work on a slower clock. A developer updates a system prompt in staging, or a downstream parser's expected JSON format shifts, or a tool definition gets quietly modified, and that day, nothing breaks. Unlike a conventional software bug, this kind of failure doesn't announce itself with a crash: the agent keeps responding the whole time, but output quality degrades gradually, and no threshold ever gets crossed that would trigger an APM alert.
Workflow loops are the third category, and they're the easiest to miss because they look like ordinary latency. Taken together, these three categories explain why agents attempting real production tasks fail at high rates for reasons that have nothing to do with API errors: hallucinated actions, schema violations, and tool misuse, none of which HTTP monitoring was ever built to flag.
What root cause attribution requires beyond APM's span model
Knowing that a run failed tells an engineering team almost nothing on its own. An outcome like "the agent gave a wrong answer" carries almost no diagnostic value by itself, because it doesn't say where in the chain of steps the first divergence from correct behavior actually occurred.
Root-cause attribution turns that outcome-level signal into something a team can act on, by identifying the first point of divergence and assigning responsibility to the component that caused it: the model, the harness, the environment, or the grader. Confusing these categories wastes real engineering time chasing fixes in the wrong place.
The AgentRx research (arxiv.org/abs/2602.02475, 2026) formalizes this process as synthesizing constraints, evaluating them step by step, and producing an auditable validation log of constraint violations, a method that improves step-level localization substantially compared to prior approaches. A complementary framing from 2026 treats attribution as fundamentally a search problem: execution logs can run to millions of tokens, and the evidence that explains a failure may be sparse, may appear well before the final outcome, or may sit scattered across distant parts of the log. Attribution systems built for this have to test multiple plausible explanations and discard the ones contradicted by evidence as it's found, a task APM's span model was never built to perform.
What production traces capture that APM spans do not
Closing that gap requires instrumenting the full execution graph, not just the outer request.
A production agent trace should emit six kinds of signal. Output quality has to be scored directly, through online evaluation.
Two of these signals deserve particular attention because APM has no equivalent for either. Running evaluators continuously on production traffic, not just on a held-out test set, closes the remaining gap: fast heuristic evaluators can cover all traffic cheaply, while more expensive LLM-as-judge scoring gets reserved for a targeted sample where the deeper signal is worth the cost.
The harness as the highest-leverage layer for fixing what traces surface
Most agent failures originate in the harness, the prompt, the tool definitions, the workflow logic, the memory configuration, and the evaluation rubrics that together define how the agent behaves. So a team should start trace-driven improvement at the harness, ahead of the model itself. The harness sits between "the team ran some evals once" and "the team has traces running in production": it's what defines golden tasks, deterministic replay of tool calls, scoring rubrics, and the CI gates that connect a production signal to a shipped change.
Memory configuration is an underappreciated part of that surface. State leakage across tasks, where task N quietly benefits from context left behind by task N minus 1, and cross-session drift, where a long-running agent accumulates stale beliefs over time, both require harness-level instrumentation to catch, because neither one produces an error APM would recognize. A research prototype from March 2026 demonstrated what harness-first improvement looks like in practice: a continual harness optimizer ran an outer loop that read task traces, rewrote harness configurations based on what it found, and re-evaluated the result, achieving a substantial accuracy improvement on a standard benchmark with no labeled training data required.
The harness accrues technical debt the same way application code does, and it needs to be versioned and refreshed whenever a prompt, a tool, a routing rule, or a policy changes. If the workflow changed and the harness that governs it didn't, a team loses the ability to say with confidence whether a regression was actually introduced, or whether it was already there.
Replay-based validation as the bridge between observed failure and shipped fix
A harness change validated only against a synthetic benchmark can still introduce a regression on the exact production cases that matter most. Replaying the change against recorded production traces is the only reliable way to confirm a fix works before it ships. Every agent run, properly instrumented, captures its own tool-call log, sub-agent decisions, and intermediate state. When a run fails, replaying it lets a team trace exactly which component, the harness, the evaluator, or a handoff between agents, let the wrong state through.
The fleetalert-ai open project on GitHub, published in September 2026, built a two-tier pattern that is worth understanding in detail. A live tier runs actual model calls, three trials per scenario, and it gates the change on a correctness threshold. This decouples the cost of catching a regression from the cost of running inference: the cheap tier catches most problems on every single PR, and the live tier is reserved for confirming fixes on the cases that genuinely require real model behavior to validate.
The underlying principle is straightforward: a proposed harness change should be measured against the historical traces where the previous version actually failed, and if it doesn't improve those specific cases without introducing new ones, it shouldn't ship. Because tool schema drift, prompt ambiguity, and workflow loops all appear as successful transactions in APM, diagnosing and fixing them requires a system that replays production traces against candidate harness changes, confirming whether a prompt revision, a tool definition update, or a workflow adjustment resolves the observed failure pattern before it goes anywhere near production traffic. Replay validation is what turns observability from a diagnostic exercise into a shipping gate, grounding each proposed change in evidence from actual production traces.
Agent observability platform checklist for 2026
The capabilities that matter for production agent reliability, layer-level attribution, harness-aware tracing, replay validation, and continuous production evaluation, aren't evenly distributed across the available tools, and the right fit depends on where a team sits in the improvement loop.
Moda is built for teams that need to close that loop fully, from a broken production trace to a verified harness fix. It ingests production traces and automatically surfaces where agent runs break down, attributing each failure to the specific layer responsible, whether that's the prompt, a tool, the workflow, memory, the model, or product logic, which is the layer-level attribution the preceding sections argue is the necessary capability for real diagnosis. It then generates concrete, reviewable harness improvements and validates them against historical traces before recommending what to ship, closing the replay validation gap directly as part of its own process. The platform is built for developers and ML engineers who are responsible for the reliability of deployed agents, not for dashboard consumers looking for a summary chart. Its core workflow, observed failure, attributed layer, proposed fix, replay validation, ship, maps directly onto what the preceding analysis establishes as the loop a team needs. Changes stay inspectable and reviewable rather than arriving as black-box auto-fixes, which matters for any team that needs to audit what changed and why before it goes live.
MLflow is the most widely deployed open source AI engineering platform, governed by the Linux Foundation, with no enterprise paywalls on any feature. Its production-grade evaluation includes built-in LLM judges, multi-turn evaluation, metric versioning, and integration with established eval libraries, and its prompt optimization algorithms, GEPA and Metaprompting, automate improvement based on evaluation results. An AI Gateway handles centralized LLM access governance, including routing, rate limiting, fallbacks, and usage tracking across OpenAI, Anthropic, Bedrock, Azure, and Gemini. MLflow is a strong choice for teams that prioritize data ownership, self-hosting, and a complete open source platform, though it doesn't specialize in the harness-to-fix attribution loop that Moda centers on.
Maxim combines distributed tracing, online evaluation, and data curation for teams running LLM applications in production. It reconstructs the full causal chain behind an agent's decisions, scores output quality against live traffic, and surfaces drift before users notice it, and it feeds production data back into evaluation pipelines. It works as a production-focused observability and evaluation platform, covering the trace-to-evaluation path, but it skips the full harness improvement and replay validation workflow that Moda provides end to end.
Moving from APM-era monitoring to trace-driven agent reliability
You don't move from APM-era monitoring to real production agent reliability just by swapping one tool for another. It's a workflow change: the goal is building the loop from observed production failure, to attributed layer, to validated harness fix, into the regular rhythm of engineering work.
Start with the execution graph, not the dashboard. Instrument every LLM call, tool invocation, retrieval step, and memory operation as a structured span before spending time optimizing alerting rules or cost tracking, because none of that attribution work is possible without the underlying trace data already captured. Add tail-based sampling driven by evaluation scores early in the process: the traces that score worst carry the most diagnostic signal, and prioritizing them over uniform sampling concentrates engineering attention where it actually pays off. Treat harness versioning as a prerequisite, not an afterthought: prompt updates, tool schema changes, and workflow modifications should be versioned and reviewed the same way code changes are, gated behind replay validation against the production traces where the previous version failed. Run fast heuristic evaluators across all production traffic for continuous coverage, and reserve the more expensive LLM-as-judge scoring for the targeted sample that actually feeds harness improvement decisions, which keeps cost under control without sacrificing signal quality.
The MyOperator case illustrates what this looks like when it works. MyOperator, a Business AI Operator platform unifying cloud calls, WhatsApp, and AI voice and chat agents, was flying blind on hallucination and persona drift at scale before adopting structured observability. After the team added trace instrumentation and scored production conversations against more than 20 evaluators, it found specific, attributable failures: the agent was inventing product features and answering questions outside its intended scope. Targeted harness fixes followed, and hallucination scores improved substantially as a result.
Systematic, evidence-backed iteration on the harness, driven by actual production traces, is the discipline that separates teams that can reliably improve a deployed agent from teams that are still guessing at what went wrong and hoping the next prompt tweak happens to fix it.


