The Agentic Harness

Closing the Loop From Observed Failure to Validated Fix

A four-stage discipline for turning production failures into validated fixes.

· 8 min read
Cover illustration for “Closing the Loop From Observed Failure to Validated Fix”
Continual Improvement · October 21, 2026 · 8 min read · 1,896 words

Closing the Loop From Observed Failure to Validated Fix is a four-stage engineering discipline for improving agents from production evidence. Teams must find failures in real traces, trace them to the layer at fault, make a concrete change, and replay that change against historical traces before shipping. Production observability compounds into better behavior only when all four stages run consistently.

Why a failed agent run is harder to diagnose than it looks

The visible failure point in an agent run is routinely not the point that caused it. An early mistake can propagate through planning, memory, tool calls, and downstream decisions before the task collapses. Later actions may therefore look defective because they inherited a bad instruction or state. At the same time, an earlier mistake may have been repaired and no longer matter. Diagnosis must tell those paths apart, which takes more than picking the first suspicious action or the last failed check.

TRAJDEBUG illustrates this distinction with an agent tasked with fixing a CSV boolean export. The agent edits a forbidden file, then makes a compensating patch that changes JSON behavior. Tests fail in a different module, while the agent still breaks the original task constraint. The JSON test failure is visible near the end, but the forbidden-file edit is the critical error. Treating the failed test as the origin would direct attention away from the decision that set the cascade in motion.

Scale AI's Continual Search paper describes the same forensic burden at a much larger scale. Investigators reconstructing a real security incident had to recover and analyze more than 70,000 agent messages and files. Investigators found evidence earlier readings had missed by making successive passes over that record. The difficulty was not merely finding an error, but reconstructing how distant actions related to the eventual outcome. Diagnosis needs the execution record beneath a task-level failure signal because that signal cannot supply that relationship.

What step-level tracing actually needs to capture, and why pass/fail monitoring misses it

The first stage is to make the run observable at the level where decisions happen. A hierarchical, step-level trace must capture the reasoning trace, tools considered, tools actually invoked, arguments passed, and responses returned. It must also record tokens spent at each step and the latency of each hop. Those records need to be stitched together so the run can be replayed. A collection of disconnected logs does not provide the same evidence as an execution history that preserves the relationship between steps.

This detail matters because identical inputs can produce different tool sequences, and plausible outputs can be semantically wrong. With silent quality degradation, the agent keeps running while its answers become wrong, misleading, or unsafe. This can happen because of document-store drift, prompt regression from version changes, silent model updates by providers, and input distribution shift. A binary monitor can report that a run completed without exposing any of these changes in quality.

Tool schema drift creates another gap between apparent operation and actual correctness. A tool may return data with wrong field types, missing required keys, invalid JSON, or truncated payloads. Without explicit schema validation, an agent cannot distinguish a structurally broken response from a transient network error and may retry both. Missing exit conditions, fallback handling, and format specifications similarly allow loops, abrupt halts, or unparseable outputs. Moda ingests production traces and automatically surfaces where runs break, attributing failures to the responsible prompt, tool, workflow, or memory layer. That is a practical implementation of the required standard, but collecting the evidence and establishing its meaning remain distinct engineering tasks.

Why locating the failure step is not the same as identifying the cause

Once there is a trace, attribution means searching through what it contains. Relevant evidence may be sparse, separated by many actions, or absent from the immediate context of the visible failure. A tool call linked to the failure may carry forward an error from an upstream instruction. Correct attribution therefore requires comparing what happened at a step with the instructions and observations that shaped it. Logging makes that comparison possible, but does not perform it.

Scale AI's Continual Search paper identifies a weakness in one-shot LLM diagnosis. A judge can settle on a plausible explanation early and leave important evidence in a long trace unexamined. Its iterative framework instead pushes the judge to challenge its standing answer and continue searching. LongRCA Bench conveys how difficult exact attribution remains across 1,140 human-annotated failed trajectories. Even RCTA, which retrieves original candidate evidence and compares it with upstream instructions, leaves the majority of root causes unrecovered exactly. Finding a relevant action is substantially easier than proving where the decisive error originated.

TRAJDEBUG approaches that distinction by identifying errors from evidence and tracking their resolution status and terminal impact. An error that was repaired belongs in a different category from one that remained harmless or caused the final failure. That distinction must then lead to a component-level assignment, namely model, harness, environment, or grader. Otherwise, a precise-looking diagnosis can still send the engineer to the wrong component. Attribution is complete only when the evidence explains both the decisive error and the intervention surface responsible for it.

How the harness — not the model — is where most fixes actually live

For most production failures, that intervention surface is the harness. Prompts, tools, workflow, memory, and evals govern how the model operates within the task. A defect in those components can produce behavior that looks like a model capability failure. Attribution shifts engineers' attention from whether the model is good enough to whether the surrounding execution system gave it the right constraints and support. Retraining is not the default answer to a failure in that system.

Planning failures make the distinction concrete. A wrong tool sequence, infinite loop, or missing step can point to the planner prompt or a plan-vs.-execute eval. Tool failures may instead call for tool allowlists, schema guards, retry policies, or MCP descriptor scans. When fallback handling and output format specifications are missing, the workflow and prompt need changes too. These are different repairs, even when they all appear to the user as an unsuccessful task.

HarnessDev treats harness repair as a distinct engineering problem, separating Creation of an execution system from Evolution through downstream execution feedback. In production-trace-to-fix systems such as Moda, the same principle grounds attribution in the full execution record. Moda distinguishes prompt, tool, workflow, memory, model, and product logic failures, then generates reviewable changes at the attributed layer. Historical-trace validation informs what it recommends shipping. Some failures still require a better model, but repeated attribution to the model layer provides the evidence for that escalation.

Generating a concrete fix and keeping it reviewable

An attributed cause must produce an actual change. Naming the planner as responsible is less useful than specifying the planner instruction that needs revision. Finding a broken tool contract must also lead to a change to that contract, which takes more than asking for more careful tool use. The fix needs to be narrow enough for an engineer to reason about its relationship to the failure. Specificity is what turns diagnosis into an inspectable engineering proposal.

AgentDebug illustrates this handoff through module-level classification and targeted corrective guidance. Its AgentErrorTaxonomy covers memory, reflection, planning, action, and system errors. The framework identifies the earliest critical error globally, supplies guidance for the attributed module, and resumes the run from that point. The important distinction is between correcting the mechanism that failed and asking the agent to “try harder.” Generic guidance can sound reasonable without addressing the causal chain established by the trace.

Reviewability adds a separate requirement beyond specificity. Engineers need to inspect what changed, why it was proposed, and which execution evidence supports it. That evidence must let them approve or reject the proposal rather than accept an opaque recommendation, especially when an agent takes consequential actions. Moda presents concrete improvements for engineer approval before anything ships, tying each change to the attributed layer and the traces that surfaced it. Approval establishes that the proposal is understandable and justified enough to test. It does not establish that the change will hold across real production inputs, which is the question the next stage must answer.

Why replay validation against real traces is the only way to know a fix actually works

Replay validation tests the proposed change under the conditions that exposed the defect. Manual inspection can establish that a prompt revision or schema guard looks appropriate, but it cannot show how that change affects complete runs. Historical traces give the proposal production cases it must pass. Replay therefore becomes a separate stage required to complete the fix. Shipping without it leaves the claimed repair unsupported by execution evidence.

Testing starts by running golden end-to-end tasks with the proposed change in place. Recording and replaying tool calls makes those tests deterministic, while explicit rubrics define how results are scored. Regression checks in CI then block changes that fail the required tests. Joining offline eval results to production traces through trace IDs preserves the connection between a test result and the run that motivated it.

TRAJDEBUG's application studies reinforce the need to verify the corrected trajectory after applying actionable feedback. AgentCheck makes pre-ship testing explicit through its Reproduce-Intervene-Mitigate workbench for LLM agents over MCP. Its validation corpus has 120 structured scenarios that cover tool execution, data quality, and security failures. For failures found in production, replay in Moda lets teams test proposed improvements against the same historical traces where the errors happened. These approaches make the intervention's observed effect, rather than the diagnosis's plausibility, the basis for release.

Synthetic evals remain useful, but they test the scenarios a team imagined. Production traces test the scenarios users actually generated, and those distributions can diverge in ways that matter. Surfacing, attribution, generation, and validation together close the loop. Skipping any stage leaves a route for the original failure to survive into the next production run, while completing all four produces evidence worth retaining.

How do teams turn validated fixes into a compounding signal-detection system

Each finished loop leaves a structured record that includes a labeled failure trajectory, an attribution, and a verified fix. Retained together, those records allow comparison across runs rather than forcing each incident to start from an empty diagnosis. A repeatedly drifting tool schema is no longer just a series of unrelated failures. The same prompt ambiguity appearing on a new input class becomes a recurring pattern. Discarding traces and attributions after debugging discards the evidence needed to see that recurrence.

Scale AI's Continual Search framework positions RCA as a feedback layer between failed executions and improvement of the broader agentic system. In-context adaptation and retrieval-augmented grounding address fast-moving drift, while prompt revisions, schema guards, and workflow exits address structural causes established by attribution. Keeping those roles separate stops teams from treating every new production signal as a reason for model retraining. The continuing value of production evidence lies primarily in better harness decisions, with retraining carrying substantially higher cost and risk.

The compounding improvement loop from production agent observability is therefore a discipline of retaining and testing causal knowledge. New traces show failures, earlier attributions help tell patterns apart from one-off anomalies, and validated fixes show which changes held under replay. None of those assets replaces the need to investigate the next run. Instead, they make that investigation better grounded and make proposed changes easier to evaluate.

Sources

  1. TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
  2. Root-Cause Attribution Is a Search Problem: Continual Search for Long-Horizon Agent Failures
  3. LongRCA Bench: Root-Cause Localization in Long-Horizon Agent Trajectories
  4. AGENTRX: Diagnosing AI Agent Failures from Execution Trajectories

More in Continual Improvement