The Agentic Harness

Treating Harness Engineering as a Continuous Engineering Discipline

Production traces reveal which harness layer caused each failure, guiding targeted fixes.

· 8 min read
Cover illustration for “Treating Harness Engineering as a Continuous Engineering Discipline”
Continual Improvement · October 23, 2026 · 8 min read · 1,840 words

A deployed agent improves continuously when production traces become evidence for layer-specific harness changes, regression tests, and validated releases. Treating harness engineering as a continuous engineering discipline means applying the same rigor to prompts, tools, memory, workflows, and validators as to application code. The compounding improvement loop connects observability to engineering decisions, so each production failure strengthens both the harness and the tests governing its next release.

Why the harness, not the model, is now the primary reliability surface for deployed agents

An agent is a model plus a harness. The model supplies capabilities, while the harness determines how those capabilities interact with context, tools, state, and task requirements. Jianyuan Guo and colleagues’ survey frames deployed agents as foundation models coupled with execution harnesses. It locates the performance bottleneck in the model, the harness, or their coupling rather than treating task completion as a property of model capability alone.

The harness handles observation, context, control, action, state, and verification/governance, which together govern whether a run reaches a valid result. Different harnesses can produce meaningfully different success rates with the same model on the same tasks. A harness-only change has produced a 13.7 percentage-point performance improvement while the model stayed fixed. The leverage is therefore not confined to better generation. It also lies in the executable environment that directs generation, preserves progress, and checks what leaves the system.

What the harness actually contains and why each component can fail independently

Prompts supply system and role instructions, with ambiguity or outdated assumptions forming their own failure class. Tool interfaces expose schemas and error surfaces, where a mismatch or misconfigured retry hook can abort a multi-turn session. Context delivery and compaction determine what information reaches each step, with context rot reducing reliability as inputs grow. Summarizing earlier conversations does not always preserve enough information to continue reliably, so the presence of a summary does not establish that context delivery succeeded.

Memory and state keep the progress made in earlier steps and sessions. When earlier decisions disappear at a context boundary, the agent reconstructs them. Control logic governs planning, branching, retries, and termination, with loops, wrong-tool calls, and premature stops among its failure modes. Validators check results before they propagate, but missing semantic checks let structurally valid, incorrect outputs escape. These components remain coupled at runtime. Their separate responsibilities nevertheless provide the map needed to distinguish an instruction problem from lost state, a broken tool contract, or a failure to verify the result.

How harness components degrade after the initial build

Even after a careful launch, the harness can get worse over time. External APIs and tool definitions have their own release schedules, and model updates can change how prompt formatting affects results, how tool calls behave, and how outputs are structured. These changes can happen even when the harness stays the same. Instructions and workflow constraints also keep assumptions about what the model cannot do. As capabilities improve, some of those assumptions expire, turning useful scaffolding into unnecessary restrictions.

Real traffic changes the operating conditions as well. A new kind of task can reveal weaknesses that the validators in place at launch were never built to catch. Within the harness, tightening execution can make unconstrained planning the dominant remaining source of run-to-run variance, so the relative importance of each component also changes as the system matures. Because the layers are coupled, degradation can cascade. A stale prompt can produce a malformed plan, which causes tool-call errors, exhausts retry budgets, and triggers incorrect termination. Continuous engineering deals with these expected changes in dependencies, capabilities, and workload, even when the original build was done with care.

Why standard monitoring misses most harness failures in production

An agent can loop, call the wrong tool, or drift off-task while returning an HTTP 200 response with normal latency and token counts. Infrastructure monitoring sees the exchange as a success but misses the failure to complete the task. The distinction is operationally important and easy to miss. A dashboard can accurately report service health while providing no evidence that the agent followed a coherent path toward the user’s goal.

Single-call LLM tracing leaves a similar gap because agent behavior unfolds across a trajectory. The Guo survey argues that quality cannot be assessed from final output alone. Tool choice, sequence, arguments, and state management contain essential evidence. Tool-call consistency research also distinguishes structural consistency from parametric variance, finding that tool order determines success while substantial argument variation can be benign. When production traces, evaluations, and alerts sit in disconnected systems, the problem becomes harder to diagnose because the signal arrives without the execution context needed to interpret it.

Attributing a failure to the right harness layer before attempting any fix

A failure may originate elsewhere than where it becomes visible. Evidence may sit in distant steps, requiring a systematic examination of the trajectory. One-shot LLM diagnosis tends to settle early on a plausible explanation and leave relevant portions of longer traces unexamined. A prompt edit based on that explanation may leave the actual fault untouched. Worse, a change to the wrong layer can introduce new failures into behavior that previously worked.

Cause classes such as prompt ambiguity, tool schema error, missing skill, control-loop bug, and middleware mismatch direct the change toward the responsible layer. In a study of harness changes, the first-pass harness improved reproducibility in one model-task cell, degraded it in two, and had no effect in the fourth. Diagnosis traced the remaining variance to unstructured planning rather than execution instability, making schema-validated planning the relevant intervention. Attribution therefore belongs at the start of every improvement cycle, supported by inspectable, layer-labeled evidence.

Validating harness changes against production traces before shipping

In shadow evaluation, the production agent serves the user while a candidate processes the same input on a separate path. The candidate’s output is stored but not returned. A batched evaluation job then compares candidate and live outputs across real traffic before rollout. This exposes behavior that a synthetic golden set may miss because anticipated tasks and actual user requests diverge over time.

The validation gate must also show the trade-offs for each deployment configuration. The harness study measured schema-validated planning across all four model-task cells and found model-dependent latency costs, reaching +24% for Gemma. The benefit of a constraint could not be assumed. Similarly, a single blended quality score can conceal regressions in stale-message echoing, tool-call argument drift, or premature termination. Separate rubrics for those failure classes make the comparison more diagnostic, while an evaluator decoupled from the optimizer reduces the risk of rewarding metric optimization that fails to improve behavior.

How do teams turn production agent traces into continuous harness improvement

Trajectories are evaluated, anomalies surface, and engineers review consequential failures. Important traces then become evaluation datasets containing the inputs, the agent’s output, and the correct behavior. Each retained failure strengthens the next evaluation cycle. This is the compounding improvement loop from production agent observability, where live traffic improves the evidence used to judge future harness changes.

The loop also needs a standing signal for drift. A curated set of traces with expected outputs or rubric scores, replayed daily or on every deploy, makes a sustained score drop an event to investigate. Real-time scoring for hallucination, faithfulness, off-task behavior, and answer relevance shortens the delay between failure and detection to within seconds of a model call. Mitchell Hashimoto’s approach turns mistakes into harness changes that stay in place, such as an AGENTS.md line or a tool built for the job. As fixes added up over three months, the share of commits that needed correction dropped to 3%. The operational distinction is persistence. A fix that also expands regression coverage informs later releases rather than remaining an isolated response to one incident.

Harness versioning and change management as engineering controls

Harness changes need the same care with version control, review, and rollback as application code. Editing a prompt changes how the system behaves in production, just as editing control-flow does. Prompt versions, tool schemas, memory configurations, and control logic therefore belong among explicitly versioned engineering artifacts. Planning artifacts, verification loops, and observability configurations also form part of the harness. Without a version record, attribution lacks the history of changes needed to explain the behavior, and there is no known version to use for rollback.

Review must preserve the deployment context as well. The harness study found model-dependent effects from structured planning, so a change suited to one configuration cannot automatically be treated as suitable for another running the same workflow. The natural-language agent harness pattern offers one way to make control logic portable and editable through artifacts executed by a shared runtime. Whatever the implementation, the control is a reviewable behavioral change linked to its validation evidence and a known-good version. Component-level rollback then allows a regression to be reversed without redeploying the model.

How harness iteration compares to model retraining as an improvement strategy

Harness iteration changes external, inspectable artifacts while model weights stay fixed. Revised prompts, tighter parsers, new tool schemas, added validators, and updated retry policies provide targeted interventions that can be reviewed and reversed. For production teams, this is generally the faster, lower-cost improvement cycle. Its audit trail also makes it easier to check how a proposed fix addresses a diagnosed failure than when the change is buried in model weights.

Model retraining remains a legitimate path with a different scope. Harness improvements do not automatically transfer to future model versions because they sit outside the post-training loop that updates the next checkpoint. Co-Harness proposes a dual-loop framework that alternates harness improvement from failed trajectories with model training on trajectories generated by the improved harness. Its comparative effectiveness over either approach alone is not fully established by the experiments. Despite that limitation, joint optimization remains valuable and the practical production priority remains intact. Most teams do not control the foundation model checkpoint, so their immediate choice is whether to operate the harness improvement loop systematically or leave its available gains unrealized.

What a mature harness engineering practice looks like operationally

Full trajectories supply the evidence for layer-specific attribution, and attributed failures inform both candidate changes and regression datasets. Replay validation tests those changes before release, while versioned artifacts preserve review and rollback. The practices depend on one another to work. Tracing without an evaluation path accumulates evidence without converting it into safer releases, while evaluation without production signal can remain anchored to tasks users no longer bring.

The organizational posture is humans on the loop. Engineers design and maintain agent environments rather than relying on output-by-output inspection, and periodic review identifies assumptions that have become obsolete constraints. That review also needs a longer horizon because models trained with particular harnesses can become overfitted to those designs. A team can test how mature its practice is by checking whether an important production failure has a trail through attribution, a reviewed change, replay evidence, and regression coverage that stays in place. The standard is a process that measures whether each change improves the relevant deployment and preserves what the team learns for the next cycle.

Sources

  1. Harness Engineering for Predictable Agentic Systems: An Empirical Study of Deterministic Execution Constraints
  2. Harness as an Asset: Enforcing Determinism via the Convergent AI Agent Framework (CAAF)
  3. From Question Answering to Task Completion: A Survey on Agent System and Harness Design

More in Continual Improvement