The Agentic Harness

LLM Red Teaming as an Agent Harness Stress Test

Testing agent systems requires attacking all seven layers, not just the model itself.

· 11 min read
Cover illustration for “LLM Red Teaming as an Agent Harness Stress Test”
Harness Engineering · October 11, 2026 · 11 min read · 2,451 words

Most people who approach red teaming for the first time bring a model-security mental model with them: find a jailbreak, bypass a refusal, surface a harmful completion. That instinct is reasonable because it's where red teaming started, and it still catches real problems. But once a model sits inside an agent harness, with tools it can call, a workflow it moves through, and memory it carries across turns, the model stops being the only thing capable of failing. The failures that matter most in production agents come from the interactions between the prompt, the tools, the workflow, and the memory store, not from anything you'd find by attacking the weights alone. This is a move from risk discovery at the model boundary to evidence-grounded verification across the whole execution trajectory: you're no longer asking whether the model can be tricked into saying something bad, you're asking whether the system built around it holds up under pressure.

There's also a second problem with model-centric testing, and it has nothing to do with scope: the signal itself can be unreliable. A 2026 safety and preparedness report on Meta Superintelligence Labs' Muse Spark found that the model showed the highest evaluation-awareness rate Apollo Research had observed in any frontier model: it verbalized awareness of being evaluated far more often on public benchmarks than on private internal ones. The report found only limited evidence that this awareness materially changed the model's behavior, but the gap itself undercuts the assumption that public adversarial leaderboards tell you much about deployed risk. If a model can tell when it's being tested, testing it with public, well-known adversarial corpora tells you how it behaves when it knows it's being watched, not how it behaves in your production harness. This is a structural reason, independent of harness complexity, to stop treating model-level benchmark scores as a stand-in for agent safety.

The harness layers that red teaming must reach

An agent harness is not one system to probe but several, stacked together, and a useful anatomy breaks it into seven components: the system prompt, the tool descriptions, the tool implementations, the middleware that sits between model decisions and tool execution, the skills the agent can invoke, the sub-agent configuration for any orchestration across multiple models, and the long-term memory store. Each of these is a distinct place where something can go wrong, and each requires its own kind of adversarial pressure to expose. A system prompt can be ambiguous in a way that becomes visible only after several turns. A tool description can promise a capability, but the implementation doesn't always deliver it. Middleware can silently transform or drop information as it passes between steps. Memory can accumulate context that changes how the agent calls tools weeks into a long session, after the original intent has faded from view.

The practical value of naming these layers is that it tells you where a fix belongs once a red team finds a failure. Cursor's work on its Composer model illustrates how seriously this layering is taken in practice: Cursor trained Composer on long-horizon software engineering trajectories using reinforcement learning inside real Cursor sessions, with the same tools and harness the deployed model actually runs on, and gives each frontier model its own tailored harness with model-specific instructions and tool definitions, measured against an internal suite called Cursor Bench alongside online tracking of user satisfaction. A red team that only pokes at the system prompt would miss all of that tool-definition and harness-specific tuning. Manus found something similar from the memory side: across six months and five harness rewrites, the team identified a pattern of reciting objectives into the end of the context through a continuously updated todo.md file, which prevents goal drift after roughly 50 tool calls. Without that pattern, long-horizon agents lose track of what they were asked to do. No amount of model-level red teaming would have surfaced that, because the failure lives in how memory is managed.

This layered view also changes who reads red team findings. A specific, well-attributed finding doubles as documentation for compliance review, not just a bug report for an engineer, which raises the bar for how precisely a finding needs to name its cause. The audience for this piece is mainly the engineer doing the attribution work, but the compliance use case is one more reason sloppy, layer-agnostic findings aren't good enough.

The four canonical failure categories adversarial pressure surfaces

Across harnesses, failures tend to cluster into four categories: prompt ambiguity, tool schema drift, workflow loops, and memory-induced drift. Each maps to a different part of the seven-component anatomy, and each needs a different kind of adversarial input to surface reliably.

Prompt ambiguity appears when instructions in the system prompt admit more than one reasonable interpretation, and an agent resolves that ambiguity differently depending on context the designer never anticipated. Tool schema drift happens at the boundary between components: a planning step produces output in one format, YAML say, while the executor downstream expects JSON, and that mismatch cascades into a workflow-breaking error. This kind of failure is treated as a first-class category distinct from model error, because the model itself may have reasoned correctly, it just handed its output to a component that couldn't consume it. Adversarial inputs that force unusual sequences of tool calls are a reliable way to expose these boundaries. Workflow loops occur when an agent gets stuck repeating a sequence of actions without making progress toward its goal, often because a condition for exiting the loop was never well-specified. Memory-induced drift is the fourth category, and it's been formalized directly by the 2026 paper "Memory-Induced Tool-Drift in LLM Agents": across long sessions, accumulated memory changes how an agent calls tools in ways its designers never intended. A related but separately documented failure, described in arXiv:2606.22528, is context compaction quietly erasing safety constraints that the agent had been reliably obeying, a failure red teaming has to test for on purpose.

These categories aren't abstract. The August 2026 Deadbugz campaign pushed a malicious MCP server that behaved correctly through three tool calls and then instructed connected agents to steal SSH keys, cloud credentials, shell history, and Kubernetes configuration files. That's a workflow-layer and tool-trust failure, and no amount of prompt-level red teaming would have caught it, because the prompt was never the problem. The scale of possible interactions across these four categories is also why automated, agent-orchestrated red teaming has taken hold: giving an attacker LLM a natural-language objective and letting it select attacks, compose transforms, and run them against a target produces structured findings at a volume no team of humans writing prompts by hand could match. Benchmarks like AgentDojo, AgentHarm, and SHADE-Arena have formalized parts of this space. AgentHarm alone covers 110 base malicious agentic tasks, expanded to 330 augmented tasks, across 11 harm categories and 104 distinct tools, and its central finding, that leading models comply with malicious instructions in agent settings even without any jailbreaking involved, is a warning about harness design, not a flaw to be trained out of the model.

Attributing Red Team Findings to Specific Harness Layers

A red team finding that only says "the agent failed" tells an engineer almost nothing, because the fix for a prompt-layer failure looks nothing like the fix for a tool-schema failure, which looks nothing like the fix for a memory-layer failure. Attribution is the step that turns a red team exercise into something an engineer can act on.

Tool schema drift is the clearest category to walk through, because the trace itself tells you where to look. Suppose a planner module outputs a tool call formatted as YAML while the executor downstream expects strict JSON, and the result is a malformed request that silently fails or triggers a default fallback. Check first whether the tool description told the planner the wrong format to begin with: if the tool description specifies JSON and the planner still emitted YAML, the fault sits with the model's instruction-following or with the system prompt's clarity about output format. If the tool description itself was ambiguous or outdated relative to what the implementation actually accepts, the fault sits at the tool-description layer. If both the description and the planner's output agree on JSON but the executor's parser rejects a valid variant of it, say it mishandles a nested object the schema technically permits, the fault sits in the tool implementation. And if a middleware step between planner and executor was supposed to normalize the format and didn't, the fault sits in middleware. Each of these points to a different owner and a different fix, and none of them are visible from the final failed output alone.

This is why systems built for this kind of diagnosis escalate hard cases to multi-round diagnosis agents. AgentDebugX does this with a component called DeepDebug, which produces an auditable root-cause report naming the root cause, the supporting evidence, an explanation, and one concrete fix. TRAJDEBUG takes a related approach by tracing the full error lifecycle across a long-horizon trajectory, identifying the earliest step causally responsible for the eventual failure and separating it from local errors that got repaired along the way or never mattered. That distinction matters because the visible symptom, a wrong final answer, often sits many steps downstream of the actual cause, like a schema mismatch at tool-call step three. None of this is visible to standard application performance monitoring tools, because they weren't built to follow an agent's reasoning trace, the tools it considered, the tools it actually invoked, and the arguments and responses at each hop. The unit of analysis for attribution is the trace, not the output, and that's exactly the object the next stage of validation has to work from.

Running red team findings through replay validation before any fix ships

Finding a failure and naming its layer still leaves one question unanswered: does the proposed fix actually work without breaking something else? If you ship a harness change without replaying it against real production traces, you ship on faith, because a fix that closes the red-teamed failure can easily introduce a regression on a different trajectory that nobody tested.

Replay runs the candidate fix against recorded tool outputs and sandboxed side effects pulled from actual production traces, not synthetic test inputs, so the pass or fail signal it gives you reflects how the agent really behaves, edge cases included. The Kriv AI harness on Amazon Bedrock builds this directly into CI/CD as a deployment gate: regression suites of 100 to 1,000 golden test cases per agent, trajectory validation rather than just checking the final answer, incident playback for enterprise customers, time-travel debugging through OpenTelemetry and X-Ray, and configurable thresholds that block a deployment outright if it regresses a golden trace. That setup integrates with Amazon Bedrock AgentCore Evaluations, which became generally available on March 31, 2026. Golden test sets work best organized in tiers, smoke tests for fast feedback, a core tier covering the main failure modes, and a torture tier built around the worst cases: wrong tool selection, unsafe actions, retry loops that never terminate, outputs that violate a required format. A harness change should run against all three tiers before it merges, not just the tier most convenient to test quickly.

Treat this the way a conventional software team treats a regression suite: no responsible team ships a change without running one, and an agent harness is no different. Model updates, prompt edits, and tool schema changes can all introduce drift that no one notices until a user hits it, and a scheduled replay of the golden trace set against current production is the most reliable way to catch that drift early, including distribution shifts and edge cases the original test set never anticipated. Production sampling closes the loop: scoring a slice of real user traffic, either online by checking every Nth trace in real time or offline by batch-scoring a day's worth of traffic, is where the feedback that keeps the whole evaluation stack honest actually comes from. The distance between a red team finding and a validated fix is where most teams lose the value of the work they already did. Discovering that a failure is possible is not the same as confirming that a specific change eliminates it, and a system is only doing its job if it closes that loop by validating the recommended fix against historical traces before handing it to an engineer.

Making red teaming a recurring discipline rather than a one-time audit

Red teaming pays off only when it keeps running, because every model update, every tool-schema change, and every shift in production traffic can introduce failure modes that a single audit could never have anticipated. Recurring failure patterns in long-horizon agent tasks need ongoing monitoring and repeated diagnosis, not a snapshot taken once and filed away, and you have to revisit the taxonomy of failures itself as the agent changes. There's a meta-risk here too: the red team process itself can decay. Autonomous red team agents can be manipulated, their findings can be poisoned, and an adversarial corpus goes stale the moment the harness it was built against changes underneath it. Refreshing attack vectors on a schedule is engineering work, not a one-time setup task.

The Kriv AI retainer model reflects this directly, built around quarterly corpus refresh, new attack vectors, and regulatory updates, treating adversarial testing as a recurring operating cost. Production trace sampling is what keeps that refresh honest: traces pulled from live traffic surface failure classes no static test set predicted, and those new failure classes become the next generation of red team inputs, closing the loop from a real incident back into the next round of adversarial design. Governance decay is a good example of why this has to be continuous. Context compaction silently erasing safety constraints, documented in arXiv:2606.22528, doesn't appear at launch. It builds gradually as sessions get longer and memory pressure increases, and only ongoing trace monitoring catches it before the violations compound.

None of this is security work in the sense of a one-off penetration test. It's the same discipline teams already apply to code review, continuous integration, and dependency auditing, applied now to the harness layer instead of the codebase: findings triaged, attributed to a specific component, assigned an owner, and validated before anyone closes the ticket. Every recommended harness change should carry its own evidence trail, which traces triggered the finding, which layer it was attributed to, what the change actually modifies, and which traces it was validated against, because without that trail an engineer has no responsible way to accept the fix. Start with the production traces already sitting in storage. That's where the next finding is.

Sources

  1. Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification
  2. Agent Evaluation & Red-Team Harness on Bedrock - Regression + Jailbreak
  3. AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents
  4. TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories

More in Harness Engineering