Building an LLM Eval Pipeline in CI/CD for Agent Changes

A prompt change ships on a Friday afternoon. By Monday morning, customer-support escalations are up 14%, and nobody on the engineering team can say which users hit the bug, what the agent did wrong, or say for certain the previous prompt version handled those cases correctly. This is the actual state of CI for agent systems at most companies right now, and it's a structural problem.

The dashboard, when someone finally goes looking, shows token usage and latency. Both are flat, or close enough to flat that nobody would have caught the regression from them. There's no faithfulness score. No record of what the agent actually reasoned through on the runs that failed. The team can see that the system got slower or more expensive, but the two questions that matter (did it get worse, and where) go unanswered because nothing was built to answer them.

That gap exists because most agent changes ship without any regression gate at all. A prompt edit, a model swap, a tool schema tweak, a change to how the agent retries a failed call: these go out the same way a copy fix would, and the first signal anyone gets is a support ticket or a churn number. Traditional software fails loud, at an exception, at the line of code that broke. An agent can fail quiet. It can make one bad decision at step three of a nine-step trajectory, and the consequences don't surface until step eight, buried in a wall of reasoning tokens that nobody reads unless something has already gone wrong. Finding the root cause after the fact means reading through a transcript instead of a stack trace, and that is a much slower, much less reliable way to debug.

Why synthetic test cases make weak CI gates for agent changes

A team that scores 95% on a hand-built test set has learned very little about what will happen on live traffic. That gap between pre-deployment scoring and production behavior is the whole problem with synthetic evals: they test what someone imagined a user would ask, not what a user actually asked.

Real failures don't usually look like a wrong answer to a fair question. They look like specification ambiguity, where two agents in a multi-agent system both think they own the same task and duplicate the work, or where one assumes the other will validate an output that never gets validated. They look like tool schema drift, where a tool's description was accurate when someone wrote it six months ago and has quietly diverged from what the tool actually returns today. And they look like workflow loops, where an early misstep doesn't get caught and instead cascades, each subsequent decision built on the bad assumption from three steps back. Hand-written happy-path test cases essentially never exercise any of this, because whoever wrote the test case was reasoning about how the system should work, not about the specific way it broke last Tuesday.

There's also a harness variable that synthetic suites tend to ignore entirely. Research comparing agent harnesses has found accuracy gaps as large as roughly six-fold across different harness configurations running the same underlying model. A synthetic test suite built against one harness configuration will not catch a regression introduced by a harness change, because the suite was never testing the harness in the first place, it was testing the model's raw capability under one fixed set of scaffolding.

Studies of production multi-agent systems put failure rates somewhere between 41% and 86.7%, and attribute 79% of those breakdowns to specification ambiguity and uncoordinated protocols between agents. Studies of production multi-agent systems put failure rates somewhere between 41% and 86.7%, and attribute 79% of those breakdowns to specification ambiguity and uncoordinated protocols between agents, not to the underlying model being incapable. That's a coordination and design problem, and coordination and design problems are exactly what synthetic, single-path test cases are worst at catching.

What production traces contain that makes them the right source for eval cases

A production trace is a complete record, containing the prompt that got sent, every tool call and the outputs those tools returned, the reasoning steps in between, any retries, any handoffs between agents, and the final output the user saw. It's the closest thing to ground truth that exists for what an agent actually does when it's not being watched.

Failed traces carry more diagnostic value than successful ones, and by a wide margin. A failed trace pins down the exact input conditions that triggered the failure, because synthetic cases are, by definition, guesses about what conditions might matter, and no synthetic case can do that. It also shows which layer broke: whether the root cause sits in an ambiguous prompt, a tool call that errored out, a workflow that looped, a memory retrieval that pulled the wrong prior state, or a model output that was simply wrong despite everything upstream working fine. And critically, failed production traces are the only record of what real users actually asked, phrased how they actually phrased it, under conditions nobody on the engineering team would have thought to script.

Reading traces well takes a framework as well as access to the logs. The MAST taxonomy organizes agent failures into 14 distinct modes rolling up to three root categories, with specification ambiguity and uncoordinated protocols between agents accounting for the largest share of breakdowns. A separate Agent Error Taxonomy classifies failures across multiple operational dimensions, grounded in an Agent Error Benchmark built from trajectories across ALFWorld, GAIA, and WebShop. A third framework, TRAIL, works at the level of individual turns, applying a fine-grained taxonomy across reasoning, planning and coordination, and system execution, and its findings point to the value of structured attribution scaffolding when working through complex agent traces.

Automated failure attribution is still an unsolved problem. The paper "Which Agent Causes Task Failures and When?", evaluated on the Who&When dataset of failure logs pulled from 127 real multi-agent systems, found the best automated attribution method correctly identifies which agent caused a failure only 53.5% of the time. Pinpointing the specific step where the failure occurred drops to 14.2%. Those numbers matter for anyone building an eval pipeline: automated tooling can point in a general direction, but a human reviewer still needs to do the final diagnosis, at least for now.

The harness layers that CI evals must cover as distinct failure surfaces

The harness is the software layer that sits around the model, made up of the prompts it's given, the tools it can call, the memory it can retrieve from, the workflow logic that sequences its steps, and any middleware or verification routines shaping what it sees and does. It is not the model; it is everything wrapped around the model, and it is where most agent failures actually originate. It is everything wrapped around the model, and it is where most agent failures actually originate.

The research consensus on this point is fairly blunt: harness design is one of the biggest levers on agent performance, bigger in many cases than swapping the underlying model. That six-fold accuracy gap across harness configurations for the same model isn't a footnote, it's the headline. If the harness is where performance swings hardest, the harness is where CI needs to watch closest.

Research on production agent projects breaks the harness into five dimensions, each with its own failure signature. Prompts are the instruction surface, and ambiguity here is the single largest source of specification failures. Any prompt change, however small it looks in a diff, needs to trigger a full eval run rather than a quick manual read-through. Tools are which capabilities the agent can reach and how those capabilities are described to it; schema drift, where a tool's written description no longer matches what it actually does at runtime, produces failures that look like the agent made a bad choice when the real fault sits in stale documentation. Workflows cover the sequencing and coordination logic, the loops, retry policies, and handoff protocols between steps or agents, and this is where a large share of MAST's coordination-breakdown failures live. Memory is the retrieved state that shapes what the agent sees at each step, and a single bad retrieval at step one can sit dormant and then wreck a task five steps later. Skills, the reusable procedural artifacts an agent can discover, load, revise, and combine, change the nature of the task itself, shifting the agent from having to invent a workflow from scratch to selecting and following one, and a broken or mismatched skill produces its own distinct failure category.

None of this is well served by a single pass/fail number at the end of a run. An engineer staring at "test failed" has no way to know if the fault sits in the prompt, a tool description, or a workflow branch three calls deep, and without that specificity there's no fix to write, only a guess to make. Attribution has to be built into the eval, not bolted on after.

Structuring the eval pipeline across three levels: outcome, trajectory, and component

A workable eval pipeline runs at three levels, stacked like a diagnostic funnel. End-to-end, or outcome-level, evaluation treats the whole system as a black box and asks one question: did the task succeed? It's the right first gate, cheap to run and easy to interpret, but a pass here says nothing about whether the path the agent took to get there was sound, efficient, or likely to hold up under slightly different conditions.

Trajectory-level evaluation opens the black box. It looks at the plan the agent formed, the reasoning steps in between, the tool calls, the retries, the handoffs, and this is the level where failures within the agent's scaffolding can be examined in detail. It's also where trace-sourced evals earn their keep, because a trajectory-level eval built from a real failed trace is testing the exact sequence of decisions that went wrong in production, not a hypothetical one.

Component-level evaluation goes narrower still, isolating individual pieces such as a retriever, a sub-agent, a single tool call, or a specific prompt template. This is the level that tells an engineer exactly which artifact needs to change, and it's the natural target once trajectory-level eval has flagged that something in the harness is off.

Sizing the held-out datasets for each level affects cost and coverage directly: agent evals are expensive to run, and it's easy to either over-invest or under-cover. Headline metrics across a full pipeline generally want somewhere between 200 and 5,000 examples. RAG components do well with 500 to 2,000 query, gold-chunk, gold-answer triples. Chat or prompt-level evals are 500 to 1,000 prompt-and-gold-response pairs, or prompt-and-judge-rubric pairs where a fixed gold answer doesn't make sense. Agent scenarios with defined success criteria are the tightest, at 100 to 500 scripted scenarios. That narrower range isn't a shortcut on rigor; it's a function of cost: multi-step agent runs are slow and expensive to replay at scale, so the dataset has to be smaller and denser with signal rather than broad and shallow.

One metric belongs specifically to the agent level and gets skipped more often than it should: pass@k and pass^k, related but distinct metrics that together measure whether an agent succeeds across repeated runs on the same input, the former capturing best-case reliability, the latter all-runs consistency. Agent behavior is non-deterministic by nature, and a single-run check can look clean purely by chance, hiding a real regression that appears on the second or third attempt. Without pass@k in the pipeline, a CI gate can wave through a change that only works when the dice land right.

Choosing metrics that reflect harness-layer behavior, not just output quality

Four families of metrics make up a reasonably complete eval stack for agent CI, and each one is suited to a different failure surface. Deterministic metrics, things like exact match, F1, and code execution pass rate, are fast, cheap, and perfectly reproducible, which makes them the right tool for tool call correctness, structured output validation, and function-calling accuracy. They're the wrong tool for anything open-ended or multi-step, because there's no single correct string to match against a nine-step trajectory.

LLM-as-judge fills that gap, using a stronger or different model to score output against a rubric, and it's genuinely the right tool for trajectory-level and open-ended harness behavior. It is also the metric family most likely to mislead a team if it isn't calibrated first, a point worth its own section below. RAG metrics (context relevance, context recall, context precision, faithfulness, and hallucination rate) apply at the component level wherever the harness includes a retrieval step. And task-adherence and safety metrics ask whether the agent actually answered the task the user gave it and stayed inside policy, which matters most at the trajectory level, where instruction-following across a harness change either holds up or quietly slips.

A lot of eval pipelines go wrong by defaulting to one blended score for everything instead of matching metrics to the harness change under test. A prompt change calls for faithfulness, task adherence, and answer correctness measured against gold answers pulled from real production traces. A tool schema change calls for tool selection accuracy, tool call success rate, and context precision on whatever content the tool returns. A workflow change calls for trajectory coherence, retry rate, and convergence rate, meaning did the agent actually reach the goal or did it loop. A memory change calls for context recall, checking whether the correct prior state got retrieved and used at the point it mattered. A skill change calls for task completion rate specifically on the scenarios that exercise the modified skill.

One design principle causes most of the trouble when skipped, since skipping it is easy and expensive: the same evaluator templates should run across offline testing, CI, and production monitoring. A regression caught in CI ought to predict what would trip a runtime alert in production, and a runtime block ought to trace back to a CI check that should have caught it earlier. If the offline eval and the production monitor are scoring different things with different rubrics, the CI gate is measuring a different reality than the one users experience, and that mismatch is exactly how a "passing" build ships a regression.

These do not belong in this stack as a CI gate: token usage, latency by itself, and single-run pass/fail without pass@k. These are symptoms that appear after something has already gone wrong, not signals that point at a root cause, and building a merge gate around them is the same mistake as a scenario where a late change goes untested before release, watching the wrong dashboard while the actual regression goes unmeasured.

Calibrating LLM-as-judge before trusting it as a CI gate

LLM-as-judge exists because deterministic metrics genuinely cannot score reasoning chains, trajectory coherence, or open-ended task adherence, and those are precisely the failure modes that matter most when a harness changes. There's no way around using a judge model for this class of eval. Using one isn't the issue; trusting the one in use to block a merge is.

The floor for that trust is measurable: a judge needs to reach at least the 0.41 to 0.60 range of what's typically called "moderate" agreement with human reviewers before its verdict should ever block a pull request. Below that threshold, the metric still has value as something to track and watch trend over time, but it lacks the authority to stop a merge. Treating an uncalibrated judge as a hard gate is how a team ends up either blocking good changes on noise or, worse, waving through bad ones because the judge happened to agree with itself.

Three practices cut down the bias that creeps into judge scoring. First, the judge model should be stronger than, or at minimum different from, the model under test, because self-preference bias is a documented and real effect: a model scoring its own outputs tends to inflate them. Second, rubrics need to be explicit, with numeric scales and worked examples attached, which cuts down both position bias (favoring whichever response gets shown first in a pairwise comparison) and verbosity bias (favoring the longer answer regardless of its actual quality). Third, and this is the step teams skip most often under time pressure, the judge needs to be validated against a human-labeled set of 100 to 500 examples, with agreement computed directly, because that validation step reveals the judge's own blind spots before they cost anything.

Calibration isn't a one-time task, either. Regular recalibration is necessary, but it should happen off-cycle any time the rubric changes, the judge model gets upgraded to a new version, the gold set is suspected of going stale, or the divergence between judge and human scoring climbs past roughly 20 to 25%. A judge that was well-calibrated in January against last quarter's traffic patterns has no guarantee of staying calibrated once the underlying model or the user base shifts underneath it.

Wiring the CI gate: what runs on every pull request, what blocks a merge, and what flags for review

Put together, the pipeline runs in layers that map to how much confidence each layer can actually support. Deterministic checks, tool call correctness, structured output validation, schema conformance, run on every single pull request, because they're cheap, fast, and reproducible enough to gate without a second thought. These are the closest thing an agent system has to a traditional unit test, and they should be treated with the same automatic strictness.

Trajectory-level and component-level evals, built from production traces and scored with a calibrated LLM-as-judge, run against every harness-affecting change: any prompt edit, tool schema update, workflow modification, or memory change. Given the cost of replaying multi-step agent runs, these don't need to run against every single commit, but they need to run against anything touching the harness dimensions covered earlier, and pass@k needs to be part of that run rather than a single-shot check.

What actually blocks a merge should be narrow and specific: deterministic metric failures, and calibrated-judge scores that fall below an agreed threshold on trajectory-level task adherence or faithfulness, tied specifically to production trace replays where the previous version is known to have passed. What flags for human review instead of blocking outright includes judge scores in a gray zone near the threshold, any metric the judge is tracking but hasn't yet cleared the 0.41 to 0.60 agreement floor for, and any regression detected on trace types where the sample size is too thin to be confident. That gray-zone category matters as much as the hard gate does, because it's where a human reviewer, not an automated score, makes the final call, judging a subtle trajectory change as an improvement or as a regression waiting to surface in next week's support queue.

The whole structure rests on one discipline that's easy to state and hard to keep: every merged change that touches the harness needs its production traces fed back into the eval suite, so the next Friday prompt change gets tested against Monday's actual failure, not against a test case someone wrote six months ago and never revisited.