AI evaluation
What does good AI agent testing in CI look like?
Your CI pipeline should test your code on every push. It should test the model only when you decide to pay for it.
Short answer
Effective AI agent testing in CI splits the work into tiers. On every commit, run fast unit tests with a scripted fake model that returns fixed tool calls, so routing, graph logic and guards are checked without paying for or waiting on a real LLM. Run routing evals and model-graded quality evals with real models on demand, before releases and model changes, and trace every run so failures point to a specific tool call.
On this page
01Why agents do not fit a normal test pipeline
A normal service has deterministic behavior: the same input gives the same output, so a unit test either passes or fails. An agent does not work that way. The model chooses which tool to call, with which arguments, and in what order. Run the same test twice and you may get two different paths. This is part of our larger guide, LLMOps for SaaS: keep AI features working in production.
Teams respond in one of two ways. Some skip automated tests and rely on manual checks, which means regressions reach users. Others call the real model in every CI run, which makes the pipeline slow, expensive and flaky, until people start ignoring red builds.
Neither works. The answer is to separate what is deterministic in your agent from what is not, and test each in the right place.
02What is actually deterministic in an agent
More of an agent is ordinary code than it looks. Around the model call sit:
- The graph or state machine that decides which node runs next.
- The router that picks which domain agent or tool set handles a request.
- The tool registry: which tools exist, which roles and channels may use them, and how arguments are validated.
- Guards that block leaked tool names, internal IDs or actions that have no matching tool.
- The code that turns API responses into context for the model.
- Retry, loop and duplicate-reply protections.
All of that can be tested like any other code, as long as you take the model out of the loop. That is what a scripted fake model is for.
03The three tiers of AI agent testing we use

| Tier | What it tests | Model used | When it runs | Typical failure it catches |
|---|---|---|---|---|
| Tier 1: unit | Graph logic, routing plumbing, tool permissions, guards | Scripted fake tool-calling model | Every push, in CI | A guard removed, a tool exposed to the wrong role, a broken state transition |
| Tier 2: routing evals | Whether real models pick the right agent and tool | Real model | On demand, before release or model change | A prompt edit that sends booking questions to the wrong agent |
| Tier 3: quality evals | Whether replies are correct, grounded and well formed | Real model, graded by a model and checked by people | On demand, before release or model change | Answers that are on route but wrong, vague or leaking internal fields |
The fake model in Tier 1 is not a mock of the whole agent. It is a stand-in for the language model only. You script its responses: for this test, it returns a call to the booking tool with these arguments, then a final message. The rest of the agent runs for real. LangChain documents this pattern directly: a generic fake chat model returns scripted messages, including tool calls, from an iterator, so agent tests run without API keys.
04How to build Tier 1

- 01
Script the model turn by turn.
For each test, define the exact sequence of model outputs: a tool call, then an answer, or a tool call that should be blocked.
- 02
Run the real graph.
Let your actual routing, nodes, tool registry and guards process those outputs.
- 03
Assert on behavior, not text.
Check which node ran, which tool was called, with which arguments, under which role. Check that a forbidden tool call was refused.
- 04
Test the guards with hostile outputs.
Script the fake model to return an internal ID, a tool name in the reply, or a call to a tool that does not exist. The guard should catch it.
- 05
Test every role and channel.
If cleaners and owners have different tool sets, script the same request for each and assert on the difference.
- 06
Keep fixtures fresh per test.
A scripted iterator that is shared across tests runs out and causes confusing failures.
- 07
Make it fast.
Tier 1 should finish in the time people are willing to wait for a pull request check.
05How to run Tiers 2 and 3
Real-model evals need a dataset of representative requests with expected routes or reference answers, and graders for each. Code-based graders work for routing: did the model pick the expected agent and tool? Model-based graders work for quality, but they need calibration against human judgment, and their prompts should be reviewed like code.
Run these tiers when something changes that affects the model's behavior: a prompt edit, a new tool, a model upgrade. Pin the dataset version in the pipeline so a dataset update does not silently change what "passing" means. Anthropic's guidance on agent evals adds a useful distinction: capability evals tell you how far the agent has to go, while regression evals should stay near a full pass rate, and a drop means something broke.
06Trace everything
Tiered tests catch most regressions, but production will still find new ones. When it does, you need traces that show the exact sequence of model calls and tool calls for that conversation. Log by subsystem, redact customer data and tokens before logs are shared, and turn every production bug into a new Tier 1 or Tier 2 case once you understand it.
For more on how agents fail once real users arrive, see why AI agents fail in production.
07What we see in production
We built a signed desktop call recorder for an insurance sales training company, so its agents' real calls on any meeting tool are scored like their practice calls. The app adds a virtual microphone that captures both sides of a call, and its audio routing stays on independently of recording and fails open, so the agent is never cut off.
That design gave us a lot of behavior worth testing that no single test call would reach, so we built a dedicated audio test rig. It drives the real app with a simulated meeting tool, scripted microphone reopens and sleep like interruptions, with automated tests behind routing, microphone reopen races and power events. Builds move from staging to production only through a release gate, and a rule from the client's integration requirements keeps cookies, tokens and signed URLs out of every log and telemetry event.
The rig paid for itself. Each reported issue was reproduced, fixed and tested again on it, and problems seen on agents' machines, such as sleep, microphone hot plug, echo and crackle, were worked through with error monitoring and audio diagnostics in the logs. The recorder is in use by the client's agents, and its desktop builds ship on Windows and Mac.
Frequently asked questions
What is a scripted fake model in agent testing?
It is a stand-in for the language model only, not a mock of the whole agent. You script its responses for each test, such as a call to the booking tool with set arguments and then a final message. The real graph, router, tool registry and guards process those outputs, so the tests run without API keys.
Should CI call the real model on every push?
No. Calling the real model in every CI run makes the pipeline slow, expensive and flaky, until people start ignoring red builds. Run fast Tier 1 unit tests with a fake model on every push, and run routing and quality evals with real models on demand, before releases and model changes.
How do you grade routing and quality evals?
Code-based graders work for routing: did the model pick the expected agent and tool? Model-based graders work for quality, but they need calibration against human judgment, and their prompts should be reviewed like code. Pin the dataset version in the pipeline so a dataset update does not silently change what passing means.
What should happen when production finds a new bug?
Use traces to see the exact sequence of model calls and tool calls for that conversation, with customer data and tokens redacted before logs are shared. Once you understand the bug, turn it into a new Tier 1 or Tier 2 case, so the same regression cannot return unnoticed.
How Datics helps
Datics specializes in Brownfield AI: production AI inside software people already rely on. Already built AI? The AI Scorecard tests it on 50 of your own cases in two weeks, for $5,000, and ranks what to fix first. Once it is live, AI Care keeps it accurate for $1,500 a month per capability.
Related work
Case studies from our projects on the same subject.
CleanHaus
Property Management · Turnover operations SaaS
Hosts and cleaners manage rental turnovers in chat, within each person's permissions
Owners, managers and cleaners all talk in one property chat. Now an AI there acts for each of them, only within that person's role.
More on AI evaluation
AI evaluation
AI evaluation: build your test set before the first release
AI evaluation starts before launch: collect real cases from your product, define pass and fail for each one, and run the set on every prompt or model change.
AI evaluation
How do you run LLM testing with real user prompts?
LLM testing for an AI assistant should use real user prompts and customer-run tests, because demos hide lost context, repeated answers and wrong filters.
AI evaluation
AI model testing: test types, metrics and messy data
AI model testing for AI features in your product: the main test types, what each catches, and how to build a messy test set that becomes your regression suite.
AI evaluation
AI model evaluation: how do you choose an LLM to build on?
AI model evaluation starts with a scorecard: define the task, metrics and test set first, then run every candidate model through the same rerunnable test.
AI evaluation
What LLM evaluation should a chat agent pass before launch?
LLM evaluation for a chat agent starts with written user journeys. See how we turn them into test cases, graders and release gates before launch.
AI evaluation
What should an AI audit of a SaaS feature test?
An AI audit of a SaaS feature tests it on your own cases in seven areas, from answer quality and data separation to cost, and ranks what to fix first.
Your customers already expect AI from your product.
Let us make it real inside the product you run today.
