What does good AI agent testing in CI look like?

01Why agents do not fit a normal test pipeline

A normal service has deterministic behavior: the same input gives the same output, so a unit test either passes or fails. An agent does not work that way. The model chooses which tool to call, with which arguments, and in what order. Run the same test twice and you may get two different paths. This is part of our larger guide, LLMOps for SaaS: keep AI features working in production.

Teams respond in one of two ways. Some skip automated tests and rely on manual checks, which means regressions reach users. Others call the real model in every CI run, which makes the pipeline slow, expensive and flaky, until people start ignoring red builds.

Neither works. The answer is to separate what is deterministic in your agent from what is not, and test each in the right place.

02What is actually deterministic in an agent

More of an agent is ordinary code than it looks. Around the model call sit:

• The graph or state machine that decides which node runs next.
• The router that picks which domain agent or tool set handles a request.
• The tool registry: which tools exist, which roles and channels may use them, and how arguments are validated.
• Guards that block leaked tool names, internal IDs or actions that have no matching tool.
• The code that turns API responses into context for the model.
• Retry, loop and duplicate-reply protections.

All of that can be tested like any other code, as long as you take the model out of the loop. That is what a scripted fake model is for.

03The three tiers of AI agent testing we use

The fake model in Tier 1 is not a mock of the whole agent. It is a stand-in for the language model only. You script its responses: for this test, it returns a call to the booking tool with these arguments, then a final message. The rest of the agent runs for real. LangChain documents this pattern directly: a generic fake chat model returns scripted messages, including tool calls, from an iterator, so agent tests run without API keys.

04How to build Tier 1

1. 01

   Script the model turn by turn.

   For each test, define the exact sequence of model outputs: a tool call, then an answer, or a tool call that should be blocked.

2. 02

   Run the real graph.

   Let your actual routing, nodes, tool registry and guards process those outputs.

3. 03

   Assert on behavior, not text.

   Check which node ran, which tool was called, with which arguments, under which role. Check that a forbidden tool call was refused.

4. 04

   Test the guards with hostile outputs.

   Script the fake model to return an internal ID, a tool name in the reply, or a call to a tool that does not exist. The guard should catch it.

5. 05

   Test every role and channel.

   If cleaners and owners have different tool sets, script the same request for each and assert on the difference.

6. 06

   Keep fixtures fresh per test.

   A scripted iterator that is shared across tests runs out and causes confusing failures.

7. 07

   Make it fast.

   Tier 1 should finish in the time people are willing to wait for a pull request check.

05How to run Tiers 2 and 3

Real-model evals need a dataset of representative requests with expected routes or reference answers, and graders for each. Code-based graders work for routing: did the model pick the expected agent and tool? Model-based graders work for quality, but they need calibration against human judgment, and their prompts should be reviewed like code.

Run these tiers when something changes that affects the model's behavior: a prompt edit, a new tool, a model upgrade. Pin the dataset version in the pipeline so a dataset update does not silently change what "passing" means. Anthropic's guidance on agent evals adds a useful distinction: capability evals tell you how far the agent has to go, while regression evals should stay near a full pass rate, and a drop means something broke.

06Trace everything

Tiered tests catch most regressions, but production will still find new ones. When it does, you need traces that show the exact sequence of model calls and tool calls for that conversation. Log by subsystem, redact customer data and tokens before logs are shared, and turn every production bug into a new Tier 1 or Tier 2 case once you understand it.

For more on how agents fail once real users arrive, see why AI agents fail in production.

07What we see in production

We build and run the AI operations assistant inside an established short-term rental operations platform, used by property owners, cleaner managers and cleaners across web, mobile, SMS, Slack and voice. The assistant acts only through the client's existing backend APIs, exposed as a tightly defined tool registry, with tools allowlisted per role and per channel. In shared property channels a decision node decides whether the AI should answer at all. Guards stop tool names, IDs and internal fields from appearing in replies.

That design gave us a lot of deterministic code worth testing, so we built testing in tiers. Tier 1 unit tests use a scripted fake tool-calling model and run on every push in CI, checking graph logic, routing plumbing and guards without paying for or waiting on a real model. Tier 2 routing evals and Tier 3 model-graded quality evals use real models and run on demand. Agent behavior is traced in an LLM observability tool and in cloud logs by subsystem, and a runbook requires redacting customer data and tokens before logs are shared.

The traces paid for themselves. In a log-verified QA review, 6 of 7 reported bugs were confirmed with root causes traced to exact tool calls, which fed the fixes that followed. The client reported that the app and AI agent were live, and the same tiers now run whenever the assistant moves to a newer model or gains a new channel.

Frequently asked questions