AI agent monitoring: catch regressions in production

01What should AI agent monitoring trace for every run?

Treat one agent run as one trace, and give every step inside it its own span: each model call, each retrieval, each tool call, each permission check and each human approval. If you only log the final answer, you can see that something went wrong but not where. This is part of our larger guide, LLMOps for SaaS: keep AI features working in production.

The OpenTelemetry GenAI semantic conventions give you a shared vocabulary for most of this. They define operation names such as invoke_agent, execute_tool and chat, and conventions for retrieval spans. They also define attributes such as gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.response.finish_reasons and error.type. Two details matter. The conventions are still marked as in development, so names can change between versions. And capturing message content and tool arguments is opt in, because the specification treats instructions, user messages and model outputs as sensitive.

The standard describes the model call, not your product. A live SaaS agent needs product context on the same trace, or you cannot answer the questions your customers and security reviewers will ask. Here is the checklist we use for every run:

1. 01

   Identity and scope:

   tenant ID, user role and the permission check result for every tool the agent tried to use.

2. 02

   Versions:

   model version as returned by the provider, prompt version, tool schema version, retrieval index version and any admin configuration version.

3. 03

   Context:

   the IDs of the documents or records retrieved, not only the text, so you can tell whether a bad answer came from bad retrieval.

4. 04

   Tool calls:

   arguments, result status, retries and whether the call changed data or only read it.

5. 05

   Human steps:

   whether an approval was requested, who approved or rejected it, and how long it waited.

6. 06

   Outcome:

   task completed, handed to a person, abandoned or failed, recorded as a field rather than inferred later.

7. 07

   Cost and time:

   tokens, model cost and latency per step and per run, so you can compute cost per successful task, not just cost per call.

8. 08

   Feature flag and release:

   which flag variant and which deploy the run belonged to, so a regression can be traced to a change.

Cost per successful task is the number we watch most closely on the cost side. A cheaper model that fails more often and loops through extra tool calls can raise it while the cost per call goes down.

02Which user signals reveal quality problems?

Thumbs up or down is rare and not a fair sample. The stronger signals are the things users do anyway:

• Edits to drafts. When a person approves an AI draft, measure how much they changed it. Rising edit distance on the same task type is often the first sign of drift.
• Rejections at approval. If a step needs human approval, the rejection rate is a direct quality measure that costs nothing to collect.
• Rephrasing and retries. A user who asks the same thing three ways in a row did not get what they needed the first time.
• Handoffs to a person. A handoff with full context is good design. A rising handoff rate on a task the agent used to handle is a regression.
• Abandonment. Sessions that end mid task, especially after a tool error or a long wait.
• Support tickets that mention the AI. Tag them and join them to traces by tenant and time.

Always slice these signals by tenant, user role and task type. An average across all customers can stay flat while one large tenant quietly gets worse.

03How do we score production traces without reviewing everything?

You do not need to read every trace. Score in three layers, from cheap and complete to expensive and selective.

Code checks on every run. These are deterministic and fast. The output matches its schema, tool arguments are valid, cited sources exist, every tool call passed its permission check, no tenant boundary was crossed, the run stayed under its step and cost limits. A failure here is not a matter of opinion, so it can drive alerts directly.

Automated judges on a daily sample. An LLM judge scores a sample of traces against a written rubric: did the answer use the retrieved context, follow the business rules and do what the user asked. Draw the sample in two parts: a random slice for an honest trend line, and a slice weighted toward runs with bad user signals. Research supports the approach and shows its limits: Zheng and colleagues found that a strong model judge agreed with human preferences more than 80% of the time, about as often as people agreed with each other. They also showed position, verbosity and self enhancement biases.

Human review on a smaller set. A domain expert reviews a fixed number of traces each week, including some the judge scored well. This finds failure types nobody wrote a rubric for, and it measures whether the judge still agrees with people. If agreement falls, fix the judge before you trust its trend line.

Every trace a human labels as a failure should become a candidate case for your regression set. That is how monitoring and testing feed each other.

04How do we detect regressions after a model or prompt change?

Keep a fixed, versioned regression set: real cases drawn from production (de-identified or synthetic where the data requires it), each with what good looks like. Start with a few dozen cases that cover your most common and most expensive task types, and add every production failure you fix.

Replay the set on every change that can alter behavior, not only code deploys:

• a new model version, or a provider update to the version you use
• a prompt or system instruction change
• a tool schema or API change
• a retrieval index rebuild or a change to the source documents
• an admin configuration change, if customers or staff can edit scripts, rubrics or instructions

Compare results against the baseline per case and per slice, not only as an overall average. A version that scores better on average while failing three cases it used to pass has still regressed, and those cases are often the ones a customer will notice.

After the set passes, release behind a feature flag to a small group of tenants and compare live signals between versions for a few days before widening. Record gen_ai.response.model on every span, and if it changes without a release on your side, run the regression set. This is not hypothetical: a Stanford and Berkeley study found that the behavior of GPT-3.5 and GPT-4 changed substantially between their March and June 2023 versions.

05What should trigger an alert or a rollback?

Not every quality dip deserves a page. Google's SRE workbook gives a useful pattern: alert on how fast you are burning an error budget, not on single bad events. Its recommended starting points page on 2% of the budget consumed in one hour (a burn rate of 14.4) or 5% in six hours, and open a ticket on 10% consumed over three days. We apply the same idea to agent quality, with one exception: boundary and safety failures page on the first occurrence.

The thresholds below are starting points we tune against each product's own baseline after the first weeks of traffic. They are not benchmarks.

Decide the rollback path before you need it. The fastest safe rollback is usually a feature flag that returns users to the previous version or the workflow they had before the agent, not a redeploy.

Most of these checks answer the reasons AI agents fail in production.

06What we see in production

Three patterns from our own work, with the clients kept anonymous.

A scoring agent is itself an agent you have to monitor. An insurance sales training business wanted its agents to practice voice calls with an AI prospect and get consistent scoring against its own checklist, instead of uneven feedback from one coach to the next. Later it wanted real sales calls scored like practice calls. We started with a short proof of concept with hard coded scenarios, then moved scripts, scenarios, checklists and global AI instructions into the database so admins control the AI. Feedback runs as a LangGraph workflow of small steps.

Every admin edit to a script or checklist changes behavior just as a deploy does, so configuration versions belong on the trace and in the regression scope. And a fixed checklist rubric makes the score itself something you can trend and audit.

A deterministic anchor makes drift visible. A founder building an automated claim checking product had a prototype with no database, cache, API or audit history, and no labeled data to measure its accuracy. We began with a gap analysis before writing code. We kept rule based pattern matching as a fast, explainable anchor, added semantic embeddings, and limited the LLM to suggestions with a confidence value, merged through calibrated weights. We also designed a signed audit trail for every claim. When an explainable layer runs beside the model, disagreement between the two becomes a measurable signal.

Knowledge changes are behavior changes. For a cloud storage product, we built an assistant inside the team chat tool that answers product, pricing and FAQ questions from the company's own wiki and pricing content. It returns a stored FAQ before any model call. An assistant like this can drift when the knowledge changes, even if the model does not, which is why we trace retrieval index versions alongside model versions.

On an agent we built that turns company data into internal apps, the deployment package traces agent runs with OpenTelemetry and Langfuse, and each organization's model choice is recorded in an append-only audit log.

Across this work, one rule holds. As our co-founder and CTO, Rana Umar Majeed, sets out in why AI agents fail in production, control has to rise with capability as an agent moves from read and recommend to prepare, modify and execute. Monitoring follows the same ladder: an agent that only reads can be sampled lightly. One that changes customer data needs every run checked by code, human approval where the risk calls for it, and a replayable audit trail.

Frequently asked questions