What LLM evaluation should a chat agent pass before launch?

01Start with journeys, not prompts

Most teams start testing a chat agent by asking it questions and judging the answers by feel. That finds the obvious problems and misses the ones that matter in production. Those are the cases where the agent answers one question well and loses the thread three turns later, calls the wrong tool, or skips a step that a person should approve. This is part of our larger guide, LLMOps for SaaS: keep AI features working in production.

We start from documented use cases instead. A use case is a journey with a beginning, an end and the steps in between. On a multi-agent market research and trading assistant we built, the journeys read like this: review my portfolio, show me how a macro change affects it, suggest new ideas, then prepare an order. Another went from the daily opportunities to a deep dive on one company to a trade sized to the user's risk profile.

Writing those journeys down before the build decided three things that no prompt could. It decided which specialist agent owns each step, which agent hands off to which, and exactly where the confirmation step sits before anything touches a live brokerage. That last point is the main reason to write journeys early. If the confirmation step is a design decision, it can be tested. If it is something the model is asked to remember, it cannot.

The same discipline applies to agents inside an established B2B product. For the wider picture of why agents break once they leave the demo, see why AI agents fail in production.

02Turn each journey into test cases

A journey becomes a test file. Each step gets an input, the expected route through the agents, and a rule for what counts as a pass. We write these as checks a script can grade wherever possible, because a suite that needs a person to read every transcript will not be run often enough to protect a release.

1. 01

   Write the happy path. The user goes from start to finish with clear requests. The test checks that each step reaches the right agent and that the final state is correct, for example an order prepared

2. 02

   Add the variations real users produce

   vague requests, a change of mind halfway through, two questions in one message, typos, and follow-ups that depend on an earlier turn.

3. 03

   Add the boundary cases. The user tries to skip the confirmation, asks for something the product does not support, or leaves out information the agent needs. The pass rule here is often a refusal or a

4. 04

   Add data failure cases. A market data feed returns nothing, a source is stale, a tool times out. The agent should say what it could not get rather than fill the gap with a plausible guess.

5. 05

   Record the expected handoffs. In a multi-agent system the route matters as much as the answer. A correct answer from the wrong agent usually means the routing will fail on the next variation.

6. 06

   Freeze the set as a baseline and run it on every prompt, model or tool change, so you can see what a change broke before your users do.

Vendor guidance points the same way. Anthropic's documentation on building evaluations recommends tests that mirror the real distribution of inputs, including edge cases, graded automatically where possible, and a larger number of cases over a few hand-graded ones.

03Choose a grader for each kind of check

Not every check needs a model to judge it. Picking the cheapest reliable grader for each check keeps the suite fast and keeps its results trustworthy.

Model graders are useful for the soft qualities, but only after you have compared their judgments with a person's on a sample. Hamel Husain's widely read guide on evals makes the same point. Log traces, label them and start with simple good or bad labels. Then measure how often the model grader agrees with people before you rely on it.

04Separate analysis from action, and test the line

The most important test in an agent that acts is the one that proves it cannot act on its own. In the trading assistant, agents analyze and recommend, but an order goes to the brokerage only after the user has explicitly consented to connect it and then confirmed the order, with parameters they can edit. Analysis and execution sit in separate layers.

That separation makes testing simple. The test does not ask whether the model decided to wait for confirmation. It checks whether the execution layer can be reached at all without a confirmation event. That is a code path, and code paths can be tested with certainty.

We apply the same rule to any agent that writes data, sends messages or changes records. Put the approval in the product, not in the prompt, and write a test that actively tries to go around it. If that test ever passes, the release stops.

05Score against explicit rules where the domain has them

Some problems come with their own rules, and those rules make excellent graders. Long before large language models, one of our algorithm engineers worked inside the team of an interior design and 3D visual commerce startup on automatic furniture placement. Each candidate room layout was scored by a rule-based fitness function on five interior design principles. They were clearance in front of objects, circulation, grouping of related objects, alignment with walls and other objects, and distribution and rhythm. Layouts that failed the test went back for another round.

The lesson carries straight over to LLM evaluation. When your domain has rules, such as required fields, risk limits, policy constraints or formatting standards, encode them as checks. A grader that knows the rules is cheaper, faster and more consistent than a model asked whether an answer looks right. That work had one more rule worth copying: processing time had to stay low, so the search was kept deliberately small. Latency is a pass rule too, and it belongs in the suite next to accuracy.

06What we see in production

On the market research and trading assistant, built for an early-stage fintech venture, the journeys changed once real users arrived. Early users got lost among too many features, so the product was pulled together into one chat-led flow, the chat opened automatically, and the design moved to mobile first. Every one of those changes altered the paths people took, so the journey tests had to change with them. A test suite written once and never updated measures a product that no longer exists.

Live data is the hardest part to test. That assistant combined prices, fundamentals, filings, news, social posts and macro indicators, and its chat answers cite their sources. The tests that earned their keep checked that a cited source was actually retrieved in that run, and that the agent said plainly when a feed returned nothing. A handful of fixed snapshots of feed data, replayed in tests, made those failures repeatable instead of random.

We also see that the confirmation step is where reviewers spend their attention, and rightly so. Teams that can show a test proving no order path skips confirmation have a much easier security and product review than teams that can only show good transcripts.

07An LLM evaluation release gate you can defend

Before launch, we want a short list of conditions that anyone on the team can check and that do not depend on opinion:

• Every documented journey passes on the current build.
• No path reaches an action without the confirmation event.
• Every figure in a sample of answers traces back to tool output.
• The model grader agrees with human labels on a recent sample.
• Failures found in the last round of user testing are now permanent test cases.

NIST's AI Risk Management Framework treats evaluation as part of design, development and use, not a step at the end. For a chat agent, the intended use is the set of journeys. Test those, keep adding to them as users show you new paths, and the launch decision becomes a reading of the results rather than a debate.

Frequently asked questions