How do you run LLM testing with real user prompts?

01Why demo prompts are not a test

A demo prompt is written by the people who built the assistant. It uses the words they expect, asks one thing at a time and avoids the edge cases they already know about. That makes it a poor test. This is part of our larger guide, LLMOps for SaaS: keep AI features working in production.

Real users ask several things at once, change their mind halfway through, refer back to something three turns ago and use words your team never considered. They also care about different things than the builders do. A builder checks whether the answer is plausible. A user checks whether it is right for their situation.

Anthropic's guidance on building evaluations makes the same point: test cases should match the real distribution of tasks and include edge cases such as ambiguous, missing or poor input. Hamel Husain's widely read essay on evals adds that teams should log every conversation and keep reading those traces, because looking at real data is where most failures are found.

02Build the LLM testing set from real conversations

The best test set is built from what users already do. We follow a simple loop.

1. 01

   Collect real prompts from pilot users, support tickets, sales calls and existing search logs.

2. 02

   Group them by intent, such as search, comparison, follow up, policy question or hand off.

3. 03

   Write the expected behavior for each, not an exact answer

   which data it should use, which filters it should apply and what it must never do.

4. 04

   Add multi turn scripts, because many failures only appear on the second or third message.

5. 05

   Run the full set on every change to the prompt, the tools, the model or the data.

6. 06

   Add every new failure from production to the set, so it cannot come back unnoticed.

Expected behavior matters more than expected wording. Two good answers can be phrased differently. A bad answer usually breaks a rule you can check: it shows a unit that is not in stock, ignores a filter, forgets a constraint or invents a price.

03What to check, and how

Different failures need different checks. A small number of automatic checks catches most regressions; human review catches the rest.

Code based checks are cheap and fast: they look at tool calls, returned identifiers and fields. Model based grading can help for tone and helpfulness, but check it against human labels before you rely on it.

04Let customers run their own structured tests

The most valuable test we have seen is one a customer designs. When a customer writes its own script of questions, based on how its staff and buyers really talk, it tests what they care about and in their order of importance. It also produces findings that the customer already believes, which makes prioritization easier.

To make this useful:

• Give the customer a simple format: the prompt, what they expected and what happened.
• Ask for whole conversations, not single questions.
• Review every finding with the customer and agree which ones block rollout.
• Re-run their script after each fix and share the result.

05Keep the test set alive after launch

A test set is only useful while it reflects how people use the product today. Usage changes after launch: new customers bring new vocabulary, seasons change what buyers ask, and new features open new paths through the conversation.

Treat the test set like code. Give it an owner, review it on a schedule and version it alongside the prompts and tools it tests. When you change the model or the provider, run the full set before and after, and compare results case by case rather than looking only at the overall pass rate. A model upgrade that improves the average can still break a case a key customer depends on.

Also watch for drift in the opposite direction. If every case passes for weeks, the set may have stopped finding problems. Pull a fresh sample of real conversations, read them, and add the ones that surprise you. Reading real conversations is slow, but it is the step that finds the failures nobody thought to write a test for.

Finally, share results with the people who own the outcome. A short weekly note that lists new failures, fixes and open issues keeps product, engineering and customer teams working from the same picture.

06What we see in production

We built a multi-tenant AI sales assistant for a dealership software company serving RV, marine and vehicle dealers. In our own demos the assistant looked fine. Then a dealer ran its own structured test and found context loss between turns, repeated units and wrong location handling. Early dealer tests also showed wrong classes being returned.

That single test reset our priorities. Instead of adding features, we focused on conversation state and filters: conversation state moved into a database backed checkpointer, inventory search moved to an index with explicit filters, and the flow was rebuilt as a tool using agent so filters are visible in each call and can be checked.

We tested against real dealer prompts and pushed to stabilize with a few live dealerships before scaling. Chat history with AI summaries, exit reasons in the analytics dashboard and token usage logged per conversation now give the team real traces to review. The assistant has been live on dealer websites in the U.S. and Canada since about January 2026, with leads and appointments flowing into the client's CRM.

Before a product has real users, golden prompts stand in for them: on an agent that builds internal apps from company data, an evaluation harness runs a set of retail cases through the real build agent in throwaway databases and scores answer accuracy against reference SQL, repairs, fallbacks and time.

07Decide what "ready to scale" means

Before rolling out to all customers, agree in writing what the assistant must pass:

• The full test set, with zero failures on rules you treat as hard (invented stock, wrong prices, leaked data).
• An agreed pass rate on softer checks, such as helpfulness.
• Clean results from at least one customer's own script.
• Logging in place, so every new failure becomes a new test.

A pass rate below 100 percent on soft checks can be a reasonable product decision. A failure on a hard rule is not. For more on the patterns that break assistants after launch, see why AI agents fail in production.

Frequently asked questions