AI evaluation: build your test set before the first release

01What an AI evaluation set is and why it comes first

An evaluation set is a versioned list of inputs taken from your product, each paired with a definition of good behavior and a way to grade it. It is the test suite for the part of your system that does not behave like normal code. The same input can produce different output from one run to the next, so asserting one exact return value is not enough. This is part of our larger guide, LLMOps for SaaS: keep AI features working in production.

Public benchmarks and leaderboards compare models in isolation. They say very little about whether your feature answers your users correctly, inside your permissions, from your data. The evaluation set we mean measures your task, on your inputs, against your definition of success.

It comes first for three practical reasons.

• It defines done. Until you write down what a correct answer looks like, the team is arguing about taste. A written case with a pass rule ends the argument.
• It turns prompt work into engineering. A prompt edit that fixes one example can quietly break five others. Only a fixed set of cases shows you that before your users do.
• It makes later decisions cheap. Switching models, changing retrieval or trimming the prompt to save cost become measurable comparisons instead of gambles.

OpenAI's guidance names the opposite approach as an anti-pattern: judging quality by whether it seems to work, or waiting until after launch to add evaluation. We agree. The first set does not need to be large. It needs to exist before the first release, built from the product rather than from imagination.

02Collecting cases from real product use

An established product already holds most of the cases you need. The work is to find them, clean them and label them.

Good places to look:

• Support tickets and help center searches. These show the questions users actually ask, in their own words.
• Manual workflows the feature will assist. If staff read a document and fill in a form, every past form is a candidate case with a known correct output.
• Application logs and search queries. These show volume, so you can weight the set toward what happens most often.
• The people who do the work today. A short session with a support lead usually surfaces the hard cases that logs hide.

1. 01

   Pick the single task the first release performs, written as one sentence.

2. 02

   Pull a few dozen real requests for that task from tickets, logs and past manual work.

3. 03

   De-identify them and record the role, tenant and any context the feature will receive.

4. 04

   Add edge, adversarial and permission cases until each failure type you fear has at least one example.

5. 05

   Write the expected behavior and the pass rule for every case before you run anything.

6. 06

   Run the current prompt once, read every output, and fix cases whose rules turn out to be unclear.

7. 07

   Commit the set to the repository with a version number, next to the prompt it tests.

Then cover the edges deliberately. Both the Anthropic and OpenAI guides recommend mixing typical cases with edge cases and adversarial ones. Examples include empty or irrelevant inputs, inputs that are far too long, requests that try to push the model off its task, and ambiguous requests where two experienced staff members would disagree.

In a product with customers, add the cases that only exist because the product is real. Record the user role and the tenant on every case. Include a request where the answer exists but the current user is not allowed to see it; the correct behavior is to decline. Include a request that mentions another customer's records. These cases catch the failures that end enterprise deals, and no public benchmark contains them.

Synthetic cases can fill gaps, such as a scenario that is rare in your logs. Label them as synthetic and keep real cases as the core of the set.

Treat privacy as part of collection. De-identify cases before they leave production and store the set with the same access controls as the code. Our own engineering runs with no standing access to client production data, so evaluation sets have to be built within that boundary.

A workable sequence for a first set:

03Defining pass and fail for each case

A case without a pass rule is only an example. For each case, write down four things:

• Expected action. Answer, ask a clarifying question, decline, or hand the request to a person.
• Must include. The facts, fields or source references a correct answer contains.
• Must never include. Data from another tenant, invented figures, instructions the user is not permitted to follow.
• Allowed sources. Which documents or records the answer may draw on.

Start with binary judgments. Pass or fail labels are far easier for reviewers to agree on than a five point scale, which invites debates about whether an answer is a three or a four.

Next, separate failures by severity. Anthropic's guidance suggests defining which errors are an inconvenience and which are serious, and setting targets for each. Some failures should have a target of zero: showing data across tenants, taking an action the user is not allowed to take, or presenting an invented figure as fact. Others, such as a slightly verbose answer, can be tolerated at some rate. The overall pass rate does not need to be 100 percent; it is a product decision about which failures you can accept.

The table below shows how this looks for a typical first feature that answers questions from existing product data.

04Automated checks versus human review

Use the cheapest grader that gives a trustworthy answer, and move up only when you must.

Code based checks come first. Anthropic's documentation ranks code based grading as the fastest and most reliable method, while noting that it lacks nuance. Much of what matters is checkable in code. Code can confirm that the output parses, required fields are present, a source link is included, the right tool was called with the right arguments, no restricted value appears and the handover flag is set.

Model graded checks come second. Some qualities, such as whether an answer is grounded in the retrieved document, need judgment. A grader model with a detailed rubric can handle these at scale. OpenAI's guide notes that models are better at choosing between options than at open ended generation, so frame the grader's job as classification: does this answer use only the provided sources, yes or no. Researchers at UC Berkeley found that model generated evaluators inherit the problems of the models they evaluate and still need human validation, so test the grader against cases a person has already labeled.

Human review never goes away. Because it is slow, focus it: write the first labels, check a weekly sample of model graded results, and look at every new kind of failure. When agreement between grader and human drops, fix the rubric before trusting the numbers again.

05Running it on every change

The evaluation set only protects you if it runs whenever something that affects output changes: prompts and templates, the model or its settings, retrieval, tool definitions, permission and filtering logic, and the source documents themselves.

OpenAI's guidance calls this continuous evaluation: run the set on every change, watch production for new behavior, and grow the set over time. The set can run in a standard CI pipeline, with results tracked against prompt versions.

A few habits make this work:

• Compare against a baseline, not an absolute. Compare each change to the last accepted run, case by case. A change that raises the average while breaking a permission case is a regression.
• Gate on severity. Block the merge if any zero tolerance case fails, or if the pass rate falls below the baseline.
• Run variable cases more than once. A case that passes once may fail on the next run. Repetition shows consistency, not luck.
• Record cost and latency with quality. Anthropic lists response time and price among the success criteria worth defining. We measure inference economics as cost per successful task, which only makes sense when you know which tasks succeeded.
• Turn every production failure into a case. De-identify it, write its pass rule and add it to the set.

Missing evaluation is one of the main reasons AI agents fail in production.

06What we see in production

Our position is simple: the hard part is not the model, it is everything around the model. In the agent harness we describe, the flow runs from user intent through product context, allowed tools, permission check, approval and action, then into evaluation and audit. An AI capability that skips evaluation is not ready for customers who depend on the product.

Several patterns from our anonymized work shape how we approach a first evaluation set.

• Test query sets before users do. In a cloud file-management product, we tested query sets and tuned prompts in a controlled environment first, and only then did the client test the Slack bot in its own workspace. Those query sets are where a first evaluation set starts.
• Human approval produces labeled cases. In a clinical documentation product for healthcare practices, transcription and patient history draft the clinical note, and the clinician reviews, edits and approves it before release. That review step is a steady source of expert labeled examples.
• Tenancy belongs in the test set. A healthcare intake product we built for many clinics needed strict separation between clinics and role based access. There, the most important cases are the ones where the right answer is to show nothing.
• Evaluate each stage of a pipeline. A claim verification engine we built classifies claims against the founder's own ontology, combining rules, embeddings and model suggestions through tunable weights. Testing only the final answer hides which stage failed.
• Test the trust signals, not only the features. On a verifiable AI platform we took to production, every claim in an answer shows its tier, confidence and source. QA tests those signals too: a claim receipt showing high confidence on a false statement was logged as a release bug.

As a capability moves up what our CTO, Rana Umar Majeed, calls the capability ladder (read, recommend, prepare, modify, execute), each step needs more policy, approval, idempotency and audit. The evaluation set has to climb with it: a feature that modifies or executes needs cases that check the action, the arguments and the approval path, not only the answer.

The same thinking sits behind the CARE lens from his writing on why agents fail in production: Consistency, Autonomy, Reliability and Explainability. An evaluation set turns those properties into numbers you can track release by release. We hold ourselves to the same standard in public: we do not publish estimated outcome percentages, only figures from work already delivered.

Frequently asked questions