AI model evaluation: how do you choose an LLM to build on?

01Why model choice goes wrong in established products

Most teams pick a model the way they pick a laptop: reputation, a few demos, a benchmark chart. Then they build on it for months. When something breaks in production, nobody can say whether the model was the right choice, because nobody wrote down what right meant.

In an established SaaS product the stakes are higher. The model will sit inside workflows your customers already rely on. Its mistakes show up as wrong numbers in a report, a misrouted request or a confident answer to a question it should have declined. Public benchmarks measure general ability. They say little about your data, your users or your edge cases.

So we treat model choice as the first output of the evaluation, not the input to it.

02AI model evaluation starts with the scorecard

The scorecard is a short document that says what the AI must do and how you will measure it. We write it before any model is chosen, and we keep it in the repository next to the code.

1. 01

   Name the task precisely.

   "Classify sentiment" is not a task. "Classify each public comment as positive, negative or neutral, flag sarcasm and toxicity, and do it in the languages our customers use" is a task.

2. 02

   Pick metrics that match the cost of errors.

   For classification, precision tells you how often a positive label is right; recall tells you how many real positives you caught; F1 balances the two; ROC-AUC tells you how well the model ranks cases across thresholds. Choose which one matters most for your product and say why.

3. 03

   Build a test set from your own data.

   Use real examples, labeled by people who know the domain, including the hard cases: slang, mixed languages, sarcasm, empty fields, long documents.

4. 04

   Set a pass threshold before you run anything.

   Otherwise the threshold drifts toward whatever the favorite model scored.

5. 05

   Add cost and latency to the scorecard.

   A model that wins by a point but costs several times more per task may be the wrong choice.

6. 06

   Write the retraining or re-evaluation plan.

   Say when you will rerun the scorecard: on every model upgrade, on every prompt change, and on a schedule as data shifts.

This matches what model vendors themselves recommend. OpenAI's evaluation guidance describes the same loop: define the objective, collect a dataset, define measurable metrics, run and compare, and keep evaluating on every change.

03Match the evaluation method to the task

Not every task has a clean label. The right grader depends on what the model produces.

Anthropic's engineering guidance on agent evaluation draws the same lines: code-based graders are fast and reproducible, model-based graders handle nuance but need calibration, and human review is the gold standard used to keep the others honest.

04Hybrid choices are usually the answer

When teams run a real scorecard, the result is rarely "one model for everything." More often it looks like this:

A small, efficient model for volume. If you must classify millions of comments or route every message, a pre-trained or fine-tuned smaller model may meet the threshold at a fraction of the cost.

A larger model for hard cases. Escalate the ambiguous inputs, the long documents and the summaries to a stronger model.

Per-customer model choice. Enterprise buyers may want their own model provider. If your scorecard is a harness rather than a spreadsheet, you can test each provider against the same cases before you switch a customer over.

The scorecard also protects you during upgrades. When a vendor releases a new model, you do not debate it. You run it.

For the cost side of this decision, see how much production AI costs in SaaS.

05What we see in production

For a startup building a sentiment analytics platform for social and influencer marketing campaigns, we started with a paid discovery that ended in an AI blueprint. Social language is full of sarcasm, slang, emojis and mixed languages, and the founder required metrics that are correct and defensible.

So the blueprint named precision, recall, F1 and ROC-AUC, plus a retraining plan, as part of the engine recommendation itself, not as something to add after launch. The recommendation was a hybrid engine: efficient pre-trained models for volume, fine-tuning for social language, and a larger language model held back for hard cases and report summaries. Because the scorecard came first, the model choice became a test that can be rerun as the platform collects its own labeled data. A feedback button that lets users flag wrong results feeds the same loop.

For a governed AI builder that turns company databases into internal apps, we built an evals harness that runs golden prompts through the real build agent in throwaway databases. It scores build success, spec validity, query success, answer accuracy against reference SQL, which tables were used, use of personal data, repairs, AI fallbacks and time. Organizations can route to different models, so the same harness is how a model choice is checked rather than assumed. In an internal test, it ran a set of retail test cases with a full pass rate on its baseline model, and it is the baseline any future model has to match.

For an established short-term rental operations platform with an AI assistant already in production, model changes were real events, including a migration to a newer model generation. We built testing in tiers: unit tests on every push, routing evals, and model-graded quality evals, with traces in an observability tool. A model upgrade became a run of the eval tiers, not a leap of faith.

06A short checklist before you commit to a model

• The task is written in one or two precise sentences.
• Metrics are chosen and justified by the cost of each kind of error.
• The test set comes from your own data and includes hard cases.
• The pass threshold was set before any model was run.
• Cost per task and latency are on the scorecard.
• The evaluation runs from the repository with one command.
• There is a written trigger for rerunning it: model upgrades, prompt changes and data drift.

If you can tick every line, the model you pick is a decision you can defend, and replace, with evidence.

Frequently asked questions