LLMOps for SaaS: keep AI features working in production

01What LLMOps means for a product team

Much of what is written about LLMOps comes from teams that train and serve their own models. Most established SaaS companies do not. They call a hosted model through an API or a cloud platform, wrap it in their own prompts, retrieval and tools, and put it inside a workflow their customers already use.

For that team, LLMOps is closer to release engineering than to machine learning research. The model is a dependency that changes on someone else's schedule. The prompt is code. The retrieval pipeline is code. The evaluation set is a test suite. The cost of each call is a production metric, like latency or error rate. The job is to give all of that the same discipline the rest of the product already has: versioning, testing, monitoring, rollback and ownership.

The difference from ordinary software is that the core component is probabilistic. The same input can produce different outputs, and "correct" is often a judgment rather than an exact match. Every practice below exists to turn that uncertainty into something a team can measure and manage.

02The LLMOps lifecycle at a glance

03Evaluation sets and release gates

Everything in LLMOps rests on evaluation. Without it, every other practice is guesswork. Even the NIST AI Risk Management Framework, written for risk owners rather than engineers, makes Measure one of its four core functions alongside Govern, Map and Manage.

An evaluation set is a collection of real cases from your product: the questions users actually ask, the documents they actually upload, the edge cases that broke the last version. Each case has a definition of a good answer. Some can be checked exactly, such as whether the right record was returned or whether the output is valid JSON. Others need a rubric, scored by a reviewer or by a second model that is itself checked against human scores.

The set does not need to be large. It needs to be representative and stable, so a score today can be compared with last month. Include the cases that must never go wrong, such as answers that cross a tenant boundary.

A release gate turns the set into a decision. No prompt, model or retrieval change reaches production unless it meets the pass bar on the evaluation set, and no single critical case is allowed to fail. To build and score these sets, see what a chat agent should pass before launch, how to choose an LLM to build on and LLM testing with real user prompts. AI agent testing in CI shows how to run them on every pull request.

04Prompt and model versioning

When a customer reports a bad answer, the first question is what was running. If nobody knows, you cannot reproduce the problem or prove it is fixed.

Treat every element that shapes behavior as a versioned artifact: the system prompt, prompt templates, tool definitions, retrieval settings, the exact model identifier and parameters such as temperature. Keep them in version control, review changes like code and record the version with every run. Pin model versions explicitly instead of relying on a floating alias, so the provider cannot change your behavior silently.

This also makes rollback real. If a release degrades quality, you revert to the previous version of the prompt and model together, exactly as you would revert a code deploy. Prompt management that puts domain experts in control shows how to separate prompt changes from code releases without losing that traceability.

05Observability and tracing of runs

Logs that say a request returned 200 tell you almost nothing about an AI feature. You need a trace of each run. It should record the input, the context retrieved and from where, and every tool call with its parameters and results. It should also record each model call with its tokens and latency, the final output and what the user did next.

Standards are emerging here. The OpenTelemetry project maintains semantic conventions for generative AI that define spans, metrics and events for model clients, so traces from different providers and frameworks can share one shape. Adopting a common shape early saves a migration later, and it lets AI traces sit next to the rest of your application telemetry instead of in a separate tool nobody checks.

Traces are also raw material: failed runs become evaluation cases, token counts become cost data. AI observability for agents in production covers what to capture and how long to keep it, since traces can contain sensitive customer text and need the same retention rules as any other customer data.

06Regression detection after model or prompt changes

AI features regress in ways ordinary software does not. A prompt edit that fixes one case can break three others. A new model version can be better on average and worse on the cases your customers care about most. A change to the documents in your retrieval index can shift answers without any code changing at all.

The defense is simple and must be routine: rerun the full evaluation set after any change that could affect behavior, and compare the results case by case, not just the average. A drop on critical cases blocks the release. A drop elsewhere gets reviewed. Production signals add a second line: falling task success, rising rates of edits or rejections by users, more escalations to a human. Watch those trends weekly and treat a sustained change as a regression to investigate, even when no one has deployed anything.

07Cost per successful task and cost controls

Cost per call is the wrong number. A cheap call that fails and is retried three times, or that produces an answer the user throws away, is expensive. The number that matters is cost per successful task: total model, retrieval and infrastructure spend for a feature, divided by the tasks it completed to an acceptable standard.

That metric changes decisions. It shows when a larger model is cheaper overall because it succeeds first time, and when a smaller model for simple steps pays off. It also connects AI spend to pricing and margin, which is where finance will eventually look. We go deeper in how much production AI costs in SaaS.

Controls matter as much as measurement. OWASP lists unbounded consumption among its top risks for LLM applications, including attacks designed to run up usage costs, and recommends rate limits, quotas, timeouts, input size limits and monitoring for unusual usage. In practice that means per-tenant and per-user budgets, a cap on steps per agent run and alerts when spend moves outside its normal band. For specific savings, see how to reduce AI infrastructure cost in production, which AI pricing model works when every answer costs money and how to forecast AI token cost.

08Kill switches and fallbacks

Every AI feature needs a way to stop, and the time to build it is before launch. A kill switch should work at several levels: the whole feature, a single capability, a single tenant and a single model provider. It should be a configuration change that takes effect in seconds, not a deploy.

Stopping is only half of it. Decide in advance what the product does when the AI is off or failing. Often the right fallback is the workflow that existed before the AI arrived: the manual form, the standard search, the queue for a human. What it must never be is a broken screen or a silent wrong answer. Why every feature and agent needs a kill switch covers the triggers and who is allowed to pull them. Why AI agents fail in production explains why recovery belongs in the control plane from the start.

09Model upgrades without surprises

Hosted models are retired on the provider's schedule, not yours. OpenAI, for example, says it gives at least six months of notice before shutting down generally available models. It gives three months for specialized variants and about two weeks for preview models, and names a recommended replacement for each. Six months sounds generous until you remember that the upgrade has to be evaluated, tuned and released alongside the rest of your roadmap.

A clean upgrade follows a fixed path:

1. 01

   Track the deprecation dates for every model you use, and assign an owner for each.

2. 02

   Run the candidate model against your full evaluation set, with your current prompts unchanged.

3. 03

   Adjust prompts and settings where scores drop, then rerun until the critical cases pass.

4. 04

   Compare cost per successful task and latency, not just quality.

5. 05

   Move a small share of traffic, watch production signals, then complete the switch with a rollback ready.

Teams that skip evaluation discover the differences from customers. What keeps an LLM in production working for years covers how to design so that a model change stays a configuration change.

10What we see in production

The most common failure we see is not a bad model. It is an AI feature with no evaluation set, so nobody can say whether a change made it better or worse. Prompts get edited to fix the latest complaint, other cases quietly break, and confidence in the feature erodes until the business stops promoting it.

The second is cost measured too late. A feature launches, usage grows, and only then does anyone divide spend by successful outcomes. By then the pricing has been set. In production systems we have built, measuring cost per successful task during the build, not after launch, is what lets the product team decide whether a capability is included or sold as a paid tier.

The third is the missing off switch. Without per-feature switches and a non-AI fallback, a provider outage turns into an emergency deploy under pressure.

On an agent we built that turns company data into internal apps, an evaluation harness scores every build on query success, answer accuracy against reference SQL, repairs and time. It also compares runs side by side, so the founder has a measured baseline instead of a demo.

Our own loop is build, measure, improve, reuse, then the next capability. We deliver AI work behind feature flags, with evaluation sets that the product team keeps and reruns on every release, a cost estimate per task and logs that let anyone retest what changed. When a product already has AI, we test it on 50 of its own cases and rank the fixes, so the team starts LLMOps from evidence rather than opinion.

Frequently asked questions