What keeps an LLM in production working for years?

01What changes under a long-lived LLM feature

A feature built on a language model sits on top of things your team does not control. Over a few years, almost all of them move:

• Models are retired. Providers deprecate older models on a schedule and name a replacement. OpenAI's deprecations page, for example, gives each retired model a shutdown date and a recommended successor, with notice periods that can be as short as a couple of weeks for preview models.
• Behavior shifts between versions. A newer model can be better on average and still change the format, length or tone of answers your code depends on.
• APIs evolve. Parameters, response shapes and tool calling formats change.
• Prices and limits change. A feature that was cheap can become expensive, or the reverse, which changes which model is the right choice.
• Your own product changes. New fields, new tenants and new workflows all touch what the model sees.

A feature survives this when each of those changes is contained to one place in the code and caught by a test before users notice.

02The structures that keep a feature alive

The key idea is that the rest of the product should never talk to the model directly. It calls an internal function with a typed input and receives a typed output. What happens inside, including which model answers, can change without any caller noticing.

Validation is what makes that promise real. Libraries such as Pydantic define models with typed fields, and when input cannot be made to fit, they raise one error that lists every problem rather than letting bad data through. Applied to both the request and the model's response, that gives you a hard boundary: a model upgrade that changes the shape of its output fails loudly in a test instead of quietly corrupting records.

03How we keep an LLM in production stable through model changes

1. 01

   Wrap every model call behind an internal interface.

   One place in the code knows about the provider, the model name and the prompt. Everything else calls that interface.

2. 02

   Validate inputs and outputs with typed schemas.

   Reject or repair malformed output. Never pass raw model text into your database.

3. 03

   Version prompts like code.

   Prompts live in the repository, are reviewed in pull requests and carry a version that is logged on every call.

4. 04

   Keep a fixed evaluation set.

   Real examples with known good outputs, run on every change and before every model switch.

5. 05

   Test in tiers.

   Fast unit tests on every push. Routing and behavior evals on demand. Slower quality evals, often scored by another model, before releases.

6. 06

   Track deprecations as part of planning.

   Each model in use has an owner and a known shutdown date. Migration is scheduled work, not an emergency.

7. 07

   Switch models behind a setting, with a fallback.

   A per-feature or per-customer model setting, with a default to fall back to, lets you move gradually and roll back quickly.

8. 08

   Trace every call.

   Model, prompt version, tokens, latency and outcome, so a quality change can be tied to its cause.

04What we see in production

From March 2024 to October 2026, one of our engineers worked inside a U.S. company's own team on the backend of its LLM application, built on FastAPI, Pydantic, Postgres and GPT-4. The client owned the product. That stack is a fair picture of how many long-lived LLM features are built: a typed API framework, a validation library at the boundaries, a relational database as the system of record, and a hosted model behind it. An engagement of that length spans several generations of models and API changes, which is exactly why the interfaces and validation around the model matter more than any single prompt.

On a field operations platform for short-term rental cleaning, where our AI assistant runs inside the client's live product, the same principles are visible in more detail. The assistant moved from one agent to a supervisor that routes to domain agents, and later to a design where each channel and role gets its own compiled graph and tool allowlist. It also moved between model generations, including a migration to a newer model family in late October 2025.

What kept the product working through those changes was not prompt wording. It was a tightly defined tool registry that limits the model to real backend endpoints, tests in tiers (unit tests on every push, routing evals, and model-graded quality evals), and tracing that lets the team diagnose bugs from logs. The client reported in September 2026 that its app and AI agent were live.

On an AI builder for internal apps that our team is developing, model choice is a per-organization setting: fast and strong models from more than one provider, with an audit entry when the setting changes and a default fallback. An evals harness runs golden prompts through the real build agent and scores success, accuracy and fallbacks, so a model change can be judged on evidence before it reaches users.

05Mistakes that shorten the life of an LLM feature

• Calling the provider from everywhere. When model calls are scattered across the codebase, every upgrade becomes a search and replace across the product.
• Trusting free text output. If your code parses model output with loose string matching, the next model version will break it.
• No fixed test set. Without known good examples, nobody can say whether a new model is better or just different.
• Ignoring deprecation notices. A shutdown date found late turns a planned migration into an outage.
• Tuning prompts to one model's quirks. Heavy prompt tricks tied to one model rarely survive the next one. Clear instructions and validated outputs travel better.

For the cost side of running LLM features over time, see how to reduce AI infrastructure cost in production. And if you are weighing which cloud AI platform to standardize on, our comparison of Bedrock, Azure AI and Vertex AI for existing SaaS covers the trade-offs.

06A yearly check for every LLM feature

Once a year, or whenever a provider announces a deprecation, we run the same review:

• Which model and prompt version does each feature use, and when is that model scheduled for shutdown?
• Does the evaluation set still reflect how customers use the feature?
• Do the input and output schemas still match the product's data?
• What would it take to switch this feature to another model or provider tomorrow?
• Are cost per call and latency still acceptable?

If the answer to the switching question is more than a settings change and an eval run, the feature has drifted toward a single model, and the next upgrade will hurt.

Frequently asked questions