LLM security: why treat model output as untrusted input?

01Why LLM security starts with untrusted output

Teams usually secure the input side of an LLM feature first: rate limits, prompt injection filters, content policies. The output side gets less attention, because the response comes from "our" model through "our" code. That is the mistake. This is part of our larger guide, AI security for SaaS: protect data and pass security reviews.

A model response is shaped by everything in its context window: the user's message, retrieved documents, tool results, earlier turns. Any of those can carry instructions the model follows. Even without an attacker, models produce malformed JSON, invent field values, return the right shape with the wrong meaning, or drift when the provider updates a model version.

OWASP names this risk Improper Output Handling and recommends exactly the stance we take: treat the model as any other user, apply zero-trust validation to its responses before they reach backend functions, and encode output for its destination. If a response can end up in HTML, a SQL query, a shell command, an email or another service's API, it can carry an attack into that place.

02Use the typed models you already have

If your backend is a typed API service, for example FastAPI with Pydantic, you already own the right tool. Every request body that comes from a browser or a partner system is parsed into a model with types, constraints and validators. Model responses deserve the same treatment.

The pattern is simple:

1. 01

   Define the shape you expect.

   Write a typed model for the model's response, with the same care as a public request schema: required fields, enums instead of free strings, numeric ranges, string length limits.

2. 02

   Ask for that shape.

   Use the provider's structured output or tool calling features so the model is steered toward the schema. This reduces failures; it does not remove the need to check.

3. 03

   Parse and validate on your side.

   Run the raw response through your model. Type errors, missing fields and values outside allowed ranges fail here, not three services later.

4. 04

   Apply business rules.

   Field validators and model validators can check what types cannot: that a referenced ID belongs to the current tenant, that a date is not in the past, that a status transition is allowed.

5. 05

   Decide what failure means.

   Retry with the validation error, fall back to a safe default, or route to a person. Never pass a partially valid object downstream.

Pydantic supports this directly with field validators and model validators that raise validation errors, which is the same mechanism your request handling already uses. The benefit is consistency: one set of rules guards your data whether it came from a person, a partner or a model.

03Where validation is not enough

Typed validation catches malformed and out-of-range output. It does not catch output that is well formed and wrong. Several other controls matter.

The last row is the important one. If a model response can cause an action, the action must still pass your normal authorization checks under the end user's identity. Validation proves the response is well formed. Authorization proves the action is allowed. You need both.

04Design responses so they are easy to check

The easiest output to validate is output with little freedom. A few habits help:

• Prefer structured flags to prose for decisions. A risk check that returns a list of named flags, each tied to its input, is testable. A paragraph of reasoning is not.
• Separate the decision from the explanation. Let the model write an explanation for the user, but drive the workflow from the structured fields.
• Ask for edits, not rewrites. When the model corrects text, have it return the spans it would change, and apply them in code. Whole-text regeneration hides changes to names, dates and figures.
• Keep identifiers out of the model's hands where you can. Resolve which record a user means in code, and give the model the resolved record rather than letting it pick an ID.

These choices also make evaluation easier, because each field can be compared with an expected value. For the wider picture of why AI features fail on the product around the model rather than the model itself, see why AI agents fail in production.

05What we see in production

One of our engineers worked inside a U.S. AI software company's own engineering team on the backend of its LLM application, a service built on FastAPI, Pydantic, Postgres and GPT-4. That stack is a common one for LLM features in established products, and it puts typed request and response models right next to every model call, which is where this validation belongs.

In products where we built the AI ourselves, the same rule shows up in different forms. For a regulated private investment platform, risk outputs are structured flags only, traceable to their inputs and stored for audit, and AI never approves a deal, a KYC check or a tokenization step.

In a short-term rental operations assistant, guards stop tool names, IDs and internal fields from appearing in replies, and block edit, delete and post actions where no tool exists. In a self-hosted document workspace for government archives, the writing tools let a rules engine detect errors and the model resolve only ambiguous spans; edits are applied as spans and never as regenerated text, and an approved term registry keeps entity names from being changed.

None of these controls trusts the model to behave. Each one checks what the model produced before it matters.

06A checklist for your next LLM endpoint

Before an LLM-backed endpoint goes live, confirm:

1. 01

   Every model response is parsed into a typed model, and parsing failures are handled explicitly.

2. 02

   Validators check tenant ownership, allowed values and allowed state changes.

3. 03

   Output placed in HTML, SQL, email or another API is encoded or parameterized for that destination.

4. 04

   Replies are filtered for internal names, IDs and secrets before display.

5. 05

   Any action triggered by model output passes normal authorization under the user's identity.

6. 06

   High-impact actions need human approval.

7. 07

   Validation failures are logged and counted, so a provider model update that changes behavior shows up as a metric, not a support ticket.

Frequently asked questions