Why AI Agents Fail in Production: It's Not an Intelligence Problem

Production is not a prompt

In a demo, an agent gets a request, reasons about it, calls a tool, and produces an impressive result.

In production, that same agent has to operate inside a real business: customers, permissions, legacy systems, human approvals, network failures, retries, timeouts, regulatory controls, audit requirements.

And one thing the demo never had: consequences.

If a chatbot repeats an answer, it's mildly annoying. If an agent sends a customer communication twice, approves a workflow twice, or executes a transaction twice, that's a production incident, a compliance conversation, and sometimes a refund.

There's a second complication: a probabilistic decision-maker now sits inside the execution path. The network can fail. The API can time out. State can go stale. And the agent can make a different decision when the workflow is retried.

So the real production question is no longer how we make the model more accurate. It is how we let AI reason probabilistically while business execution stays reliable, authorized, and explainable.

This is the moment Agentic AI stops being an AI problem and becomes a systems architecture problem.

The 5 patterns that break good AI demos

These are the failure modes I see over and over when AI moves from prototype to production.

1. 01

   Confusing model intelligence with system reliability

   A model can reason perfectly while the workflow still fails. An API times out. State goes stale. A retry duplicates an operation. Reliability is a property of the whole system, never of the model alone.

2. 02

   Letting the agent become the source of truth

   Agent memory and conversation history are useful context. They should never decide whether a consequential business operation actually happened. Authoritative state belongs in the systems that own it.

3. 03

   Granting tools without defining authority

   Being able to call a tool is not permission to execute everything that tool exposes. Tool access, user authority, and business policy are three different concerns, and most implementations collapse them into one.

4. 04

   Designing the happy path, not the recovery path

   The interesting question isn't whether the agent can finish this workflow. It's what happens when it gets 70% through and something fails.

5. 05

   Observing infrastructure, but not decisions

   HTTP 200 tells you almost nothing. For consequential AI you need to know what the agent understood, which tool it picked, which policy applied, whose authority it acted under, whether a human approved, and what actually changed.

If you recognise three or more of these in your own stack, the model isn't your bottleneck.

This brought me back to distributed systems

None of these are new engineering problems.

Distributed-systems engineers have wrestled with partial failure, concurrency, retries, consistency, and recovery for decades. What's new is that we've inserted probabilistic reasoning into the execution path. Which brought me back to CAP.

What CAP actually says

If you've seen the CAP theorem before, it was probably drawn as a triangle and you were told to pick two. That framing is memorable and slightly wrong. Here's the version that's actually useful.

Picture your system running in two data centres. The network link between them drops. Each side is still healthy, still serving traffic, but neither can see the other. That's a partition.

Now a write lands on one side. A read for that same record lands on the other. You have exactly two options, and no third:

• Refuse the read until the link is restored, because you can't be certain the data is current. You just sacrificed availability to protect correctness.
• Answer it from local state, knowing it may be stale, because a response is better than a hang. You just sacrificed consistency to stay responsive.

Make your guarantees and trade-offs explicit before things go wrong, because when they do, the system will make the choice for you.

The real CAP decision is per operation

There is no clever engineering that gets you both consistency and availability while the partition lasts. That is not an implementation limitation. It is a proven constraint.

Consistency means every read returns the latest coherent state, or fails outright. Availability means every request gets a response, even if it might be stale. Partition tolerance means the system keeps functioning while the network is broken.

Partition tolerance is not a choice. Networks fail. You do not get to opt out of reality. So the real decision reduces to consistency or availability, and critically, that decision is made per operation, not once for the whole system.

Your product catalogue can happily serve slightly stale data. Your payment ledger cannot. Same system, opposite choices, both correct.

Why an AI agent hits the same fork

Your agent fires a payment operation and the call times out. The agent genuinely does not know whether the payment committed. It is, in the most literal sense, partitioned from the truth.

Most agent implementations never make this choice deliberately. They default to availability: the model reasons that it has no record of finishing, so it tries again. A duplicate transaction is the result.

CAP has a well-worn answer here, and it isn't a better prompt. It's an authoritative record the agent can query instead of guessing, plus an idempotency key that makes the retry safe even when the guess is wrong.

That's the whole argument in miniature: the hard problems in agentic AI already have engineering answers. They're just answers from distributed systems, not from prompt engineering.

• Stop and wait for authoritative confirmation. Correct, but the workflow stalls and the customer is left hanging. That's choosing consistency.
• Proceed on its best guess. Responsive, but it may have just paid a supplier twice. That's choosing availability.

Agents add a category of their own

Agentic systems inherit every classic distributed-systems problem: partitions, stale state, duplicate messages, race conditions, and partial transactions. Then they add a new category: non-deterministic reasoning, wrong tool selection, hallucination, context contamination, and prompt injection.

A retried network request can be made idempotent. A retried reasoning step may choose a completely different path.

A production AI agent has to manage uncertainty in both execution and reasoning.

CARE: a design lens for production agents

I've started using a simple mental model when designing production agentic systems. I call it CARE.

C: Consistency. Can I trust the state, and prevent contradictory or duplicate actions?

A: Autonomy. How much freedom am I actually granting this agent to reason, delegate, and execute?

R: Reliability. Can the workflow still produce a predictable business outcome when components fail?

E: Explainability. Can I reconstruct the context, reasoning, tools, policies, approvals, and resulting state change?

CARE is not a theorem, and it isn't a replacement for CAP. It's a design lens, a way of forcing the right questions early.

Push autonomy up, and consistency, reliability, and explainability all get harder at once. That trade-off should be a deliberate architectural decision, not an accident of implementation.

The architecture changes when AI moves from recommending to doing

Compare two sentences: Based on these conditions, I recommend reviewing your exposure. I have adjusted the portfolio.

A few words apart in English. Worlds apart architecturally.

The first produces information. The second changes enterprise state, pulling identity, authorization, policy, idempotency, durable state, human approval, compensating actions, auditability, and blast radius into your AI architecture.

The control boundary has three layers. The probabilistic layer reasons, plans, and selects tools. The deterministic layer applies identity, delegated authority, policy, approval, idempotency, transaction validation, and audit. Enterprise action updates records, sends messages, or executes transactions.

The model can request an action. It should never be the component that authorizes it.

Don't try to make the LLM deterministic. Make the boundaries around consequential actions deterministic.

The failure mode that's worse than a timeout

A timeout at least tells you something went wrong. Here's the version that doesn't.

The agent invokes a payment-like operation. The downstream system commits it successfully. Microseconds later, the agent runtime crashes before it records that the step completed.

Nothing looks broken. There was no error, no partition, no timeout. The business state is correct and the agent's state is simply missing.

If recovery just asks the model to continue from what it remembers, the model attempts the payment again, confidently, because from its point of view the step never happened.

This is why making the model smarter doesn't help. The model isn't wrong. Its record is wrong, and no amount of reasoning ability can fix a fact that was never durably written down.

The fix is ordinary, unglamorous systems engineering: idempotency keys, durable workflow state, authoritative transaction records, reconciliation, transactional messaging, and compensating actions.

On recovery, the agent should ask whether operation 87423 actually committed, not reason that it doesn't remember finishing and should try again.

That is the difference between agent memory and authoritative system state.

Identity and authority matter as much as tool access

An enterprise agent should not be handed a powerful service account just because it needs to talk to business systems.

User identity, workload identity, and delegated authority are three separate things. An agent may authenticate as a workload, but the effective permission for any action should still reflect the authority of the person or process it represents.

Not every tool deserves the same level of control. It is useful to think of agent capability as a ladder: Read, Recommend, Prepare, Modify, Execute.

As you climb it, consequence increases. So should authorization, validation, approval, idempotency, transaction control, reconciliation, and audit. Most teams apply one uniform policy across the entire ladder. That's how you end up over-governing search and under-governing payments.

An AI agent should never gain more authority simply because a model sits between the user and the system.

Observability has to explain behavior, not just uptime

Traditional observability answers: is the service up, how long did it take, which dependency failed?

Agent observability also has to answer: what did the system decide, what context shaped that decision, which tools were available, what policy permitted the action, was a human involved, and what ultimately changed?

For agentic systems, the trace becomes part of the evidence.

The durable architecture is probably the layer around the model

Most enterprise AI conversations are about which model, which agent framework, which vector database, which cloud.

Many of those choices will become interchangeable. Models will improve. Frameworks will churn. Protocols will mature.

The layer that lasts is the one surrounding them: identity, authorization, state, tool contracts, policy, evaluation, observability, audit, resilience, and data boundaries.

Those are the contracts that let the intelligence layer keep evolving without rebuilding the enterprise around every new model release.

The question is no longer whether AI can reason

Models will get better. Agents will get more capable. Tool use will get more reliable.

But as AI moves from answering to recommending to planning to acting, the architectural question changes: where does probabilistic reasoning end, and deterministic enterprise control begin?

Five questions I'd run against any agent you're about to put in front of real operations:

1. 01

   Can I trust the state this agent is acting on?

   Identify the authoritative system and how the agent confirms current state before action.

2. 02

   How much autonomy am I actually granting?

   Make the level of delegated freedom an explicit product and architecture decision.

3. 03

   What happens when it fails halfway through?

   Design recovery, reconciliation, and compensating actions before rollout.

4. 04

   Can I reconstruct exactly what happened, months later?

   Retain the context, decision, policy, approval, tool call, and resulting state change.

5. 05

   Who really holds the authority to act?

   Keep identity, workload access, and delegated user authority distinct.

Let AI reason probabilistically. Make enterprise authority deterministic.

A production system, not a very good demo

If you can't answer all five questions, you don't have a production system yet. You have a very good demo.

The intelligence layer is new. The need for good systems engineering is not. If anything, the more autonomous AI becomes, the more distributed-systems discipline it demands around it.