AI token cost: forecast LLM spend as usage grows

01What unit should we use to measure AI cost?

Your provider bills you in tokens. Your customers do not buy tokens. If you track only token spend, you see the bill grow but cannot tell which feature, customer or plan drives it, or whether the money produced anything useful. This is part of our larger guide, LLMOps for SaaS: keep AI features working in production.

We measure inference economics as cost per successful task:

Cost per successful task = all model spend for a feature (including retries, failed attempts, guardrail calls and evaluation runs) / tasks that met the acceptance bar

"Successful" has to be defined per feature first. For drafting, it might mean the user accepted the draft with light edits. For extraction, the fields passed validation. For an agent, the action was approved and completed without a human redoing it.

This unit changes decisions. A cheaper model that fails one task in three can cost more per successful task than a stronger model that rarely fails, because every failure is paid for and then paid for again.

Three views of the same data serve three readers:

• Cost per call is for engineers tuning prompts, context and model choice.
• Cost per successful task is for product leaders deciding whether a feature earns its place.
• Cost per tenant per month is for finance, set against what that tenant pays.

02How do we attribute cost to tenants and features?

Attribution has to happen at the moment of the call. A monthly provider invoice knows your API key, not your customers.

Every model call should pass through one internal gateway that writes a usage record with these fields:

• Tenant ID, user ID and user role
• Feature ID and the step within that feature
• Plan tier at the time of the call
• Model and provider
• Uncached input, cached input, cache write and output tokens
• Latency, retry count and the task outcome once it is known

With that record, cost per tenant and cost per feature become ordinary queries. Allocate shared costs, such as embedding a common knowledge base or nightly evaluations, by each tenant's share of usage, and keep them visible as their own line.

Two details matter for multi-tenant products. First, caching interacts with tenancy: Anthropic documents that caches are isolated between organizations and, on several platforms, by workspace as well. That means the way you separate tenants can change how often context is reused. OpenAI offers an optional cache key that helps with separate cache accounting. Second, the outcome field is the one teams skip, and it is the one that turns spend into cost per successful task. Join it from product events you already log, such as accepted, edited, approved or abandoned.

03Which architecture choices cut inference cost the most?

The biggest savings come from sending less work to the model, sending it to the right model, and not paying twice for the same context. The table ranks the levers in the order we would apply them in an established product, starting with the ones that remove work entirely.

• Baseline. Three calls on the mid-size model: extract, calculate, write. Cost per task: $0.096.
• Move the calculation into code. Two model calls remain. Cost per task: $0.064.
• Route extraction to the small model. Cost per task: about $0.034.
• Cache the 10,000 stable tokens, assuming 80 percent of calls hit the cache and the rest pay the write rate. Cost per task: about $0.019.

If 80 percent of tasks succeed, cost per successful task falls from $0.12 to about $0.023. These figures illustrate the arithmetic; they are not client results. Your numbers will differ, which is why you measure them.

Research supports lever 2. The FrugalGPT paper found that fees across popular model APIs can differ by two orders of magnitude. The RouteLLM paper reported routers that cut cost by more than two times in some benchmark cases without lowering response quality.

Provider documentation shows why levers 3 and 5 matter. Anthropic lists cache reads at 0.1 times the base input price on most models, and 0.05 times on its current Opus and Sonnet models. Five-minute cache writes cost 1.25 times base input. Its Batch API takes 50 percent off input and output tokens. OpenAI says cached input is discounted by up to 95 percent and that caching is on by default for supported models.

A worked example.

Take an illustrative feature that turns account data into a written summary. Prices are Anthropic list prices as published in October 2026. A mid-size model costs $2 per million input tokens and $10 per million output tokens. A small model costs $0.10 and $0.50 for prompts up to 100,000 tokens. Cache reads cost 0.05 times base input on this mid-size model and 0.1 times on the small one. Each call carries 12,000 input tokens (10,000 of them stable instructions and schema) and returns 800 output tokens.

04How do we forecast AI token cost as usage grows?

Forecast from product behavior, not from last month's invoice.

Monthly inference cost = Σ over features (active AI users × tasks per user per month × cost per successful task) × (1 + overhead)

Overhead covers evaluation runs, embeddings, background jobs and spikes. The example uses 15 percent; replace it with measured values after a few months.

1. 01

   List each AI feature separately.

   A chat assistant and a nightly report generator grow on different curves.

2. 02

   Measure adoption.

   Active AI users per feature, not total seats.

3. 03

   Measure task frequency.

   Tasks per active user per month, by plan tier and tenant size. This driver moves fastest, especially when agents chain steps.

4. 04

   Measure cost per successful task

   from the usage records above.

5. 05

   Build three scenarios.

   Expected, high adoption and heavy usage. Agree which one finance budgets against.

6. 06

   Reforecast monthly

   with actual values, and flag any feature whose cost per successful task moved past an agreed threshold.

Continuing the example: 15,000 active AI users running 40 tasks a month is 600,000 tasks. At the baseline $0.12 per successful task plus 15 percent overhead, that is $82,800 a month. With the four levers applied, at about $0.0234 per successful task, it is about $16,100.

Now grow usage over a year to 30,000 active users at 60 tasks each, or 1.8 million tasks a month. The baseline design reaches $248,400 a month. The optimized design reaches about $48,400. Growth multiplied both lines by three, but only one stays inside a margin plan. The architecture sets the slope of the cost curve; usage growth decides how far along it you travel.

05How should pricing and usage limits protect margins?

Once you know cost per successful task, you can price against it. In the example, a typical user at 40 tasks a month costs $4.80 on the baseline design and about $0.94 on the optimized one. A heavy user at 400 tasks a month costs $48 or about $9.37. Flat, unlimited access priced for the typical user is exposed to the heavy one.

Guardrails that protect margins without hurting the experience:

• Include an allowance in each plan that covers typical use with room to spare, sized from measured task frequency by tier.
• Price heavy use separately, through credits, a higher tier or a paid AI tier, so revenue rises with cost.
• Set per-tenant budgets with alerts at 50, 80 and 100 percent of allowance, visible to the account team.
• Degrade gracefully at the limit. Move the tenant to a smaller model, queue non-urgent work for batch, or confirm before long agent runs, instead of failing outright.
• Show usage in the product, so the conversation about upgrading starts from shared facts.

Review one short table per feature monthly: active users, tasks, success rate, cost per successful task, revenue and gross margin. If a feature's cost per task rises while its success rate falls, that is an engineering problem before it is a pricing problem.

For the full picture, see how much production AI costs in SaaS and how to reduce AI infrastructure cost in production.

06What we see in production

The most reliable cost reduction we know is to give the model less to do. Three builds show the pattern.

A consumer nutrition planning app. Users describe their goals in text or by voice and receive a personalized meal plan with calories and nutrients. We split the work into LangGraph nodes: one detected the input type, one extracted keywords into a structured schema, and one checked that the required profile details were present before any food search ran. Calories, ingredient weights and daily values were computed in code from external nutrition data. The model only rephrased the results as a structured table. Arithmetic in a model is slow, costly and hard to trust; in code it is exact and close to free.

A claim verification product. Claims had to be broken into atomic units, classified against a fixed ontology, scored and verified, with a provable record. We kept rules as a fast, explainable anchor, added semantic embeddings, and limited the LLM to suggestions with a confidence value, merged through calibrated weights. The model became one signal among several, called where it adds judgment rather than everywhere.

A records platform for a public-sector archive. Staff needed traceable answers over large, sensitive bilingual records, extraction of people, places and acronyms, and writing tools that never alter names, dates or figures. The documents could not go to an outside AI service, so the data boundary decided the hosting model before cost did. We settled architecture ownership first, consolidated the backend on one framework, and chose a self-hosted OCR model after side-by-side tests, including on handwriting. Self-hosting shifts the cost question from tokens to capacity and utilization.

On an agent we built that turns company data into internal apps, starter app ideas come from catalog metadata with a rule based fallback instead of table rows. Every generated query passes an EXPLAIN check before any sample run, so the model is never handed data it does not need.

One more lesson from a multi-clinic intake product we took from prototype to SaaS: clinic isolation, role-based access and billing arrived together. Per-tenant metering belongs in that same layer, built at the start, not added after the first surprising invoice.

Frequently asked questions