All insights

Self-hosted LLM

Self-hosted LLM vs API: where should your AI run?

Most "our data must stay with us" rules are met by your own cloud account, not your own GPUs.

Co-founder and CEO, Datics AI7 min read

Short answer

The self-hosted LLM or API decision has three answers, not two: call a provider API, run a managed model inside your own AWS, Azure or Google Cloud account, or serve open-weight models on GPUs you control. For most established SaaS products, the managed model in your own cloud account keeps customer data where contracts require, with the least operations work. Self-host when a contract or a network boundary rules out every managed endpoint.

On this page
  1. 01Three places your product's AI can run
  2. 02Self-hosted LLM vs API vs your own cloud, side by side
  3. 03What "the data must stay with us" really means
  4. 04How we decide where a capability runs
  5. 05What we see in production
  6. Frequently asked questions

01Three places your product's AI can run

The question rarely starts with engineering. It starts with a customer's security questionnaire: where do prompts go, who can read them, and does anything leave our region or our cloud account? That is the first question in our article on enterprise AI security, and the answer depends on where the model runs.

A provider API. Your product calls OpenAI or Anthropic directly. It is the fastest start, you get new models the day they ship, and there is no infrastructure to run. Prompts and outputs leave your cloud account and are processed under the provider's terms. Enterprise plans add controls on retention and training, so get in writing what applies to your account before a customer asks.

A managed model in your own cloud account. Amazon Bedrock, Azure OpenAI in Microsoft Foundry and Google Vertex AI serve the same families of models through an endpoint in your account, under the identity, logging and network controls your product already uses. AWS states that model providers have no access to the accounts where Bedrock runs their models, so they do not have access to customer prompts and completions. Microsoft states that prompts and completions are not available to OpenAI or other model providers. Google states it will not use your data to train or fine-tune models without your permission. Claude models are sold through all three clouds, so model choice rarely forces the cloud.

A self-hosted open-weight model. You run a model such as Llama, Qwen or Mistral on GPUs in your own cloud account or data center, served by an engine such as vLLM. No prompt reaches any model service, including your cloud's own. In exchange, you own GPU capacity, scaling, patching, model upgrades and the evaluation of every new model version.

02Self-hosted LLM vs API vs your own cloud, side by side

Provider APIManaged model in your cloudSelf-hosted open-weight model
Where data goesTo the provider's service, outside your accountStays in your cloud account and chosen region, under your access controls and logsNever leaves hardware you control
Contracts and BAAsThe provider's terms; some enterprise plans offer zero retention and a BAA, confirm per accountYour existing cloud agreement; a BAA covers the services the cloud lists, so confirm the model service is on that listNo model vendor in the data path; your own controls carry the audit
Cost shapePer token, nothing idlePer token, or reserved capacity for steady volumeGPUs by the hour, busy or idle; cheaper per token only at high, steady use
Operations workLowest: keys, rate limits, a fallbackLow: networking, quotas, region and model versionsHigh: GPU capacity, serving engine, scaling, patching, upgrades, on-call
Model quality and choiceNewest frontier models firstMany frontier and open models; availability varies by region and versionOpen-weight models only; strong on narrow tasks, usually behind the largest hosted models on hard reasoning
LatencyPublic internet round trip, shared capacityPrivate networking inside your region, capacity you can reserveShortest network path, but only as fast as the GPUs you sized for peak
Usually right forPrototypes and data that carries no contract termsMost established SaaS products with enterprise customersCustomers who require their own network or hardware, or steady high volume

The middle column is our default for a reason. It answers the customer's question with the identity, logging and private networking your product already has, and it keeps the newest models within reach. Which cloud to use is a separate question, decided mostly by where your data and identity already live; we compare the three in Bedrock vs Azure AI vs Vertex AI for existing SaaS.

Self-hosting is a real option, not a fallback. It is the only option when a customer requires that nothing leave its own network, and it can be cheaper when a narrow task runs all day on a small model. But the GPU bill arrives every hour whether customers use the feature or not, so price it on measured volume, the way we describe in how much production AI costs.

03What "the data must stay with us" really means

Before anyone buys GPUs, pin down the requirement. In our experience, the sentence "customer data must stay with us" hides five separate questions, and each one points to a different answer.

  • Which data? Customer records, the prompts built from them, the model's outputs, embeddings and logs can each carry different rules.
  • Stay where? Inside your cloud account, inside one region, inside one country, or inside a customer's own network.
  • Away from whom? The model provider, the cloud provider, other customers, or your own staff.
  • For how long? Zero retention, or a fixed window such as abuse monitoring logs.
  • Written where? A contract clause, a data processing agreement, a BAA, or an answer on a security questionnaire that sales already sent.

Most answers point to your cloud account and a region, which the managed option meets. Only "never leaves the customer's network" or "no third party at all, including the cloud's model service" points to a self-hosted LLM, and our article on private LLMs covers how to run one. For health data, the deciding document is the BAA and the services it covers, which we walk through in HIPAA-ready AI. If you want to score your own case line by line, our free scorecard Self-host a model or call an API? does it in ten lines.

04How we decide where a capability runs

We decide per capability, not per company. A support search feature and a feature that reads patient records can run in different places inside the same product.

  1. 01

    Start from the contract and the customer question.

    Write down what your largest customers require, in their words, before looking at models.

  2. 02

    Default to a managed endpoint in the cloud where the data already lives.

    Moving records and search indexes to reach a model is usually the costlier choice.

  3. 03

    Test two or three models on your own cases.

    Include an open-weight model if self-hosting is on the table, so its quality is measured, not assumed.

  4. 04

    Price each option as cost per successful task.

    Count retries, failed attempts and evaluation runs, and compare at today's volume and at next year's.

  5. 05

    Keep the model a setting.

    Call models through one interface in your product's code, with the test set beside it, so a move from API to managed to self-hosted is a configuration change and a test run.

  6. 06

    Self-host only when a requirement or the numbers demand it.

    Then size GPUs for peak load, plan model upgrades, and name who is on call.

Step five is the one that keeps every later choice open. A team that owns its test set and its model interface can change where the AI runs in days. A team that wired one provider's SDK through the product cannot.

05What we see in production

Across our tagged AI projects, 17 of the 52 ran in the client's own cloud account. Our default is that AI runs in the client's cloud, on the model the client picks, and we change the cloud only where the AI, its cost or a customer's review requires it. Three of our case studies show the three shapes this takes.

For Datascape, a data consultancy that works under HIPAA as a business associate, we built physician contract extraction inside the client's own restricted cloud. It uses Amazon Bedrock in the client's AWS account, with public traffic denied at the VPC level. In the case study's words: "We work only with the AWS permissions the client grants, under a signed business associate agreement."

For a data copilot that answers inside the customer's network, first proven with a Fortune 500 company on its own operations data, the model runs on Azure OpenAI inside the customer's own environment, so company data never goes to a public model provider, and every answer carries its SQL.

For NenoDoc Lexicon by Neno Technologies, a knowledge workspace for government archives, the need was different: any tenant had to be able to move from cloud GPUs to local hardware. We rebuilt it to run fully self-hosted on one client-owned GPU, with Qwen2.5-14B served by vLLM, and the local footprint fits under 30 GB of GPU memory. Archive teams get cited answers in French and English.

Two of the three ran managed models in the client's own account. The third self-hosted because its tenants needed a path to their own hardware. That is the order we recommend.

Frequently asked questions

Is a self-hosted LLM cheaper than an API?

Only at high, steady volume. A provider API or a managed model charges per token, so a quiet month costs little. A self-hosted model charges by the GPU hour whether it is busy or idle, plus the engineering time to run it. Price both as cost per successful task on your measured volume before deciding.

Do Amazon Bedrock and Azure OpenAI train on our prompts?

Their documentation says no. AWS states that model providers have no access to Bedrock's model accounts or to customer prompts and completions, and Microsoft states that prompts and completions are not available to OpenAI or other model providers. Google makes a similar commitment for Vertex AI. Read the retention and processing location details for each service and region you use.

Can we start on an API and move to a self-hosted LLM later?

Yes, if the product calls models through one interface and keeps its own test set. Then moving is a configuration change followed by a run of your test cases on the new model. Teams that spread one provider's SDK through the product find the move much harder, so set up the interface on day one.

Do we need a BAA to use an LLM with health data?

If you are a covered entity or business associate and the model service will receive protected health information, you need a BAA with the vendor that processes it, covering that specific service. Self-hosting removes the model vendor from the data path, but your cloud agreement and your own controls still apply. Our HIPAA-ready AI article covers the details.

How Datics helps

Datics specializes in brownfield AI: production AI built into established software, one capability at a time, in your codebase and under your name.

›References (4)
  1. Amazon Bedrock User Guide: Data protection
  2. Microsoft Learn: Data, privacy and security for Foundry Models sold by Azure
  3. Google Cloud: Gemini Enterprise Agent Platform (Vertex AI) and zero data retention
  4. vLLM documentation

Related work

Case studies from our projects on the same subject.

Datascape

Healthcare · Contract data extraction pipeline

Compensation terms read from any physician contract, inside the client's own restricted cloud

Extraction code broke whenever a contract changed layout. One RAG pipeline now reads any contract, inside the client's own AWS account.

Forward deployedBrownfield AIDocument intelligence
Read case study

Cosmic AI

Other software · Agentic data copilot

Business teams ask their databases plain English questions without data leaving their network

Answers used to wait on an analyst. Now teams ask in plain English, and the copilot answers inside their network with the SQL shown.

AI-native productAgentic workflowsRuns in client cloud
Read case study

NenoDoc Lexicon by Neno Technologies

GovTech · Archive knowledge workspace

Archive teams get cited answers in French and English on one client-owned GPU

Sensitive archives could not go to an outside AI. Now teams ask in French or English and get answers cited to the page, all on the client's GPU.

AI-native productCited answersDocument intelligence
Read case study

Your customers already expect AI from your product.

Let us make it real inside the product you run today.