Fine tuning vs RAG: when should you train on your own data?

01Fine tuning vs RAG: two different problems

Fine tuning and retrieval augmented generation are often presented as rival answers to the same question. They are not. They fix different problems.

Retrieval augmented generation, introduced in research from 2020, pairs a language model with a retriever that fetches relevant documents at query time, so the model answers from evidence rather than from memory alone. It is the right tool when the problem is missing knowledge: the model does not know your help center, your contract terms or today's inventory. Knowledge changes, and retrieval lets you update it by updating the index, without retraining anything.

Fine tuning changes the model itself by training it further on examples. It is the right tool when the problem is behavior: the model knows the language but handles a specific task, format or style poorly, and better prompts do not close the gap. OpenAI's own optimization guide treats fine tuning as an optional step in a loop that starts with evaluations and prompt engineering, and it lists the cases where fine tuning earns its place: you have more examples than fit in a prompt, you want shorter prompts for cost and latency, you need to train on data you do not want to send in every request, or you want a smaller model to do one task well.

02How we decide in a live product

We do not start with either. We start with a baseline and a measurement, because without one there is no way to tell whether fine tuning improved anything. Our order of work:

1. 01

   Define the task and the metric.

   For classification, that means labels, a test set and a score such as accuracy or F1. For question answering, it means questions with known correct answers.

2. 02

   Run a pre-trained model with careful prompting.

   Many tasks are solved here. If not, the baseline tells you how far off you are.

3. 03

   Find the type of failure.

   If the model answers with wrong or missing facts, add retrieval. If it understands the facts but misreads the input, consider a specialized model or fine tuning.

4. 04

   Route by difficulty.

   Efficient models handle volume, and a larger model is held back for hard cases and summaries. This often closes much of the quality gap without any training.

5. 05

   Collect labeled data from real use.

   Build the feedback path into the product so users can flag wrong results, and store those corrections for review.

6. 06

   Fine tune only when the data and the gap justify it.

   Train on reviewed corrections, compare against the baseline on the same test set, and keep the old model ready as a fallback.

Step five is the one most teams skip, and it is the one that decides whether fine tuning is ever possible. Fine tuning needs labeled examples from your own domain. Public datasets rarely match how your customers write. The only reliable source of those labels is the product itself.

There is also a vendor point worth noting. Hosted fine tuning ties a custom model to one provider's training service, and providers change those services over time. OpenAI's optimization guide itself notes that its fine tuning platform is winding down for new users. A team that keeps its own labeled dataset can retrain elsewhere. A team that only kept the fine tuned model cannot.

03What we see in production

On a new sentiment analytics platform for social and influencer marketing campaigns, the core of the product is classification: sentiment, emotion, toxicity and sarcasm on public comments and reactions, across many channels and post formats. Social language is full of sarcasm, slang, emojis and mixed languages, so plain sentiment models misread it. The founder also required metrics that are correct and defensible, because the product positions itself as expert grade.

After a paid discovery, we recommended a hybrid AI engine. Efficient pre-trained models handle volume. Fine tuning for social language improves them on the kind of text the platform sees. A larger language model is held back for hard cases and for the report summaries, which users can edit before sharing. Scores are normalized across channels and formats so campaigns can be compared fairly, and reactions get their own scoring because each platform offers different reactions.

Fine tuning on the product's own labeled data is planned as a later step, not a launch requirement. What makes it possible is a small feature: a feedback button that lets users flag wrong results. Those flags are stored for review and model improvement. Over time they become a labeled dataset drawn from exactly the comments, platforms and campaigns the product serves. The platform is not live yet, and no accuracy results have been published, so the honest summary is that the engine was designed to start useful on pre-trained models and to earn its fine tuning from real corrections.

This is a classification product, so retrieval plays a small role in the sentiment engine itself. Where retrieval does matter is in the admin assistant that answers natural language questions about the data, which shows a confidence indicator based on how much data supports the answer. That split is typical: classification tasks lean toward specialized or fine tuned models, and question answering over changing data leans toward retrieval.

04Mistakes we see teams make

• Fine tuning to add knowledge. Training facts into a model is slow, hard to update and hard to audit. If the problem is missing facts, use retrieval.
• Fine tuning without a baseline. If you never measured the pre-trained model, you cannot show the fine tuned one is better.
• No feedback path. Without stored user corrections, you will never have the domain labels fine tuning needs.
• Ignoring cost and latency. A smaller fine tuned model can be cheaper per call, but training, evaluation and retraining are ongoing costs. Our article on how much production AI costs in SaaS covers how those costs show up.
• Locking into one provider. Keep the labeled dataset and the evaluation set as your own assets, so the model can move. See when an AI workload should move between clouds for the wider picture.

05A practical rule of thumb

If a product manager can describe the failure as "it did not know," add retrieval. If they describe it as "it knew, but it got it wrong in the same way again and again," measure it, collect corrections, and consider fine tuning once you have enough of them. In both cases, ship the feedback button first.

Frequently asked questions