RAG vs fine-tuning: which one does your product need?

This decision gets made backwards surprisingly often. Fine-tuning sounds like the serious engineering option (you are training a model, after all) and retrieval sounds like a workaround. In practice it is the other way round for most product work.

What each one actually changes

Retrieval-augmented generation changes what the model can see. At request time you search your own data, put the relevant passages into the prompt, and ask the model to answer from them. The model’s weights are untouched. It is, at heart, a search problem with a language model on the end of it.

Fine-tuning changes how the model behaves. You show it several hundred or several thousand examples of input and desired output, and it adjusts its weights toward that pattern. It learns shape: the structure of your output, your tone, the distinctions your domain cares about.

The distinction that matters: retrieval adds knowledge, fine-tuning adds behaviour. Almost every “the AI does not know our business” complaint is a knowledge problem, and knowledge problems do not respond to fine-tuning.

Why retrieval is the default

Your data changes. A customer updates their address, a policy is revised, a document is superseded. With retrieval, the next request picks up the new version because it reads from the source. With a fine-tuned model, the old fact is baked into the weights until you retrain.

Retrieval also lets you show your sources, which does more for user trust than any accuracy improvement. A user shown three passages and a summary can verify the answer themselves. A user shown a confident paragraph from nowhere has to take it or leave it.

And permissions work. Retrieval respects whatever access rules already exist in your application, because it is your query that fetches the context. A fine-tuned model trained on everything knows everything, and cannot be made to forget it for a particular user, which is a compliance problem waiting to be discovered.

When fine-tuning genuinely earns its place

There are real cases, and they share a shape: the behaviour you want is hard to specify in words but easy to demonstrate.

Rigid output formats. If every response must be a particular structure and prompting gets you to 95% compliance, fine-tuning will get you closer, on a smaller and cheaper model.

Domain classification. Sorting support tickets into your own twenty-two categories, where the boundaries are institutional knowledge rather than anything a definition captures. A few thousand historical examples teach that better than any prompt.

Voice, consistently. A specific register, held across thousands of outputs without a 600-word style prompt attached to every call.

Cost and latency at volume. A fine-tuned small model that matches a large model’s performance on your one narrow task, at a fraction of the price per call. This is a real and underused reason, but it is an optimisation, which means it comes after something works.

How to choose, concretely

Ask what a failure looks like.

If the failure is “that is not true” or “that is out of date” or “it does not know about our client”: retrieval. Every time.

If the failure is “that is correct but formatted wrong”, or “it keeps hedging when we need a decision”, or “it put this in the wrong category and a person would not have”: fine-tuning is on the table, once you have the examples to prove the pattern is consistent.

If the failure is “it is too slow” or “it costs too much”, look at your context size first. Teams routinely send far more retrieved text than the answer requires, and trimming it is an afternoon’s work rather than a training pipeline.

The order to build in

Retrieval first, always, because it is what makes answers correct and correctness is the thing you cannot fake. Get accuracy where you need it against a real eval set. Only then, if the remaining complaints are about form rather than fact, consider fine-tuning, and reuse that same eval set to prove the tuned model is actually better, not just different.

The teams that get this backwards spend a month building a training dataset, ship a model that writes beautifully formatted incorrect answers, and conclude that AI is not ready for their domain. It usually was. The knowledge just never made it into the prompt.