RAG or fine-tuning? How to choose for a business assistant

When a business wants an AI assistant that knows its products, policies or procedures, the first technical question is usually the same: should we fine-tune a model on our data, or retrieve relevant content at question time and let a general model answer from it?

Both approaches work. They solve different problems, and choosing the wrong one tends to cost weeks. This guide explains how we decide.

What each approach actually changes

Retrieval-augmented generation (RAG) leaves the model unchanged. Your documents are split into passages, indexed, and searched for each question. The most relevant passages are placed into the prompt, and the model answers using them.

Fine-tuning changes the model itself by training it further on examples you provide. It adjusts how the model responds — tone, format, the steps it follows — far more reliably than it adds new facts.

That distinction drives almost every decision below.

Start with retrieval when the problem is knowledge

If the assistant needs to answer questions about information that lives in documents — product specifications, pricing rules, HR policies, support articles — retrieval is usually the right foundation:

• Content changes without retraining. Update a policy document and the next answer reflects it.
• Answers can cite sources. Users and reviewers can check where an answer came from.
• Access control is possible. Retrieval can be filtered by the user's role, so a student never sees staff-only material.
• The assistant can say "I don't know". If nothing relevant is retrieved, the system can decline instead of guessing.

Fine-tuning cannot easily offer any of these. Facts learned during training are hard to update, impossible to cite and not governed by permissions.

Consider fine-tuning when the problem is behaviour

Fine-tuning earns its place when the gap is consistency of behaviour rather than missing knowledge:

• Output must follow a strict structure every time, such as a specific JSON schema or report format.
• The assistant must adopt a specialised style or classification scheme that prompting alone does not hold.
• A smaller, cheaper model must perform a narrow task as well as a larger one, at high volume.

Even then, fine-tuning is often combined with retrieval rather than replacing it: the tuned model provides the behaviour, and retrieval provides the current facts.

A decision checklist

Before choosing, answer these questions:

1. Does the information change more than a few times a year? If yes, favour retrieval.
2. Do answers need citations or audit trails? If yes, retrieval is required.
3. Should different users see different information? If yes, retrieval with permission filtering.
4. Is the main failure that answers are wrong, or that they are badly formatted? Wrong facts point to retrieval quality; wrong format points to prompting first, then fine-tuning.
5. Do you have hundreds of high-quality example inputs and outputs? Without them, fine-tuning is premature.

Where retrieval projects actually go wrong

In our experience the model is rarely the weak point. Retrieval quality is. The common causes are:

• Poor chunking. Passages split mid-table or mid-procedure lose the context needed to answer.
• Keyword blind spots. Pure vector search can miss exact product codes or names; combining it with keyword search fixes much of this.
• Stale or duplicate content. Three versions of the same policy produce three different answers.
• No evaluation set. Without a list of real questions and expected answers, every change is a guess.

Building a small evaluation set from real user questions — even fifty of them — is the single most useful step in any assistant project. It turns "the answers feel better" into a measurable comparison.

Our default approach

For most business assistants we start with retrieval over a curated set of sources, with hybrid search, permission filtering, citations and an evaluation set built from real questions. We consider fine-tuning only when that system is working and a specific, measured behaviour problem remains.

Starting this way keeps the first version simple to update and easy to trust — which matters more to adoption than any single model choice.