Fine tuning vs RAG is a choice about what your assistant needs to do. RAG, short for retrieval-augmented generation, supplies relevant source material when the model answers; fine-tuning means further training a model on examples. Start by testing a clear task instruction, or prompt. Then use the result to decide whether to investigate missing source material, inconsistent output or both before commissioning extra work.
One supplier proposes training an AI system for your business, while another proposes a searchable knowledge base. Those descriptions do not define the task the finished system must perform or what would count as a failed result. We start with that written task. It gives you something concrete to compare when the proposals arrive.
What does fine tuning vs RAG change?
RAG changes the material supplied for an answer; fine-tuning changes the model through training. OpenAI’s accuracy guide separates a lack of context from inconsistent behavior. This is a useful starting distinction for a buyer, provided you keep room for overlapping needs. A document-based assistant may also need a consistent way to write its reply.
According to Amazon Bedrock’s explanation of knowledge bases, retrieval selects relevant material and includes it in the generation input. For a proposed assistant answering questions about a returns policy, ask to see which approved passage supports the answer. A source reference still needs checking against what the answer says.
Fine-tuning uses examples to further train a model on how to respond. OpenAI’s guide describes it as an option for improving behavior on a task. Think of a desired handoff note with an agreed structure: the supplier needs examples of acceptable notes and the inputs used to produce them. That is a different deliverable from a folder of policy documents.
Neither label tells you whether the demonstration meets your requirements. Ask the supplier to identify what the proposed component changes, show the material it uses and explain how you will check the result. Keep the discussion tied to the actual work you are buying.
Start with a task and a simpler demonstration
Describe the input and the acceptable output before choosing an approach. Anthropic’s prompting guide starts with clear success criteria, a test using examples and a draft prompt. We use that order to prepare a buying brief: write the instruction, supply approved material and inspect what the simplest proposed setup can do. This is our suggested evaluation method, not a reported benchmark.
For an illustrative returns assistant, the input could be a customer’s question and the applicable policy. An acceptable draft answers only what that policy supports and hands exceptions to a person. For an illustrative sales handoff, the input could be an enquiry and an approved note format. The output should preserve the enquiry’s meaning and leave missing details unfilled.
Have the process owner write what makes each output acceptable. Include a request the assistant should decline or hand over. Use material you are permitted to share, with unnecessary personal details removed. A demonstration that depends on information you cannot provide is not yet a workable proposal.
If you already have an assistant giving unreliable answers, use the separate guide to checking why AI gives wrong answers before deciding what to commission.
Choose the next test from the remaining gap
Use the observed failure to choose what to test next. AWS recommends starting with retrieval for questions that reference custom documents, and notes the difficulty of keeping training aligned with frequently changing material. It also allows retrieval and fine-tuning together. The table below suggests a next test for the gap you found in your demonstration.
| What the demonstration still needs | What to test next |
|---|---|
| Information from approved, changing documents | Retrieval that supplies the relevant current material |
| A repeatable output that stays inconsistent after clearer instructions and examples | Training on accepted examples, if the chosen model supports it |
| Source material and a more consistent response | A combined setup, with a check for each requirement |
When the simpler demonstration already meets the requirements, keep that option in the comparison. Ask what additional problem a proposed training job or retrieval system would solve. The choice can also remain open while you prepare the missing inputs.
AWS notes that fine-tuning is not available for every model. Ask the supplier to name the provider and model, explain where and how the system will run, and confirm current support for the proposed training and deployment.
Who maintains the system after launch?
Compare the responsibilities each proposal leaves with your team. For retrieval, name the person who approves source documents and checks that updates reach the assistant. For training, name who prepares accepted examples and reviews later changes. Record who grants access, who handles failures and how staff return the task to the manual process when the system is unsuitable.
We put those questions into AI implementation planning before choosing components. A useful brief records:
- The task, permitted inputs and expected output.
- The authoritative sources, their owner and their update process.
- Examples of acceptable work and unacceptable omissions or additions.
- The conditions for declining a request or handing it to a person.
- The acceptance checks, the reviewer and the material to be handed over.
For work involving a live order or contact record, ask how the proposed system will obtain the permitted current record. Include that requirement in the CRM integration part of the brief. A proposal to train on exported records still needs an explanation of how later changes will be handled.
Ask where documents, searchable copies, examples and outputs will be stored, who can access them, and how removal is handled. Record the answers for the actual deployment. Do not treat either architecture name as a decision about hosting or permission.
Check the proposal on unseen requests
Keep test requests out of the material used to configure or train the proposed system. Ask each supplier to run those requests against the same acceptance criteria. Have the process owner inspect the outputs, the inputs used to produce them and the corrections needed. This checks the proposal on your task; it does not prove performance on every future request.
- Keep a baseline. Save what the manual process and the simpler prompt produce, alongside their inputs and review notes.
- Check a source update. Change an approved test document and repeat the affected question. Ask the supplier to show which version was used and how the update reached it.
- Check the output rules. Reword a request and leave some information out. Inspect whether the note keeps the required structure without inventing a detail.
- Test access and scope. Try a request outside the agreed scope and a user who should not receive a restricted document. Inspect the actual refusal or handoff.
- Record the work your team must keep doing. Note corrections, review time and who must maintain the materials. Compare these with the baseline before expanding the work.
When a check fails, record the failed requirement and ask for a revised demonstration. Do not pay for a more elaborate approach solely because its name sounds more capable. Accept the scope you can verify, or keep the task with people while the missing conditions are resolved.
Where we come in
We start by writing the task, the permitted data and the acceptance checks together. Then we compare proposed approaches against that brief and document who will maintain the sources, examples and evaluation. A useful first step is to write a real request and approve an example of the result you need.
