RAG, fine-tuning, and tool use solve different problems.

They are often discussed as competing techniques because all three can improve an AI application. That framing creates bad architecture decisions.

Retrieval-augmented generation changes what context the model can see at inference time.

Fine-tuning changes how the model tends to behave by adapting it from examples.

Tool use changes what the model can inspect or do outside the model itself.

Those are three different control surfaces.

If you treat them as substitutes, you will eventually use training data to solve a data-access problem, retrieval to solve a behavior problem, or prompts to simulate actions that should have been explicit tools.

The better question is not:

Which technique is best?

It is:

What exactly is missing from the base model: knowledge, behavior, or access to an external system?

The shortest decision rule

Use RAG when the answer depends on information that is private, large, frequently updated, or outside the model’s training data.

Use fine-tuning when the model already has the information it needs but repeatedly produces the wrong style, format, classification boundary, or task behavior.

Use tools when the model must read live state, perform calculations, query an authoritative system, or take an action.

In production, strong systems often combine all three.

What RAG actually changes

Retrieval-augmented generation does not teach the model new facts permanently.

It retrieves relevant material at request time and places that material into the model’s working context.

A typical RAG path looks like this:

  1. ingest documents;
  2. split or index them;
  3. convert them into a searchable representation;
  4. retrieve the most relevant passages for the user’s question;
  5. give those passages to the model;
  6. generate an answer grounded in the retrieved material.

OpenAI’s Retrieval API, for example, performs semantic search over vector stores, while File Search can combine semantic and keyword search over uploaded files before the model generates a response.

The important architectural property is not the vendor implementation. It is that the source material remains external to the model weights.

That makes RAG a strong fit for:

  • internal documentation;
  • policies and procedures;
  • product catalogs;
  • support knowledge bases;
  • contracts and manuals;
  • research collections;
  • frequently changing reference material;
  • user-specific or tenant-specific information.

Use RAG when freshness matters

If a price, policy, product specification, or support document changes tomorrow, you should not need to retrain a model.

You should update the source of truth and let retrieval surface the new material.

This is the central reason RAG is a data architecture technique more than a model customization technique.

What fine-tuning actually changes

Fine-tuning adjusts model behavior from examples.

Instead of giving the model a document during every request, you provide training examples that demonstrate what good outputs should look like. The resulting model becomes more likely to reproduce those patterns on similar inputs.

That can be useful for:

  • stable classification tasks;
  • consistent output structure;
  • domain-specific response conventions;
  • nuanced transformations;
  • style or tone consistency;
  • recurrent instruction-following failures;
  • reducing large prompts when a behavior is stable enough to encode into the model.

OpenAI’s supervised fine-tuning documentation explicitly frames the technique around example inputs and desired outputs, and recommends building evals before investing in training.

That last point is important.

Fine-tuning without a measurable baseline is optimization without a target.

Fine-tuning is not a database

A common mistake is trying to fine-tune a model so that it ‘knows’ a changing body of documents.

That creates several problems:

  • updates require new training;
  • provenance becomes difficult to inspect;
  • corrections are harder to isolate;
  • memorization is not equivalent to reliable retrieval;
  • you lose the clean separation between model behavior and source-of-truth data.

If the problem is ‘the model does not have the latest policy,’ retrieval is usually the more direct architecture.

If the problem is ‘the model sees the right policy but still formats the decision incorrectly,’ fine-tuning may be relevant.

What tool use changes

Tools give the model a controlled interface to capabilities outside its context window.

Function calling, for example, lets an application expose functions through schemas. The model can decide that it needs a function, return arguments, receive the application’s result, and continue from that result.

This is different from RAG.

Retrieval normally answers:

Which information should I read?

Tool use can answer:

What is true right now?

or:

What should I do in another system?

Examples include:

  • checking the current balance of an account;
  • querying an order by ID;
  • calculating shipping or tax;
  • reading a database record;
  • opening a support ticket;
  • issuing a refund after authorization;
  • running code;
  • searching the web;
  • sending a message;
  • changing infrastructure configuration.

The model should not invent these results from its weights, and they often should not be copied into a vector database first.

The authoritative answer already lives in the operational system.

RAG vs. tools: the distinction that matters

RAG and tools can both provide external information, so the boundary sometimes looks blurry.

A useful rule is:

Use retrieval for knowledge. Use tools for state and operations.

Suppose a customer asks:

What is your refund policy, and can you refund order 18472?

The policy is knowledge. It belongs in documentation and can be retrieved.

Whether order 18472 exists, whether it is eligible, how much was paid, and whether a refund can be executed are live operational facts. Those belong behind tools.

A strong system may therefore:

  1. retrieve the refund policy;
  2. call get_order(18472);
  3. compare order state against policy;
  4. ask for approval if required;
  5. call issue_refund(...);
  6. produce a response with the resulting transaction state.

Trying to solve all of that with RAG would be an architectural mistake.

Fine-tuning vs. prompting

Before fine-tuning, test whether the behavior can be solved with:

  • clearer instructions;
  • better examples in the prompt;
  • structured output schemas;
  • a better decomposition of the task;
  • stronger evaluation criteria.

Fine-tuning has real operational cost: dataset construction, quality review, training, versioning, evaluation, rollback, and monitoring.

If a short prompt change fixes the behavior, training is unnecessary.

Fine-tuning becomes more compelling when the desired behavior is stable, repeated at high volume, and difficult to obtain consistently through prompting alone.

A practical decision table

Problem RAG Fine-tuning Tool use
Private documentation Strong fit Usually no Sometimes
Frequently changing knowledge Strong fit Weak fit Sometimes
Exact current account/order state Weak fit No Strong fit
Take an external action No No Strong fit
Consistent output format Sometimes Strong fit Sometimes
Domain-specific classification Sometimes Strong fit Sometimes
Cite source material Strong fit Weak fit Strong if source system supports it
Reduce prompt examples at scale No Strong fit No
Large searchable corpus Strong fit Weak fit Sometimes
Dynamic calculation No No Strong fit

The architecture can be layered

These techniques become more useful when treated as layers instead of alternatives.

Imagine an enterprise support assistant.

It may use:

RAG to retrieve product documentation and internal policies.

Fine-tuning to make escalation decisions follow a stable taxonomy and produce a strict support format.

Tools to inspect the customer’s account, check service status, create a ticket, or execute an approved change.

The model coordinates the layers, but each layer keeps a distinct responsibility.

That separation improves debugging.

If the answer uses the wrong policy, inspect retrieval.

If it sees the correct evidence but repeatedly violates the required format, inspect prompting or fine-tuning.

If it reports the wrong current balance, inspect the tool integration and source system.

Without this separation, every failure looks like ‘the model was wrong.’

The evaluation strategy should match the layer

Each technique also needs a different evaluation target.

Evaluate RAG on retrieval quality

Measure whether the system retrieves the evidence required to answer correctly.

Useful signals include:

  • recall of relevant passages;
  • precision of retrieved chunks;
  • ranking quality;
  • citation correctness;
  • answer faithfulness to retrieved evidence;
  • behavior when evidence is missing.

Evaluate fine-tuning on behavior

Compare the tuned model against a baseline on a held-out dataset.

Measure the thing you trained for:

  • classification accuracy;
  • formatting compliance;
  • task success;
  • style consistency;
  • policy adherence;
  • latency and token savings where relevant.

Evaluate tools on execution correctness

Check:

  • whether the correct tool was selected;
  • whether arguments were valid;
  • whether permissions were respected;
  • whether side effects matched intent;
  • whether retries caused duplicate actions;
  • whether the model handled tool failures correctly.

An application can have excellent retrieval and still be unsafe because its tool layer is poorly controlled.

Three failure modes to avoid

1. Putting everything into a vector database

Not every external fact belongs in RAG.

Highly structured or live data often belongs behind an API or database tool.

2. Fine-tuning before you have evals

If you cannot measure improvement, you cannot know whether training helped or merely changed the failure mode.

3. Giving tools too much authority

Tool access should be narrow, typed, auditable, and permissioned.

A model that needs to read an order should not automatically receive permission to refund it.

The design rule

Use the technique that matches the missing capability.

If the model needs more relevant knowledge, retrieve it.

If it needs more consistent behavior, optimize or fine-tune it.

If it needs live state or the ability to act, give it a controlled tool.

Then combine the techniques only where the application actually needs the combination.

This produces systems that are easier to evaluate because knowledge, behavior, and actions remain separate.

The best architecture is not the one with the most AI techniques.

It is the one where every technique has a clear responsibility and a measurable reason to exist.

Sources