Fine-tuning vs RAG: when each approach makes sense

The Wrong Way to Frame the Decision

Most articles frame the fine-tuning vs RAG decision as a capability comparison: RAG is better for up-to-date information, fine-tuning is better for style control. That framing misses the most important dimension: what kind of failure are you trying to fix?

RAG fixes knowledge gaps. If the model doesn't know about your proprietary data, internal processes, or recent events that post-date its training cutoff, RAG gives it access to that information at inference time.

Fine-tuning fixes behaviour gaps. If the model knows the facts but responds in the wrong format, at the wrong reading level, following the wrong decision logic, or without domain-specific vocabulary, fine-tuning teaches it to behave differently.

Before choosing between them, answer this question precisely: when the model fails, is it failing because it doesn't know something, or because it behaves wrong despite knowing?

When RAG Wins

RAG should be your first choice when:

The knowledge changes frequently. Anything that updates more often than you can afford to fine-tune — product catalogues, documentation, support tickets, legal regulations, news — belongs in a retrieval store. Fine-tuning bakes knowledge into weights; retrieving it keeps it live.

You need source attribution. RAG grounds every answer in specific retrieved documents. You can show users exactly which source the answer came from, implement citation-level confidence scoring, and audit why the model said what it said. Fine-tuning provides none of this.

The knowledge corpus is large. A knowledge base with 100,000 documents cannot fit in a context window. RAG selects the relevant subset at query time. Fine-tuning can absorb facts but loses specificity — models hallucinate plausible-sounding but wrong details when knowledge is distributed across many training examples.

You want to update knowledge without retraining. Adding a new product to your RAG store takes seconds. Adding it to a fine-tuned model requires a new training run, evaluation, deployment, and monitoring cycle. For rapidly evolving knowledge bases, fine-tuning is operationally expensive.

A minimal RAG implementation with the Anthropic SDK illustrates the pattern:

import anthropic
from dataclasses import dataclass
from typing import List

@dataclass
class Document:
    content: str
    source: str
    score: float

def retrieve(query: str, top_k: int = 5) -> List[Document]:
    """Stub — replace with your actual vector search."""
    raise NotImplementedError

def rag_answer(query: str) -> str:
    docs = retrieve(query, top_k=5)

    context_block = "\n\n".join(
        f"[Source: {doc.source}]\n{doc.content}"
        for doc in docs
    )

    client = anthropic.Anthropic()
    message = client.messages.create(
        model="claude-opus-4-5",
        max_tokens=1024,
        system=(
            "You are a helpful assistant. Answer questions using only the "
            "provided context. If the context does not contain enough "
            "information to answer, say so explicitly."
        ),
        messages=[
            {
                "role": "user",
                "content": f"Context:\n{context_block}\n\nQuestion: {query}"
            }
        ]
    )
    return message.content[0].text

When Fine-Tuning Wins

Fine-tuning earns its place when:

You need consistent output format. If every response must follow a specific JSON schema, include mandatory sections in a fixed order, or match a company-specific document template, prompt engineering reaches its limits. A fine-tuned model internalises the format and produces it reliably even under adversarial prompts or edge cases.

You're optimising latency or cost through model downsizing. A fine-tuned gpt-4o-mini or Claude Haiku can match the task quality of a much larger model on narrow domains. If you process millions of requests per day in a specific vertical, the economics of running a smaller, fine-tuned model can justify months of engineering effort.

You want to teach linguistic behaviour, not facts. Medical note summarisation that follows SOAP format, code review comments in a team's specific style, customer support responses that match brand voice — these are behavioural patterns that fine-tuning handles better than any prompt.

You need to suppress unwanted behaviours reliably. Prompt-level instructions to avoid certain topics or language patterns are stochastic. Fine-tuning can more reliably encode "this model never produces output in form X" into its weights.

A fine-tuning dataset for format teaching looks like this:

{"messages": [{"role": "user", "content": "Summarise this medical note: [note text]"}, {"role": "assistant", "content": "**Subjective:** Patient reports...\n**Objective:** Vitals show...\n**Assessment:** Impression is...\n**Plan:** 1. ..."}]}
{"messages": [{"role": "user", "content": "Summarise this medical note: [note text 2]"}, {"role": "assistant", "content": "**Subjective:** ...\n**Objective:** ...\n**Assessment:** ...\n**Plan:** 1. ..."}]}

The minimum viable dataset for format fine-tuning is 50-100 high-quality examples per format variant. For behavioural style, 200-500 examples. For factual domain knowledge, the quantity required scales with the breadth and specificity of the domain — expect 1,000+ for meaningful improvement.

The Hidden Costs That Change the Calculus

Neither approach is free. The real costs are rarely the headline API or compute fees.

RAG hidden costs:

• Retrieval latency adds 100-500ms to every request, more if you do re-ranking
• Chunking strategy requires significant tuning — wrong chunk sizes destroy recall
• Embedding models must be kept consistent; changing the model requires re-embedding the entire corpus
• Infrastructure for a vector store, embedding pipeline, and index refresh adds operational burden
• Retrieval failures are silent: if the wrong document is retrieved, the model confidently answers from bad context

Fine-tuning hidden costs:

• Data collection and labelling is the biggest cost. 500 high-quality examples from domain experts takes weeks
• Each training run must be evaluated before deployment; bad fine-tunes can be worse than the base model
• Model version management: your application is now tied to a specific model checkpoint that may become deprecated
• Knowledge staleness: the fine-tuned model knows nothing about events after its training data
• Catastrophic forgetting: aggressive fine-tuning on a narrow domain degrades general capabilities

Combining Both: When the Answer is "Yes, and"

The most capable production systems use both. A common architecture:

1. Fine-tune the model on your domain vocabulary, output format, and reasoning style
2. RAG provides the specific factual context for each query

A customer support system might fine-tune on thousands of resolved tickets to teach the model how to structure diagnoses and escalation decisions, while RAG retrieves the specific product documentation, firmware notes, and account history relevant to each incoming ticket.

def hybrid_answer(query: str, user_context: dict) -> str:
    """
    Fine-tuned model (for behaviour) + RAG (for knowledge).
    The fine-tuned model is called via its specific model ID after deployment.
    """
    docs = retrieve(query, top_k=3)
    account_docs = retrieve_account_history(user_context["user_id"], top_k=2)
    all_docs = docs + account_docs

    context_block = "\n\n".join(
        f"[{doc.source}]\n{doc.content}" for doc in all_docs
    )

    client = anthropic.Anthropic()
    # This model ID would be your fine-tuned model endpoint
    message = client.messages.create(
        model="claude-haiku-4-5",  # or your fine-tuned model endpoint
        max_tokens=512,
        system=(
            "You are a customer support specialist for Acme Corp. "
            "Use the provided context to answer accurately. "
            "Always structure your response as: Diagnosis, Next Steps, Escalation (if needed)."
        ),
        messages=[
            {
                "role": "user",
                "content": f"Context:\n{context_block}\n\nCustomer issue: {query}"
            }
        ]
    )
    return message.content[0].text

Evaluation Frameworks for Both Approaches

An honest evaluation must measure what the approach is intended to fix.

For RAG, measure:

• Retrieval recall: what fraction of queries retrieve the document containing the answer?
• Answer faithfulness: does the model's answer contradict the retrieved context? (RAGAS faithfulness metric)
• Answer relevance: does the answer address the user's actual question?

For fine-tuning, measure:

• Format compliance rate: what percentage of outputs match the target format exactly?
• Style consistency: cosine similarity between model outputs and gold-standard examples in embedding space
• General capability preservation: run the fine-tuned model against a standard benchmark to detect catastrophic forgetting
• Human preference: for behavioural fine-tunes, A/B human evaluation is irreplaceable

def evaluate_format_compliance(responses: List[str], required_sections: List[str]) -> float:
    """
    Check what fraction of responses contain all required sections.
    """
    compliant = 0
    for response in responses:
        if all(section in response for section in required_sections):
            compliant += 1
    return compliant / len(responses)

# Example usage
required = ["**Subjective:**", "**Objective:**", "**Assessment:**", "**Plan:**"]
compliance_rate = evaluate_format_compliance(model_outputs, required)
print(f"Format compliance: {compliance_rate:.1%}")

A Decision Tree for Practical Use

Work through these questions in order:

1. Does the failure happen because the model lacks access to the right information? → RAG
2. Does the knowledge update more often than monthly? → RAG
3. Do you need source citations or auditability? → RAG
4. Does the model have the knowledge but produce the wrong format, style, or structure? → Fine-tuning
5. Do you need to reduce latency or cost by using a smaller model on a narrow task? → Fine-tuning
6. Do you need both knowledge access and consistent behaviour? → Both
7. Is neither the core problem, but the prompt just poorly written? → Fix the prompt first

That last item is not a joke. The most common reason teams reach for fine-tuning when they shouldn't is that they haven't invested in prompt engineering. A well-structured prompt with a clear system message, concrete examples in the few-shot block, and explicit output formatting instructions solves the majority of behavioural problems without any training cost. Exhaust prompt engineering before committing to the operational overhead of fine-tuning, and exhaust in-context retrieval patterns before building a full RAG pipeline.

The right choice is always the one that fixes the specific failure with the least operational complexity. That answer changes as your product matures, your team grows, and your traffic scales.