RAG pipelines that actually work in production

A machine learning engineer demos a RAG system that answers questions about company documentation with impressive accuracy. The retrieval is semantic, the answers are coherent, and the citations are correct. Her manager asks to ship it to users in four weeks. Three months later, the system is in production but user satisfaction is poor. Answers are inconsistent. Some queries return confidently wrong information. Others return "I don't know" for questions the documentation clearly answers. The demo worked. The production system does not.

The gap between the demo and production is almost never the language model. It is the retrieval. Specifically: how documents are chunked, how embeddings represent those chunks, how the retrieval matches query intent to chunk content, and how retrieved chunks are assembled into a context that the model can use accurately.

Why the demo works and the product fails

Demo conditions are optimized for showing the best case. The documentation is clean and well-structured. The test queries are close to the documentation's phrasing. The retrieval returns relevant chunks because the query and the chunk use similar vocabulary. The model answers correctly because the context contains exactly the information needed.

Production conditions are the average case. Users ask questions in natural language that may not match the documentation's phrasing. Documents have headers, tables, code blocks, and inline references that create unnatural chunk boundaries. Some questions require combining information from multiple sections of different documents. Some queries are ambiguous.

The RAG system that works in production is designed for the average case, not the best case.

Chunking: the step that determines everything downstream

Retrieval quality is bounded by chunk quality. A chunk that splits a concept across a boundary cannot be retrieved as a unit. A chunk that contains multiple unrelated concepts will match queries for either concept but will include irrelevant information in the context.

The naive chunking strategy — split every N tokens, optionally with overlap — works poorly on structured documents. A technical documentation page that starts with a conceptual overview, has a configuration reference table in the middle, and ends with code examples will produce chunks that span the concept-table and table-code boundaries, creating chunks that are semantically incoherent.

A better approach parses the document structure and chunks by semantic unit:

from dataclasses import dataclass
from typing import List, Optional
import re

@dataclass
class Chunk:
    text: str
    metadata: dict  # document_id, section, subsection, chunk_index

def chunk_markdown_document(
    document: str,
    doc_id: str,
    max_tokens: int = 512,
    overlap_tokens: int = 64
) -> List[Chunk]:
    """
    Chunks a markdown document by section structure rather than fixed token count.
    Keeps header context with each chunk for better retrieval relevance.
    """
    sections = split_by_headers(document)
    chunks = []

    for section in sections:
        header_context = section["header"]  # e.g., "# Config Reference > ## Authentication"

        # If section fits in max_tokens, keep it as one chunk
        if estimate_tokens(section["content"]) <= max_tokens:
            chunks.append(Chunk(
                text=f"{header_context}\n\n{section['content']}",
                metadata={
                    "doc_id": doc_id,
                    "section": section["level1"],
                    "subsection": section.get("level2"),
                    "chunk_type": "full_section"
                }
            ))
        else:
            # For large sections, split by paragraph with overlap
            paragraphs = split_paragraphs(section["content"])
            current_chunk_tokens = estimate_tokens(header_context)
            current_chunk_text = header_context + "\n\n"

            for para in paragraphs:
                para_tokens = estimate_tokens(para)

                if current_chunk_tokens + para_tokens > max_tokens:
                    chunks.append(Chunk(
                        text=current_chunk_text.strip(),
                        metadata={
                            "doc_id": doc_id,
                            "section": section["level1"],
                            "chunk_type": "partial_section"
                        }
                    ))
                    # Start new chunk with overlap: include header and last paragraph
                    current_chunk_text = header_context + "\n\n" + para + "\n\n"
                    current_chunk_tokens = estimate_tokens(current_chunk_text)
                else:
                    current_chunk_text += para + "\n\n"
                    current_chunk_tokens += para_tokens

            if current_chunk_text.strip():
                chunks.append(Chunk(
                    text=current_chunk_text.strip(),
                    metadata={"doc_id": doc_id, "section": section["level1"]}
                ))

    return chunks

Code blocks need special handling. A code block that is split in the middle is meaningless as a retrieval result. Keep code blocks intact as their own chunks, with the surrounding paragraph as context:

def extract_code_chunks(document: str, doc_id: str) -> List[Chunk]:
    """Extracts code blocks as individual chunks with surrounding context."""
    chunks = []
    pattern = r'(.*?```(\w*)\n(.*?)```)'

    for match in re.finditer(pattern, document, re.DOTALL):
        preceding_text = match.group(1).strip()
        language = match.group(2)
        code = match.group(3)

        # Include the paragraph before the code block for context
        context_paragraph = get_last_paragraph(preceding_text)

        chunks.append(Chunk(
            text=f"{context_paragraph}\n\n```{language}\n{code}\n```",
            metadata={
                "doc_id": doc_id,
                "chunk_type": "code_example",
                "language": language
            }
        ))

    return chunks

Retrieval quality: the hybrid approach

Pure semantic retrieval — embedding the query and retrieving the nearest chunks by cosine similarity — works well when the query and the relevant chunk use similar vocabulary. It fails when the query uses terminology different from the document.

A hybrid retrieval strategy combines semantic similarity (embedding-based) with keyword matching (BM25 or full-text search):

from typing import List, Tuple

def hybrid_retrieve(
    query: str,
    vector_store,
    keyword_index,
    top_k: int = 10,
    semantic_weight: float = 0.6,
    keyword_weight: float = 0.4
) -> List[Chunk]:
    """
    Combines semantic and keyword retrieval with reciprocal rank fusion.
    Returns ranked list of chunks.
    """
    # Semantic retrieval
    query_embedding = embedding_model.embed(query)
    semantic_results = vector_store.search(
        query_embedding,
        top_k=top_k * 2,  # Retrieve more candidates for fusion
        return_scores=True
    )  # List of (chunk, score)

    # Keyword retrieval (BM25 or PostgreSQL full-text search)
    keyword_results = keyword_index.search(
        query,
        top_k=top_k * 2,
        return_scores=True
    )

    # Reciprocal rank fusion
    chunk_scores = {}

    for rank, (chunk, _score) in enumerate(semantic_results):
        chunk_id = chunk.metadata["chunk_id"]
        chunk_scores[chunk_id] = chunk_scores.get(chunk_id, 0) + \
            semantic_weight * (1.0 / (rank + 1))

    for rank, (chunk, _score) in enumerate(keyword_results):
        chunk_id = chunk.metadata["chunk_id"]
        chunk_scores[chunk_id] = chunk_scores.get(chunk_id, 0) + \
            keyword_weight * (1.0 / (rank + 1))

    # Sort by combined score, return top-k unique chunks
    all_chunks = {c.metadata["chunk_id"]: c
                  for c, _ in semantic_results + keyword_results}
    ranked_chunk_ids = sorted(chunk_scores.keys(),
                              key=lambda cid: chunk_scores[cid],
                              reverse=True)

    return [all_chunks[cid] for cid in ranked_chunk_ids[:top_k]]

The hybrid approach handles the vocabulary mismatch problem. A query about "authentication timeout settings" retrieves chunks about "session expiry configuration" via semantic similarity, while also retrieving chunks that literally contain "authentication" and "timeout" via keyword matching.

Context assembly: the LLM sees what retrieval provides

The context provided to the language model determines the quality ceiling. Relevant chunks poorly assembled produce poor answers.

Specific problems in context assembly:

Duplicate information. Multiple retrieved chunks from the same document section contain overlapping information. The model's context is filled with repetition, reducing the effective context for other relevant information.

Missing document context. A chunk that references "the setting described above" or "as configured in step 1" is meaningless without the surrounding document context. The model does not know what "above" or "step 1" refers to.

No source attribution in context. If the model is expected to cite sources, the source information must be in the context, not just in the metadata.

A context assembly function that addresses these:

def assemble_context(
    chunks: List[Chunk],
    max_context_tokens: int = 3000,
    deduplicate: bool = True
) -> str:
    """
    Assembles retrieved chunks into a context string for the LLM.
    Deduplicates, adds source attribution, and stays within token budget.
    """
    seen_content = set()
    assembled_sections = []
    current_tokens = 0

    for chunk in chunks:
        # Deduplicate by content hash (handles near-duplicate chunks from same section)
        content_hash = hash(chunk.text[:200])  # Hash of first 200 chars
        if deduplicate and content_hash in seen_content:
            continue
        seen_content.add(content_hash)

        chunk_tokens = estimate_tokens(chunk.text)
        if current_tokens + chunk_tokens > max_context_tokens:
            break

        # Add source attribution inline
        doc_id = chunk.metadata.get("doc_id", "unknown")
        section = chunk.metadata.get("section", "")
        source_header = f"[Source: {doc_id} — {section}]" if section else f"[Source: {doc_id}]"

        assembled_sections.append(f"{source_header}\n{chunk.text}")
        current_tokens += chunk_tokens

    return "\n\n---\n\n".join(assembled_sections)

def build_rag_prompt(query: str, context: str) -> str:
    return f"""Answer the following question using only the information provided in the context below.
If the context does not contain enough information to answer the question, say so explicitly.
Do not use information from outside the provided context.

Context:
{context}

Question: {query}

Answer:"""

Query transformation

The query as typed by the user is not always the best query for retrieval. A question like "my API calls keep failing after a while" has no obvious overlap with documentation about "connection pool timeout configuration."

Query expansion and rewriting improve retrieval recall:

async def expand_query(user_query: str, llm_client) -> List[str]:
    """
    Generates multiple retrieval queries from the user's question.
    Returns the original query plus expanded variations.
    """
    prompt = f"""Given the following user question, generate 3 alternative search queries
that might retrieve relevant documentation. The alternatives should use different terminology
and phrasings than the original question.

Original question: {user_query}

Return only the 3 alternatives as a JSON array of strings, no explanation."""

    response = await llm_client.complete(prompt, max_tokens=200)
    try:
        alternatives = json.loads(response)
        return [user_query] + alternatives[:3]
    except json.JSONDecodeError:
        return [user_query]  # Fall back to original query if parsing fails

Running retrieval against all four queries and merging results (deduplicated, re-ranked by frequency of appearance) substantially improves recall for queries that use non-documentation vocabulary.

Evaluating retrieval quality in production

The demo had no retrieval quality evaluation. The production system needs it. The minimum viable evaluation:

• Retrieval precision: for a set of test queries with known correct answers, what percentage of retrieved chunks actually contribute to the correct answer?
• Retrieval recall: for queries with known correct answers in the documentation, does the retrieval actually retrieve the relevant chunk?
• Answer factual accuracy: for a test set of questions with ground truth answers, does the model's answer match the ground truth?

These metrics catch the specific failures that make RAG systems unreliable: wrong chunks retrieved, correct chunks retrieved but answer still wrong (model hallucination or context assembly failure), and correct answer available but chunks not retrieved (recall failure).

Running this evaluation monthly on a fixed test set detects regressions when the document corpus changes, when embedding models are updated, or when prompt templates are modified. The RAG system that ships to production and is never evaluated against a fixed test set accumulates quality regressions that only surface as user complaints.

Common mistakes in production RAG implementations

Using a single embedding model for all content types. General-purpose embedding models perform well for prose text but poorly for code, tables, and structured data. A RAG system that serves both prose documentation and code examples should use different retrieval strategies for each content type — code search benefits from lexical matching (exact function names, API names) more than from semantic similarity. Using a single embedding model for everything reduces recall for content types where that model is weak.

Not updating the index when documents change. The embedding index reflects the document corpus at indexing time. When documentation is updated, the old chunks remain indexed alongside the new ones. The retrieval returns stale chunks that answer the user's question incorrectly based on outdated information. Production RAG systems need a document change detection mechanism — a webhook from the CMS, a scheduled re-index job, or a hash-based staleness check — that triggers re-chunking and re-embedding when source documents change.

Retrieving too many chunks and exceeding the context window. A retrieval that returns 20 chunks at 500 tokens each produces 10,000 tokens of context. For models with 8K context windows, this consumes the entire context budget and leaves no room for the model's response. For models with larger context windows, filling the context with marginally relevant chunks dilutes the most relevant information. Calibrate the retrieval top-k to the context budget available after accounting for system prompt, conversation history, and response length.

Not filtering by metadata before embedding search. A user asking about version 3.x of a product should not receive chunks from the version 2.x documentation. Metadata filtering — restricting the embedding search to chunks matching the user's product version, language, or permissions — reduces the search space and prevents irrelevant results that pass the semantic similarity threshold simply because the topic is similar.

def retrieve_with_filters(
    query: str,
    vector_store,
    metadata_filters: dict,  # e.g., {"product_version": "3", "language": "en"}
    top_k: int = 5
) -> List[Chunk]:
    """
    Filters the vector store by metadata before running the similarity search.
    Prevents version mismatch and permission violations in retrieval.
    """
    query_embedding = embedding_model.embed(query)
    return vector_store.search(
        embedding=query_embedding,
        top_k=top_k,
        filter=metadata_filters  # Applied server-side before similarity ranking
    )

Treating retrieval failures as model failures. When the RAG system produces a wrong answer, the debugging process often starts with the model — adjusting temperature, changing the system prompt, trying a different model. In most cases, the problem is in retrieval: the wrong chunks were retrieved, or the relevant chunk was not in the index. Before adjusting model parameters, verify that the retrieval is returning the correct chunks by logging retrieved chunks alongside the query and the model's response. The diagnosis should start with retrieval, not with the model.

Measuring RAG quality before and after changes

Every change to a RAG system — new documents added, chunk size adjusted, embedding model updated, retrieval parameters changed — should be evaluated against a fixed set of test queries with known correct answers. The evaluation dimensions:

Retrieval recall at k: For test queries where the correct answer is in the document corpus, does the retrieval return the correct chunk within the top k results? Low recall means the retrieval is missing relevant content.

Context relevance: What fraction of retrieved chunks are relevant to the query? High recall but low context relevance means the model's context is diluted with irrelevant information.

Answer accuracy: For test queries with known ground truth answers, what fraction of the model's answers are factually correct? This depends on both retrieval quality and model quality.

Answer groundedness: Are the model's claims present in the retrieved chunks? High accuracy but low groundedness suggests the model is using its training knowledge rather than the retrieved content — which may be correct for this test set but will fail for domain-specific queries outside the model's training distribution.

Tracking these metrics as time series over system changes catches regressions before they affect users and confirms improvements when changes are beneficial.

The path forward for production RAG

RAG is not a static architecture. The approaches that perform well today — hybrid retrieval, structured chunking, context assembly with deduplication — continue to improve as tooling matures. Vector databases are adding native hybrid search that eliminates the need for external BM25 indexes. Embedding models trained on domain-specific corpora outperform general-purpose models for specialized content. Reranker models, applied as a post-retrieval step to re-score retrieved chunks by relevance, consistently improve precision without requiring changes to the retrieval architecture itself.

The most durable investment is in the evaluation infrastructure. A system that can measure retrieval recall, context relevance, and answer accuracy will correctly identify which improvements actually help — and which are expensive changes that produce no measurable benefit. The teams that build this measurement layer before optimizing are the ones whose RAG systems actually improve over time rather than just changing.