Semantic chunking strategies for RAG that actually improve recall

The chunking problem no one talks about enough

Retrieval-augmented generation (RAG) systems have a dirty secret: the embedding model and the vector database are rarely the bottleneck. The bottleneck is chunking — how you split documents before you index them.

Bad chunking surfaces in two ways. The retriever returns chunks that contain the right words but lack enough surrounding context to answer the question correctly. Or it returns chunks where the relevant sentence is buried between irrelevant content, and the LLM either misses it or is distracted by the noise.

Most tutorials use fixed-size character splitting — split every 1000 characters with 200 characters of overlap — and call it done. That works for toy demos. In production, with real documents that have structure, headings, tables, and cross-references, fixed-size splitting is one of the main reasons RAG recall is worse than it should be.

This article covers five chunking strategies, when each applies, and production-grade Python implementations of each.

Strategy 1: Fixed-size chunking and why it fails

Fixed-size chunking is the baseline. It splits text every N tokens or characters with M tokens of overlap.

from langchain.text_splitter import RecursiveCharacterTextSplitter

def fixed_size_split(text: str, chunk_size: int = 1000, overlap: int = 200) -> list[str]:
    splitter = RecursiveCharacterTextSplitter(
        chunk_size=chunk_size,
        chunk_overlap=overlap,
        length_function=len,
    )
    return splitter.split_text(text)

The failure mode is predictable. A sentence like "The maximum recommended dose is 500mg twice daily" gets split into "The maximum recommended dose is" and "500mg twice daily" if the chunk boundary falls in the middle. Neither chunk is retrievable for a query like "what is the recommended dosage."

Overlap helps but doesn't solve it — you're just duplicating content, which inflates index size and wastes embedding budget without guaranteeing the relevant sentence lands intact in any single chunk.

Fixed-size is appropriate when: documents are short and homogeneous (individual records, forum posts), when you want baseline simplicity, or when the downstream LLM context window is large enough that imperfect chunks are acceptable.

Strategy 2: Sentence-boundary chunking

Sentence-boundary chunking respects natural language structure by splitting only at sentence boundaries and grouping sentences into chunks that stay under a token limit.

import spacy
from typing import Iterator

nlp = spacy.load("en_core_web_sm")

def sentence_chunks(
    text: str,
    max_tokens: int = 512,
    overlap_sentences: int = 1,
) -> list[str]:
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents if sent.text.strip()]

    chunks: list[str] = []
    current: list[str] = []
    current_tokens = 0

    for i, sentence in enumerate(sentences):
        sentence_tokens = len(sentence.split())  # rough token estimate

        if current_tokens + sentence_tokens > max_tokens and current:
            chunks.append(" ".join(current))
            # Keep last N sentences as overlap
            current = current[-overlap_sentences:]
            current_tokens = sum(len(s.split()) for s in current)

        current.append(sentence)
        current_tokens += sentence_tokens

    if current:
        chunks.append(" ".join(current))

    return chunks

This eliminates mid-sentence splits, which immediately improves recall for fact-based queries. The remaining problem is that sentence boundaries don't respect semantic topic shifts. A paragraph about one topic followed immediately by an unrelated paragraph will land in the same chunk if they're short enough.

Sentence-boundary chunking is appropriate when: documents are primarily prose, queries target specific facts stated in single sentences or short paragraphs, and document structure is flat (no clear section headings).

Strategy 3: Structural/recursive chunking

Many documents have inherent structure — headings, sections, code blocks, lists. Recursive chunking exploits this by first splitting at the highest-level structural boundaries (headings), then recursively splitting any sections that are still too large.

import re
from dataclasses import dataclass

@dataclass
class DocumentChunk:
    content: str
    heading: str
    level: int
    metadata: dict

def extract_sections(markdown: str) -> list[tuple[str, str, int]]:
    """Return list of (heading, content, level) tuples from markdown."""
    pattern = re.compile(r"^(#{1,6})\s+(.+)$", re.MULTILINE)
    matches = list(pattern.finditer(markdown))

    sections = []
    for idx, match in enumerate(matches):
        level = len(match.group(1))
        heading = match.group(2).strip()
        start = match.end()
        end = matches[idx + 1].start() if idx + 1 < len(matches) else len(markdown)
        content = markdown[start:end].strip()
        sections.append((heading, content, level))

    return sections

def recursive_chunk(
    markdown: str,
    max_tokens: int = 512,
    source_url: str = "",
) -> list[DocumentChunk]:
    sections = extract_sections(markdown)
    chunks: list[DocumentChunk] = []

    for heading, content, level in sections:
        if not content:
            continue

        token_count = len(content.split())
        if token_count <= max_tokens:
            chunks.append(
                DocumentChunk(
                    content=f"{heading}\n\n{content}",
                    heading=heading,
                    level=level,
                    metadata={"source": source_url, "section": heading},
                )
            )
        else:
            # Section too large — split by sentence boundary within the section
            sub_chunks = sentence_chunks(content, max_tokens=max_tokens)
            for i, sub in enumerate(sub_chunks):
                chunks.append(
                    DocumentChunk(
                        content=f"{heading} (part {i + 1})\n\n{sub}",
                        heading=heading,
                        level=level,
                        metadata={
                            "source": source_url,
                            "section": heading,
                            "part": i + 1,
                        },
                    )
                )

    return chunks

The key insight is prepending the section heading to every chunk. When you embed the chunk, the heading provides topic context that improves similarity scoring for queries about that topic. Without the heading, two chunks from different sections of a long document may embed almost identically if they use similar vocabulary.

Recursive chunking is appropriate when: documents have Markdown or HTML structure, content spans multiple distinct topics, or you need to surface provenance ("this came from section X") in citations.

Strategy 4: Semantic similarity chunking

Semantic chunking takes a more aggressive approach: embed every sentence, then find boundaries where semantic similarity between adjacent sentences drops sharply. Group sentences until you hit a topic shift.

import numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b) + 1e-10))

def semantic_chunk(
    text: str,
    breakpoint_percentile: float = 85,
    min_chunk_sentences: int = 3,
    max_chunk_sentences: int = 20,
) -> list[str]:
    doc = nlp(text)
    sentences = [s.text.strip() for s in doc.sents if s.text.strip()]

    if len(sentences) < 2:
        return sentences

    # Embed all sentences in a single batch call
    embeddings = model.encode(sentences, batch_size=32, show_progress_bar=False)

    # Compute similarity between each adjacent pair
    similarities = [
        cosine_similarity(embeddings[i], embeddings[i + 1])
        for i in range(len(embeddings) - 1)
    ]

    # Find breakpoints where similarity drops below the Nth percentile
    threshold = float(np.percentile(similarities, 100 - breakpoint_percentile))
    breakpoints = {
        i + 1
        for i, sim in enumerate(similarities)
        if sim < threshold
    }

    chunks: list[str] = []
    start = 0

    for bp in sorted(breakpoints):
        chunk_sentences = sentences[start:bp]
        if len(chunk_sentences) >= min_chunk_sentences:
            chunks.append(" ".join(chunk_sentences))
            start = bp
        elif len(chunk_sentences) >= max_chunk_sentences:
            # Force a split even if semantic similarity is high
            chunks.append(" ".join(chunk_sentences))
            start = bp

    # Append the final chunk
    remaining = sentences[start:]
    if remaining:
        if chunks and len(remaining) < min_chunk_sentences:
            # Merge tiny tail into the previous chunk
            chunks[-1] += " " + " ".join(remaining)
        else:
            chunks.append(" ".join(remaining))

    return chunks

The breakpoint_percentile parameter controls granularity. At percentile 85, only the 15% lowest-similarity adjacent pairs become boundaries, producing large chunks. At percentile 95, the 5% lowest pairs become boundaries, producing fine-grained chunks. Tune this against your retrieval evaluation set.

Semantic chunking is expensive: it requires embedding every sentence, not just the final chunks. For a 50-page document with 1500 sentences, that's 1500 embedding API calls (or one large batch). The cost is paid at index time, not query time, so it's often acceptable.

Semantic chunking is appropriate when: documents lack structural markup, topics shift mid-paragraph, or you have heterogeneous documents where structural chunking would produce inconsistent results.

Strategy 5: Late chunking (embed-then-chunk)

Late chunking is a newer approach that addresses a fundamental limitation of all the strategies above: they chunk first, then embed, which means the embedding for each chunk is computed without the broader document context.

With late chunking, you embed the entire document (or a long passage) first using a long-context embedding model, get token-level embeddings, then pool those embeddings into chunk-level representations based on chunk boundaries you determine independently.

from transformers import AutoTokenizer, AutoModel
import torch

class LateChunker:
    def __init__(self, model_name: str = "jinaai/jina-embeddings-v2-base-en"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModel.from_pretrained(model_name)
        self.model.eval()

    def encode_with_context(
        self,
        text: str,
        chunk_boundaries: list[tuple[int, int]],  # character offsets
    ) -> list[np.ndarray]:
        """
        Embed full text, then pool token embeddings into chunk embeddings
        using character-to-token alignment.
        """
        encoding = self.tokenizer(
            text,
            return_tensors="pt",
            return_offsets_mapping=True,
            truncation=True,
            max_length=8192,
        )
        offset_mapping = encoding.pop("offset_mapping")[0]  # (num_tokens, 2)

        with torch.no_grad():
            outputs = self.model(**encoding)
        token_embeddings = outputs.last_hidden_state[0]  # (num_tokens, hidden_dim)

        chunk_embeddings: list[np.ndarray] = []
        for char_start, char_end in chunk_boundaries:
            # Find tokens that overlap with this character range
            mask = (
                (offset_mapping[:, 0] >= char_start) &
                (offset_mapping[:, 1] <= char_end)
            )
            if not mask.any():
                continue
            # Mean-pool the token embeddings for this chunk
            chunk_embed = token_embeddings[mask].mean(dim=0).numpy()
            chunk_embeddings.append(chunk_embed)

        return chunk_embeddings

    def chunk_and_embed(
        self, text: str, chunk_size_chars: int = 1000, overlap_chars: int = 200
    ) -> list[tuple[str, np.ndarray]]:
        # Determine chunk boundaries first
        boundaries: list[tuple[int, int]] = []
        start = 0
        while start < len(text):
            end = min(start + chunk_size_chars, len(text))
            boundaries.append((start, end))
            start += chunk_size_chars - overlap_chars

        # Get context-aware embeddings for each chunk
        embeddings = self.encode_with_context(text, boundaries)

        result = []
        for (start, end), embedding in zip(boundaries, embeddings):
            result.append((text[start:end], embedding))

        return result

The advantage of late chunking is that each chunk's embedding carries context from the rest of the document. A chunk that says "it was approved by the FDA in 2023" will embed with knowledge of what "it" refers to if the full document context is available during tokenisation. This significantly improves recall for pronoun-heavy or reference-heavy text.

The limitation is that late chunking requires a long-context embedding model (Jina v2, OpenAI text-embedding-3-large, or similar) and is slower than standard chunking.

Evaluating chunking quality

No chunking strategy is unconditionally better — the best choice depends on your documents and queries. Build a retrieval evaluation set to compare strategies:

from dataclasses import dataclass

@dataclass
class EvalSample:
    question: str
    document_id: str
    expected_chunk_contains: str  # substring that should appear in the top-k retrieved chunks

def evaluate_retrieval(
    chunks: list[str],
    chunk_embeddings: list[np.ndarray],
    eval_samples: list[EvalSample],
    query_model: SentenceTransformer,
    top_k: int = 5,
) -> dict[str, float]:
    results = {"recall@k": 0.0, "mrr": 0.0}
    total = len(eval_samples)

    for sample in eval_samples:
        query_embedding = query_model.encode(sample.question)
        scores = [
            cosine_similarity(query_embedding, emb)
            for emb in chunk_embeddings
        ]
        ranked = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)

        # Recall@k: was the relevant chunk in the top k?
        hit = any(
            sample.expected_chunk_contains in chunks[i]
            for i in ranked[:top_k]
        )
        results["recall@k"] += int(hit) / total

        # MRR: reciprocal rank of first relevant chunk
        for rank, idx in enumerate(ranked[:top_k], start=1):
            if sample.expected_chunk_contains in chunks[idx]:
                results["mrr"] += (1 / rank) / total
                break

    return results

Run this evaluation across strategies with the same document set and query set. In practice, semantic chunking and late chunking consistently outperform fixed-size chunking on recall@5 by 8–20 percentage points on technical documentation, while structural chunking performs best on well-organised reference material like API docs or legal contracts.

Choosing a strategy for your use case

A decision tree based on document type:

• API documentation, legal contracts, structured reports → recursive/structural chunking
• Academic papers, long-form prose with clear topic shifts → semantic chunking
• Short-form content (FAQs, support tickets, product descriptions) → sentence-boundary chunking
• Any content where pronoun resolution and cross-sentence references are critical → late chunking
• Prototype or very high document volume where speed matters most → fixed-size with large overlap

For most production RAG systems, a hybrid approach works best: use structural chunking where structure exists, fall back to semantic chunking for unstructured sections, and apply late chunking for the highest-value document corpus where recall precision is critical.

The investment in better chunking pays off in ways that are hard to recover through downstream tuning. A well-chunked index makes the reranker's job easier, reduces hallucination from irrelevant context, and makes the system's retrieval failures easier to diagnose and fix.