Summarisation at scale: chunked map-reduce with LLMs
Why the naive approach breaks at scale
The tempting approach to summarising a 200-page document is to pack it into the largest context window available and ask for a summary. This works for documents that fit, but it fails in three ways as you scale:
Cost grows linearly with input length. A 150,000-token document sent to a frontier model for every summary request is expensive. For a knowledge base with 10,000 documents, even a one-time batch summarisation at scale requires careful cost control.
Quality degrades in very long contexts. Research consistently shows that models pay less attention to content in the middle of very long inputs. A single-call summary of a 200-page technical specification often drops important details from chapters 4 through 8.
Latency is unacceptable for interactive use. Processing 150K tokens at typical inference speeds takes 20–60 seconds. Users do not wait that long.
Map-reduce summarisation solves all three problems by dividing the document into chunks, summarising each chunk independently (the map phase), then synthesising those partial summaries into a final result (the reduce phase).
Chunking strategies
How you divide a document determines the quality of the final summary more than any other design decision. A chunk boundary that falls mid-sentence or mid-argument produces an incoherent partial summary that degrades the final output.
Fixed-size chunking with overlap
The simplest approach: split every N tokens with an M-token overlap between consecutive chunks. The overlap ensures that information near a boundary appears in both adjacent chunks.
function chunkByTokens(
text: string,
chunkSize: number = 2000,
overlapSize: number = 200
): string[] {
// Simplified: split by words as proxy for tokens.
// In production use a proper tokenizer like tiktoken or @anthropic-ai/tokenizer.
const words = text.split(/\s+/);
const chunks: string[] = [];
let start = 0;
while (start < words.length) {
const end = Math.min(start + chunkSize, words.length);
chunks.push(words.slice(start, end).join(" "));
if (end >= words.length) break;
start = end - overlapSize;
}
return chunks;
}
The limitation is that document structure is invisible to this algorithm. A paragraph boundary at word 1,999 and a section heading at word 2,001 will be split with the heading in a new chunk, which may cause the next chunk's summary to lack context.
Structure-aware chunking
For documents with clear structure — Markdown headings, PDF section markers, HTML <h2> tags — split on structural boundaries and trim or merge chunks that exceed the target size.
function chunkByHeadings(
markdown: string,
maxChunkTokens: number = 2000
): Array<{ heading: string; content: string }> {
const sections = markdown.split(/(?=^#{1,3} )/m).filter(s => s.trim());
const chunks: Array<{ heading: string; content: string }> = [];
for (const section of sections) {
const lines = section.split("\n");
const heading = lines[0].replace(/^#+\s*/, "");
const body = lines.slice(1).join("\n").trim();
const approxTokens = body.split(/\s+/).length; // rough estimate
if (approxTokens <= maxChunkTokens) {
chunks.push({ heading, content: body });
} else {
// Section too long — split it further by fixed size.
const subChunks = chunkByTokens(body, maxChunkTokens, 150);
subChunks.forEach((c, i) =>
chunks.push({ heading: `${heading} (part ${i + 1})`, content: c })
);
}
}
return chunks;
}
For PDFs, use a library that preserves paragraph boundaries (pdfplumber in Python, pdf-parse in Node). Feeding raw PDF text to the chunker produces poor results because PDF text extraction does not preserve reading order for multi-column layouts.
Semantic chunking
The most accurate (and most expensive) approach is to embed sentences and group them by embedding similarity. Sentences that are thematically related land in the same chunk regardless of their physical position in the document.
from anthropic import Anthropic
import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
def semantic_chunk(
sentences: list[str],
embeddings: np.ndarray,
similarity_threshold: float = 0.75,
max_chunk_sentences: int = 30
) -> list[list[str]]:
"""Group sentences into semantically coherent chunks."""
chunks = [[sentences[0]]]
current_embedding = embeddings[0]
for i in range(1, len(sentences)):
sim = cosine_similarity(
current_embedding.reshape(1, -1),
embeddings[i].reshape(1, -1)
)[0][0]
if sim >= similarity_threshold and len(chunks[-1]) < max_chunk_sentences:
chunks[-1].append(sentences[i])
# Update running centroid.
current_embedding = np.mean(
[current_embedding, embeddings[i]], axis=0
)
else:
chunks.append([sentences[i]])
current_embedding = embeddings[i]
return chunks
Semantic chunking is worth the embedding cost for high-value documents (legal contracts, research papers) where thematic coherence is critical. For commodity summarisation at scale, structure-aware or fixed-size chunking with overlap is usually sufficient.
The map phase: summarising individual chunks
Each chunk summary should be written in a way that will compose well at the reduce phase. Instruct the model to write summaries that stand alone, identify the section's main claim, and note any key entities or decisions that should appear in the final output.
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
interface ChunkSummary {
chunkIndex: number;
heading?: string;
summary: string;
keyPoints: string[];
inputTokens: number;
outputTokens: number;
}
async function summariseChunk(
chunk: { heading?: string; content: string },
index: number,
documentContext: string
): Promise<ChunkSummary> {
const system = `You are a precise technical summariser.
Document context: ${documentContext}
Summarise the following excerpt. Return ONLY valid JSON:
{
"summary": "2-4 sentence summary of the excerpt",
"keyPoints": ["key point 1", "key point 2"]
}
Focus on facts, decisions, and conclusions. Omit background filler.`;
const userContent = chunk.heading
? `## ${chunk.heading}\n\n${chunk.content}`
: chunk.content;
const response = await client.messages.create({
model: "claude-haiku-3-5", // use a fast, cheap model for the map phase
max_tokens: 512,
system,
messages: [{ role: "user", content: userContent }],
});
const text = (response.content[0] as { type: "text"; text: string }).text;
const parsed = JSON.parse(text);
return {
chunkIndex: index,
heading: chunk.heading,
summary: parsed.summary,
keyPoints: parsed.keyPoints,
inputTokens: response.usage.input_tokens,
outputTokens: response.usage.output_tokens,
};
}
Run all chunk summaries in parallel, with a concurrency limit to avoid hitting provider rate limits:
async function mapPhase(
chunks: Array<{ heading?: string; content: string }>,
documentContext: string,
concurrency: number = 20
): Promise<ChunkSummary[]> {
const sem = new Semaphore(concurrency);
const results = await Promise.all(
chunks.map((chunk, i) =>
sem.acquire().then(async () => {
try {
return await summariseChunk(chunk, i, documentContext);
} finally {
sem.release();
}
})
)
);
return results.sort((a, b) => a.chunkIndex - b.chunkIndex);
}
For a 200-page document split into 100 chunks, the parallel map phase completes in roughly 3–8 seconds with concurrency = 20, versus 5+ minutes if done serially.
The reduce phase: synthesising partial summaries
The reduce phase takes all chunk summaries and combines them into a coherent final summary. If the total length of chunk summaries exceeds the model's practical context window, apply reduce recursively — group summaries into batches and reduce each batch first, then reduce the batch results.
async function reducePhase(
summaries: ChunkSummary[],
documentContext: string,
targetWordCount: number = 500
): Promise<string> {
const summaryBlock = summaries
.map(s => {
const head = s.heading ? `### ${s.heading}\n` : "";
const points = s.keyPoints.length > 0
? `Key points:\n${s.keyPoints.map(p => `- ${p}`).join("\n")}\n`
: "";
return `${head}${points}${s.summary}`;
})
.join("\n\n");
const response = await client.messages.create({
model: "claude-sonnet-4-5", // use a stronger model for synthesis
max_tokens: Math.ceil(targetWordCount * 1.5),
system: `You are synthesising section summaries into a single coherent document summary.
Document context: ${documentContext}
Target length: approximately ${targetWordCount} words.
Produce a flowing prose summary that covers the document's main argument, key findings, and conclusions.
Do not use bullet points. Do not repeat the same information from multiple sections.`,
messages: [{ role: "user", content: summaryBlock }],
});
return (response.content[0] as { type: "text"; text: string }).text;
}
Recursive reduce for very large documents
When summarising a very large document produces chunk summaries that themselves exceed 50,000 tokens, apply reduce in tiers:
async function recursiveReduce(
summaries: ChunkSummary[],
documentContext: string,
batchSize: number = 20,
targetWordCount: number = 500
): Promise<string> {
if (summaries.length <= batchSize) {
return reducePhase(summaries, documentContext, targetWordCount);
}
// Group into batches and reduce each batch.
const batches: ChunkSummary[][] = [];
for (let i = 0; i < summaries.length; i += batchSize) {
batches.push(summaries.slice(i, i + batchSize));
}
const batchSummaries = await Promise.all(
batches.map(async (batch, i) => {
const batchText = await reducePhase(batch, documentContext, 300);
return {
chunkIndex: i,
summary: batchText,
keyPoints: [],
} as ChunkSummary;
})
);
// Final reduce over batch summaries.
return reducePhase(batchSummaries, documentContext, targetWordCount);
}
This produces a two-tier reduce for documents with 100–500 chunks and a three-tier reduce for books or document collections. Each tier uses cheaper models for intermediate summaries and reserves the strongest model for the final synthesis.
Caching intermediate results
The map phase is the most expensive part of the pipeline for large documents. Cache chunk summaries by a hash of the chunk content so unchanged sections are not re-summarised on subsequent runs.
import { createHash } from "crypto";
import { createClient } from "redis";
const redis = createClient({ url: process.env.REDIS_URL });
function chunkHash(content: string): string {
return createHash("sha256").update(content).digest("hex");
}
async function cachedSummariseChunk(
chunk: { heading?: string; content: string },
index: number,
documentContext: string
): Promise<ChunkSummary> {
const key = `chunk-summary:${chunkHash(chunk.content)}`;
const cached = await redis.get(key);
if (cached) return { ...JSON.parse(cached), chunkIndex: index };
const result = await summariseChunk(chunk, index, documentContext);
// Cache for 7 days — documents rarely change.
await redis.set(key, JSON.stringify(result), { EX: 60 * 60 * 24 * 7 });
return result;
}
For a knowledge base that re-summarises documents daily to pick up edits, this cache means only modified chunks pay for new API calls. A document with 100 chunks where 3 were edited pays for 3 map calls, not 100.
Prompt caching for the system prompt
If the system prompt contains a large, stable document (a style guide, company-specific terminology, or a domain glossary), mark it as cacheable with Anthropic's prompt caching API. This reduces the cost of the system prompt by 90% on cache hits.
const response = await client.messages.create({
model: "claude-haiku-3-5",
max_tokens: 512,
system: [
{
type: "text",
text: LARGE_STABLE_DOMAIN_GUIDE, // 5,000+ tokens
cache_control: { type: "ephemeral" } as any,
},
{
type: "text",
text: "Summarise the following excerpt. Return JSON: {...}",
},
] as any,
messages: [{ role: "user", content: chunk.content }],
});
The cache is valid for 5 minutes by default. For a batch summarisation job, all 100 parallel map calls issued within that window hit the cache, and you pay full price for only the first call.
Preserving document structure in the final summary
A purely extractive summary loses the reader's sense of document organisation. For structured documents, instruct the reduce phase to produce a structured output that mirrors the document's headings:
async function structuredReduce(
summaries: ChunkSummary[],
documentContext: string
): Promise<{ title: string; sections: Array<{ heading: string; content: string }>; conclusion: string }> {
const summaryBlock = summaries
.map(s => (s.heading ? `[${s.heading}] ${s.summary}` : s.summary))
.join("\n\n");
const response = await client.messages.create({
model: "claude-sonnet-4-5",
max_tokens: 2048,
system: `You are synthesising section summaries into a structured document summary.
Return ONLY valid JSON:
{
"title": "...",
"sections": [
{"heading": "...", "content": "2-3 sentences"}
],
"conclusion": "2-3 sentences capturing the document's main conclusion"
}
Document context: ${documentContext}`,
messages: [{ role: "user", content: summaryBlock }],
});
return JSON.parse((response.content[0] as { type: "text"; text: string }).text);
}
A structured summary is easier to render in a UI (as a collapsible outline) and makes it easier to generate downstream artefacts like slide decks or executive briefings.
Measuring summary quality
Quality measurement for summarisation is harder than for classification. Automatic metrics like ROUGE score capture n-gram overlap but miss factual accuracy. An LLM-as-judge approach using a quality rubric is more reliable for production monitoring:
async function scoreSummary(
originalExcerpt: string,
summary: string
): Promise<{ coverage: number; accuracy: number; conciseness: number; overall: number }> {
const response = await client.messages.create({
model: "claude-sonnet-4-5",
max_tokens: 256,
system: `Score this summary on four dimensions from 0.0 to 1.0.
Return ONLY valid JSON:
{
"coverage": <0.0-1.0 — how much of the key information is captured>,
"accuracy": <0.0-1.0 — factual correctness vs the original>,
"conciseness": <0.0-1.0 — ratio of information density to length>,
"overall": <0.0-1.0 — holistic quality>
}`,
messages: [{
role: "user",
content: `Original:\n${originalExcerpt}\n\nSummary:\n${summary}`,
}],
});
return JSON.parse((response.content[0] as { type: "text"; text: string }).text);
}
Run this scorer on a random 5% sample of daily summarisation output and alert when the average accuracy score drops below 0.85 — that usually signals a prompt regression or a change in the distribution of document types being processed.
Putting it all together
A complete pipeline for summarising an arbitrary document:
async function summariseDocument(
document: string,
options: {
chunkStrategy: "fixed" | "structured" | "semantic";
targetWords: number;
documentContext: string;
outputFormat: "prose" | "structured";
}
): Promise<string | object> {
// 1. Chunk
const chunks =
options.chunkStrategy === "structured"
? chunkByHeadings(document, 2000)
: chunkByTokens(document, 2000, 200).map(c => ({ content: c }));
// 2. Map
const summaries = await mapPhase(chunks, options.documentContext);
// 3. Reduce (recursively if needed)
if (options.outputFormat === "structured") {
return structuredReduce(summaries, options.documentContext);
}
return recursiveReduce(summaries, options.documentContext, 20, options.targetWords);
}
The map-reduce pattern is not a workaround for context length limits — it is a first-class architecture for document summarisation that produces better quality, lower cost, and better latency than shoving everything into a single long context call. The key is investing in chunk quality: boundaries that preserve semantic coherence, overlap that prevents edge effects, and caching that amortises cost across repeated runs.