LLM context window management: chunking, compression, and summarisation
Why context management is a first-class engineering concern
Every LLM call has a finite context window. This is not merely a technical limitation to work around—it is a fundamental design constraint that shapes the entire architecture of an LLM application. Ignoring it leads to expensive, slow, and unreliable systems.
The problems that emerge from poor context management are not always obvious. Silent truncation is the most dangerous: if your prompt plus user message exceeds the context limit, many API clients will truncate silently or raise a cryptic error. The model never sees the truncated content and cannot tell you that information is missing. Your application appears to work while producing incomplete responses.
High context usage also compounds into a cost and latency problem. Input tokens are cheaper than output tokens but not free. A system that stuffs 80,000 tokens of context into every call when 8,000 would suffice is burning 10× the input token budget. At scale, this difference is significant. And because both encoding time and first-token latency scale with context length, bloated contexts make responses feel slow even when the model itself is fast.
This article covers four orthogonal strategies for keeping contexts lean and relevant: hierarchical chunking for ingestion, selective retrieval for injection, compression techniques for prompt reduction, and rolling summarisation for long conversation threads. Each addresses a different source of context bloat.
Understanding context budgets
Before optimising anything, establish a context budget: an explicit allocation of tokens across the different components of your prompt.
interface ContextBudget {
systemPrompt: number; // Usually fixed
conversationHistory: number; // Slides as conversation grows
retrievedContext: number; // RAG documents
userMessage: number; // Current turn input
outputReserve: number; // Space for the response
total: number; // Must equal model's context limit
}
const CLAUDE_SONNET_CONTEXT = 200_000; // tokens
const budget: ContextBudget = {
systemPrompt: 2_000,
conversationHistory: 20_000,
retrievedContext: 30_000,
userMessage: 4_000,
outputReserve: 8_000,
total: CLAUDE_SONNET_CONTEXT,
};
// Total allocated: 64,000 tokens
// Remaining headroom: 136,000 tokens for expansion
Allocating a budget makes implicit decisions explicit. It forces you to decide how much history matters relative to retrieved context, and it gives you clear thresholds to enforce in code. When any component exceeds its budget, you have a defined strategy rather than undefined behaviour.
Counting tokens accurately requires using the tokeniser, not character-based approximations. Character-based approximations (1 token ≈ 4 characters) are fine for rough estimates but too imprecise for budget enforcement at the margins.
import Anthropic from "@anthropic-ai/sdk";
const client = new Anthropic();
async function countTokens(messages: Anthropic.MessageParam[], systemPrompt: string): Promise<number> {
const response = await client.messages.countTokens({
model: "claude-sonnet-4-5",
system: systemPrompt,
messages,
});
return response.input_tokens;
}
Chunking strategies for ingestion
Chunking is the process of splitting source documents into segments that can be individually retrieved and injected into context. The chunk size is one of the highest-leverage parameters in a RAG system because it directly affects both retrieval precision and the token cost per retrieved result.
Fixed-size chunking
The simplest approach: split every N characters with M characters of overlap.
def fixed_size_chunks(
text: str,
chunk_size: int = 512,
overlap: int = 64,
) -> list[str]:
chunks = []
start = 0
text_len = len(text)
while start < text_len:
end = min(start + chunk_size, text_len)
chunks.append(text[start:end])
start += chunk_size - overlap
return chunks
Fixed-size chunking is fast and predictable but ignores document structure. A 512-character chunk may start mid-sentence and end mid-paragraph, fragmenting meaning. This degrades retrieval quality because the embedding of a broken fragment is less representative than the embedding of a complete thought.
Recursive character text splitting
Split on structural boundaries (paragraphs, then sentences, then words), falling back to character splitting only when necessary:
def recursive_split(
text: str,
max_chunk_size: int = 512,
overlap: int = 64,
separators: list[str] | None = None,
) -> list[str]:
if separators is None:
separators = ["\n\n", "\n", ". ", "! ", "? ", " ", ""]
# Try each separator in order
for sep in separators:
if sep not in text:
continue
parts = text.split(sep)
chunks = []
current = ""
for part in parts:
candidate = current + (sep if current else "") + part
if len(candidate) <= max_chunk_size:
current = candidate
else:
if current:
chunks.append(current)
# If part itself is too large, recurse with next separator
if len(part) > max_chunk_size:
sub_chunks = recursive_split(
part,
max_chunk_size,
overlap,
separators[separators.index(sep) + 1:],
)
chunks.extend(sub_chunks)
current = ""
else:
current = part
if current:
chunks.append(current)
# Add overlap between chunks
if len(chunks) <= 1:
return chunks
overlapped = [chunks[0]]
for i in range(1, len(chunks)):
prev_end = chunks[i - 1][-overlap:] if overlap else ""
overlapped.append(prev_end + chunks[i])
return overlapped
return [text] # text is shorter than max_chunk_size
Semantic chunking
Semantic chunking groups sentences that are semantically related, producing chunks with coherent topic boundaries rather than arbitrary size limits. The algorithm embeds each sentence, then splits when adjacent sentences have low cosine similarity (a semantic shift):
import numpy as np
def semantic_chunks(
sentences: list[str],
embeddings: np.ndarray,
threshold: float = 0.7,
min_chunk_sentences: int = 3,
max_chunk_sentences: int = 15,
) -> list[str]:
if len(sentences) != len(embeddings):
raise ValueError("sentences and embeddings must have the same length")
# Compute cosine similarities between adjacent sentences
norms = np.linalg.norm(embeddings, axis=1, keepdims=True)
normalised = embeddings / np.maximum(norms, 1e-8)
similarities = np.sum(normalised[:-1] * normalised[1:], axis=1)
# Find split points where similarity drops below threshold
split_points = [0]
current_chunk_len = 0
for i, sim in enumerate(similarities):
current_chunk_len += 1
is_semantic_break = sim < threshold
is_too_long = current_chunk_len >= max_chunk_sentences
is_too_short = current_chunk_len < min_chunk_sentences
if (is_semantic_break or is_too_long) and not is_too_short:
split_points.append(i + 1)
current_chunk_len = 0
split_points.append(len(sentences))
# Assemble chunks
chunks = []
for i in range(len(split_points) - 1):
start, end = split_points[i], split_points[i + 1]
chunk_text = " ".join(sentences[start:end])
chunks.append(chunk_text)
return chunks
Semantic chunking produces better retrieval quality than fixed-size or recursive splitting at the cost of requiring an embedding pass over the entire document at ingestion time. For static corpora where ingest happens once, this is a worthwhile trade.
Selective context injection: putting the right chunks in context
Fetching many chunks and injecting all of them is wasteful. The retrieval stage should return a ranked list; the injection stage should apply a budget and select the best fit.
interface ChunkSelection {
chunks: string[];
totalTokens: number;
droppedChunks: number;
}
async function selectContextChunks(
rankedChunks: Array<{ text: string; score: number }>,
budget: number,
minScore: number = 0.5,
): Promise<ChunkSelection> {
const selected: string[] = [];
let totalTokens = 0;
let dropped = 0;
for (const chunk of rankedChunks) {
if (chunk.score < minScore) {
dropped++;
continue; // Skip low-relevance chunks regardless of budget
}
// Approximate: 1 token ≈ 4 chars
const chunkTokens = Math.ceil(chunk.text.length / 4);
if (totalTokens + chunkTokens > budget) {
dropped++;
continue;
}
selected.push(chunk.text);
totalTokens += chunkTokens;
}
return { chunks: selected, totalTokens, droppedChunks: dropped };
}
The minScore threshold is as important as the budget. A chunk with a similarity score of 0.3 is probably not relevant to the query, even if there is budget space for it. Including irrelevant context increases the risk of the model being distracted by it.
Prompt compression
Prompt compression reduces the token count of content that must be in context by removing redundant, low-information content while preserving the semantics that the model needs.
Selective detail compression
System prompts tend to accumulate verbose explanations, examples, and edge case handling over time. Compress them by:
- Removing connector phrases ("Please note that...", "It is important to remember...")
- Shortening examples to the minimal form that still demonstrates the pattern
- Using bullet lists instead of prose for enumerated rules
function compressSystemPrompt(verbose: string): string {
return verbose
// Remove filler phrases
.replace(/\b(please note that|it is important to|keep in mind that|remember that)\b/gi, "")
// Compress multiple newlines
.replace(/\n{3,}/g, "\n\n")
// Remove trailing whitespace on each line
.split("\n")
.map((line) => line.trimEnd())
.join("\n")
.trim();
}
LLMLingua-style selective token dropping
More aggressive compression uses a smaller language model to identify which tokens are least important for preserving the prompt's meaning, then drops them. Libraries like LLMLingua implement this automatically:
from llmlingua import PromptCompressor
compressor = PromptCompressor(
model_name="microsoft/llmlingua-2-bert-base-multilingual-cased-meetingbank",
use_llmlingua2=True,
device_map="cpu",
)
def compress_retrieved_context(
context: str,
target_ratio: float = 0.5, # keep 50% of tokens
) -> str:
result = compressor.compress_prompt(
context,
rate=target_ratio,
force_tokens=["[", "]", "Source"], # preserve citation markers
)
return result["compressed_prompt"]
LLMLingua compression at a 0.5 ratio typically preserves 90%+ of answer recall for factual QA while halving the context tokens. The compressed text is not human-readable—it looks like garbled prose—but the LLM processes it correctly because the model being compressed for and the model doing generation are both trained on token distributions where such compressed representations are parseable.
Rolling summarisation for long conversations
Multi-turn conversations accumulate history that eventually overflows the conversation history budget. Rolling summarisation compresses old turns into a summary while keeping recent turns verbatim.
interface ConversationManager {
systemPrompt: string;
recentMessages: Anthropic.MessageParam[];
summary: string | null;
summaryTurnCount: number;
}
async function addTurnAndCompress(
manager: ConversationManager,
userMessage: string,
assistantResponse: string,
historyBudget: number,
summarisationThreshold: number = 0.8, // summarise when at 80% of budget
): Promise<ConversationManager> {
const newMessages: Anthropic.MessageParam[] = [
...manager.recentMessages,
{ role: "user", content: userMessage },
{ role: "assistant", content: assistantResponse },
];
// Count current token usage
const currentTokens = await countTokens(newMessages, manager.systemPrompt);
if (currentTokens < historyBudget * summarisationThreshold) {
// Within budget — no compression needed
return { ...manager, recentMessages: newMessages };
}
// Summarise the oldest half of the conversation
const midpoint = Math.floor(newMessages.length / 2);
const toSummarise = newMessages.slice(0, midpoint);
const toKeep = newMessages.slice(midpoint);
const summaryText = await summariseMessages(toSummarise, manager.summary);
return {
...manager,
recentMessages: toKeep,
summary: summaryText,
summaryTurnCount: manager.summaryTurnCount + midpoint / 2,
};
}
async function summariseMessages(
messages: Anthropic.MessageParam[],
existingSummary: string | null,
): Promise<string> {
const client = new Anthropic();
const conversationText = messages
.map((m) => `${m.role.toUpperCase()}: ${m.content}`)
.join("\n\n");
const prompt = existingSummary
? `Previous summary:\n${existingSummary}\n\nNew conversation to incorporate:\n${conversationText}\n\nWrite an updated summary that incorporates the new exchanges. Preserve key decisions, facts established, and user preferences. Be concise.`
: `Summarise this conversation, preserving key decisions, established facts, user preferences, and important context. Be concise.\n\n${conversationText}`;
const response = await client.messages.create({
model: "claude-haiku-4-5",
max_tokens: 1024,
messages: [{ role: "user", content: prompt }],
});
return response.content[0].type === "text" ? response.content[0].text : "";
}
When injecting context into subsequent calls, prepend the summary as a system note:
function buildMessagesWithHistory(
manager: ConversationManager,
newUserMessage: string,
): { system: string; messages: Anthropic.MessageParam[] } {
const summaryNote = manager.summary
? `\n\n## Conversation summary (earlier turns)\n${manager.summary}`
: "";
return {
system: manager.systemPrompt + summaryNote,
messages: [
...manager.recentMessages,
{ role: "user", content: newUserMessage },
],
};
}
Using claude-haiku-4-5 for summarisation is intentional—it is fast and cheap for this task, letting you summarise frequently without significant cost impact.
Hierarchical context: document-level summaries plus chunk retrieval
For very large document corpora, two-level retrieval reduces both retrieval cost and context bloat. First, retrieve at the document level using document summaries. Then, retrieve at the chunk level only for the top-matching documents.
import anthropic
client = anthropic.Anthropic()
def generate_document_summary(document_text: str, max_words: int = 100) -> str:
"""Generate a short summary for document-level retrieval."""
response = client.messages.create(
model="claude-haiku-4-5",
max_tokens=200,
messages=[{
"role": "user",
"content": f"Summarise this document in under {max_words} words, focusing on the main topics and key facts:\n\n{document_text[:10000]}"
}],
)
return response.content[0].text
class HierarchicalRetriever:
def __init__(self, documents: list[dict]):
"""
documents: list of {id, title, full_text, chunks: [{text, embedding}]}
"""
self.documents = documents
# Generate and embed document-level summaries
self.doc_summaries = {}
self.doc_summary_embeddings = {}
for doc in documents:
summary = generate_document_summary(doc["full_text"])
self.doc_summaries[doc["id"]] = summary
# Embed the summary (not shown: call to embedding API)
self.doc_summary_embeddings[doc["id"]] = embed_text(summary)
def retrieve(
self,
query: str,
top_docs: int = 3,
chunks_per_doc: int = 3,
) -> list[dict]:
query_embedding = embed_text(query)
# Stage 1: Rank documents by summary similarity
doc_scores = []
for doc in self.documents:
score = cosine_similarity(
query_embedding,
self.doc_summary_embeddings[doc["id"]],
)
doc_scores.append((doc, score))
top_docs_ranked = sorted(doc_scores, key=lambda x: x[1], reverse=True)[:top_docs]
# Stage 2: Retrieve best chunks from top documents
results = []
for doc, doc_score in top_docs_ranked:
chunk_scores = [
(chunk, cosine_similarity(query_embedding, chunk["embedding"]))
for chunk in doc["chunks"]
]
top_chunks = sorted(chunk_scores, key=lambda x: x[1], reverse=True)[:chunks_per_doc]
for chunk, chunk_score in top_chunks:
results.append({
"text": chunk["text"],
"doc_id": doc["id"],
"doc_score": doc_score,
"chunk_score": chunk_score,
# Combined score: document-level context + chunk-level relevance
"final_score": 0.3 * doc_score + 0.7 * chunk_score,
})
return sorted(results, key=lambda x: x["final_score"], reverse=True)
The hierarchical approach pays off when the corpus has more than ~500 documents. Below that, full corpus chunk retrieval is simpler and fast enough.
Monitoring context usage in production
Instrument every LLM call to capture token usage and alert on budget violations:
interface LLMCallMetrics {
inputTokens: number;
outputTokens: number;
cacheReadTokens: number;
cacheWriteTokens: number;
budgetUtilisation: number; // inputTokens / contextBudget
droppedChunks: number;
compressionApplied: boolean;
summarisationApplied: boolean;
}
function recordMetrics(
response: Anthropic.Message,
budget: number,
chunkSelection: ChunkSelection,
compressionApplied: boolean,
summarisationApplied: boolean,
): LLMCallMetrics {
const usage = response.usage;
const metrics: LLMCallMetrics = {
inputTokens: usage.input_tokens,
outputTokens: usage.output_tokens,
cacheReadTokens: (usage as any).cache_read_input_tokens ?? 0,
cacheWriteTokens: (usage as any).cache_creation_input_tokens ?? 0,
budgetUtilisation: usage.input_tokens / budget,
droppedChunks: chunkSelection.droppedChunks,
compressionApplied,
summarisationApplied,
};
// Emit to your observability stack
console.log(JSON.stringify({ event: "llm_call", ...metrics }));
if (metrics.budgetUtilisation > 0.9) {
console.warn("Context budget at 90% utilisation — review compression strategy");
}
return metrics;
}
Track budgetUtilisation as a percentile metric over time. If P95 crosses 0.8, your compression strategy needs work before you hit the ceiling and start seeing silent truncation. Track droppedChunks to understand how often relevant context is being left out due to budget constraints—a consistently high drop count indicates your retrieval is returning more chunks than your budget can accommodate, and you need to either increase the budget or improve retrieval precision to fetch fewer, higher-quality chunks.
Context window management is infrastructure-level work. It does not show up in a single prompt's output quality—it shows up in the long-tail reliability of a system across diverse query types, document sizes, and conversation lengths. Getting it right early prevents a whole class of hard-to-diagnose production issues.