Context window management: the engineering problem behind every failing AI feature

A product manager is demoing an AI writing assistant to a potential enterprise customer. The demo goes perfectly for the first 15 minutes. The assistant helps structure a document, suggests improvements, and maintains consistent tone. At the 20-minute mark, she asks the assistant to "update the introduction based on the feedback from earlier in the conversation." The assistant responds with an answer about the last paragraph, not the introduction. It has lost track of the conversation's beginning.

The context window is full. Earlier messages have been dropped. The assistant does not know what was discussed in the first 15 minutes. It is responding based on an incomplete picture of the conversation, but it does not say so. It gives a confident response that refers to the wrong part of the document.

This is not a bug in the assistant's reasoning. It is a predictable consequence of context window limits that were not explicitly managed. The feature worked in development — where no test conversation ran longer than 10 minutes — and broke in production when users had longer sessions.

Why context window management matters

Every LLM application passes text to the model. The model sees only what is in its context window at inference time. Everything outside the window — earlier messages, retrieved documents, instructions from previous steps — does not exist from the model's perspective.

Context windows are large by historical standards but finite. A 128,000-token context can hold substantial conversation history, but a 2-hour technical discussion with code examples, document drafts, and user feedback accumulates tokens faster than many developers expect.

The specific failure modes:

Truncation without acknowledgment. The application removes earlier messages to fit within the context limit. The model responds as if those messages never happened. Continuity breaks. The user's experience degrades without explanation.

Recent-token bias. Even within a context that fits within the limit, LLMs tend to attend more to recent tokens than earlier ones. Long contexts have performance implications beyond just fitting: the model may "forget" information from early in the context even when it is technically present.

Cost and latency. Processing more tokens costs more and takes longer. An application that appends every message to an ever-growing context will have escalating costs and increasing latency as conversations grow longer.

The three context management strategies

Sliding window

The simplest approach: keep the most recent N tokens of conversation history. Older messages are dropped when the total exceeds the limit.

from typing import List

def apply_sliding_window(
    messages: List[dict],
    max_tokens: int = 4000,
    always_include_system: bool = True
) -> List[dict]:
    """
    Applies a sliding window to message history, keeping the most recent messages.
    Always includes the system message.
    """
    token_count = 0
    included_messages = []

    # Count system message first (always included)
    system_messages = [m for m in messages if m["role"] == "system"]
    conversation_messages = [m for m in messages if m["role"] != "system"]

    for msg in system_messages:
        token_count += estimate_tokens(msg["content"])
        included_messages.append(msg)

    # Add conversation messages from most recent, going backward
    for msg in reversed(conversation_messages):
        msg_tokens = estimate_tokens(msg["content"])
        if token_count + msg_tokens > max_tokens:
            break
        token_count += msg_tokens
        included_messages.append(msg)

    # Restore chronological order for conversation messages
    system = [m for m in included_messages if m["role"] == "system"]
    conversation = list(reversed([m for m in included_messages if m["role"] != "system"]))
    return system + conversation

Sliding window is simple and predictable. Its failure mode is the writing assistant demo: the beginning of the conversation — which may contain critical context (the user's goals, the document structure, key decisions) — is dropped without acknowledgment.

Summarization compression

Instead of dropping old messages, periodically summarize them. The summary retains the important information from old messages in compressed form, freeing space for new messages.

async def compress_conversation_history(
    messages: List[dict],
    current_token_count: int,
    max_tokens: int,
    llm_client,
    compression_trigger: float = 0.8  # Compress when 80% of limit is reached
) -> List[dict]:
    """
    Compresses old conversation messages into a summary when approaching the context limit.
    """
    if current_token_count < max_tokens * compression_trigger:
        return messages  # No compression needed yet

    # Find messages to compress — oldest half of non-system messages
    system_messages = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]

    # Keep the most recent 25% fresh; compress the older 75%
    keep_count = max(4, len(conversation) // 4)
    to_compress = conversation[:-keep_count]
    to_keep = conversation[-keep_count:]

    if not to_compress:
        return messages

    # Summarize the older messages
    compression_prompt = f"""The following is the beginning of a conversation.
Summarize the key points, decisions, and important context from this conversation
in a concise paragraph. Focus on information that would be needed to continue
the conversation coherently.

Conversation to summarize:
{format_messages_for_summary(to_compress)}

Provide only the summary, no preamble."""

    summary_text = await llm_client.complete(compression_prompt, temperature=0, max_tokens=400)

    summary_message = {
        "role": "system",
        "content": f"[Earlier conversation summary: {summary_text}]"
    }

    return system_messages + [summary_message] + to_keep

Summarization preserves more semantic information than truncation. Its failure mode: the summary may lose important details that become relevant later. A user who mentioned a constraint early in the conversation — "I'm using Python 2.7" — may find that constraint lost in the summary and the assistant later recommending Python 3 features.

Retrieval-augmented memory

For long-horizon tasks, store earlier conversation turns in a vector database and retrieve relevant ones based on the current turn. The context contains recent messages plus retrieved messages from earlier in the conversation that are semantically relevant to the current query.

async def build_context_with_memory(
    current_message: str,
    recent_messages: List[dict],  # Last 10 messages
    conversation_id: str,
    memory_store,  # Vector database of conversation history
    max_tokens: int = 6000
) -> List[dict]:
    """
    Builds context using recent messages + retrieved relevant earlier messages.
    """
    # Retrieve semantically relevant earlier messages
    current_embedding = embedding_model.embed(current_message)
    relevant_history = memory_store.search(
        embedding=current_embedding,
        conversation_id=conversation_id,
        top_k=5,
        exclude_recent=True  # Don't retrieve messages already in recent_messages
    )

    # Assemble context: system + relevant history + recent messages
    context_messages = []
    token_budget = max_tokens

    # System message (always first)
    system_prompt = get_system_prompt()
    context_messages.append({"role": "system", "content": system_prompt})
    token_budget -= estimate_tokens(system_prompt)

    # Insert relevant earlier context with attribution
    if relevant_history:
        history_text = "\n\n".join([
            f"[Earlier in conversation, turn {h.turn_number}]:\n"
            f"User: {h.user_message}\nAssistant: {h.assistant_message}"
            for h in relevant_history
        ])
        history_message = {
            "role": "system",
            "content": f"Relevant context from earlier in this conversation:\n{history_text}"
        }
        context_messages.append(history_message)
        token_budget -= estimate_tokens(history_text)

    # Add recent messages within remaining budget
    for msg in reversed(recent_messages):
        msg_tokens = estimate_tokens(msg["content"])
        if token_budget - msg_tokens < 500:  # Reserve 500 tokens for response
            break
        context_messages.append(msg)
        token_budget -= msg_tokens

    # Restore chronological order
    return (
        [context_messages[0]] +  # System
        (context_messages[1:2] if len(context_messages) > 1 else []) +  # History
        list(reversed([m for m in context_messages[2:] if m["role"] != "system"]))
    )

Retrieval-augmented memory handles long conversations well but adds latency (vector search per turn) and requires indexing every conversation turn as it happens.

Telling the user when context is limited

The writing assistant demo failed not just because context was dropped, but because the assistant did not acknowledge that it had lost context. A model that responds confidently from incomplete context is less useful than one that acknowledges the limitation.

def build_context_with_truncation_notice(
    messages: List[dict],
    max_tokens: int
) -> tuple[List[dict], bool]:
    """
    Returns the truncated message list and whether truncation occurred.
    The caller can then add a notice to the system prompt.
    """
    total_tokens = sum(estimate_tokens(m["content"]) for m in messages)
    if total_tokens <= max_tokens:
        return messages, False

    truncated = apply_sliding_window(messages, max_tokens)
    return truncated, True

# In the calling code:
context, was_truncated = build_context_with_truncation_notice(messages, MAX_TOKENS)

if was_truncated:
    # Add a notice to the system prompt so the model knows
    for msg in context:
        if msg["role"] == "system":
            msg["content"] += (
                "\n\n[Note: Earlier parts of this conversation have been removed "
                "due to length. If the user refers to something from earlier in the "
                "conversation that you cannot see, acknowledge that you may have "
                "lost that context and ask them to remind you.]"
            )
            break

Acknowledging context loss to the user — or better, to the model itself via the system prompt — produces better behavior than a model that responds as if it has full context when it does not.

Token estimation

Accurate token estimation is required for all context management strategies. The easiest approach uses the tokenizer directly:

import tiktoken  # For OpenAI models

_encoder = tiktoken.get_encoding("cl100k_base")  # GPT-4 encoding

def estimate_tokens(text: str) -> int:
    return len(_encoder.encode(text))

def estimate_message_tokens(message: dict) -> int:
    # Each message has overhead for role and formatting
    return estimate_tokens(message["content"]) + 4  # ~4 tokens for role overhead

Using character-count heuristics (divide by 4 for a rough estimate) introduces errors that accumulate across a conversation. Use the actual tokenizer for accuracy.

Common mistakes in context window management

Estimating token count by character count. A common approximation is to divide character count by 4 to estimate token count. This is inaccurate for code (which tokenizes differently from prose), multilingual content (non-Latin scripts often use more tokens per character), and specialized vocabulary. Use the actual tokenizer for the model being used. The tiktoken library for OpenAI models and the equivalent for other providers provides exact token counts.

Including the full conversation system prompt in every message. A system prompt that grows as the feature evolves is never trimmed. System prompts that start at 500 tokens grow to 2,000 tokens as new instructions are added. At 2,000 tokens, the system prompt is consuming 15% of an 8K context before a single user message is included. Audit system prompt length at each release and remove instructions that are redundant, overly verbose, or that could be moved to the query context.

Using sliding window without any continuity signaling. Sliding window truncation drops earlier messages silently. The model responds as if it has full context. When the user references something from earlier that was dropped, the model either misunderstands or hallucinates a response. Adding a system prompt note when truncation occurs — "Note: earlier conversation context has been removed" — allows the model to acknowledge the gap when asked rather than confabulating from incomplete information.

Not considering the output token budget. Context window management focuses on input tokens, but the model's output also consumes context window capacity. For models with a fixed context window that includes both input and output, a 4K-token system prompt plus 2K-token conversation history leaves only 2K tokens for the model's response. For applications that expect long responses (document generation, code generation), the context budget must account for the expected output length.

Rebuilding the full context from database on every turn. For applications that store conversation history in a database, rebuilding the context by loading all messages on every turn is expensive and slow. Use a cached in-memory representation of the context that is updated incrementally as new messages arrive, rather than rebuilding from the full conversation history on each turn.

Testing context management behavior

Context management logic is among the most important application logic to test explicitly, because failures are invisible — the application does not error, it produces subtly wrong output.

Specific test cases for context management:

def test_sliding_window_preserves_recent_context():
    """Verify that after truncation, the most recent messages are retained."""
    messages = [create_message(role, content) for role, content
                in [("user", "Hello"), ("assistant", "Hi"),
                    ("user", "My name is Alice"), ("assistant", "Nice to meet you, Alice"),
                    ("user", "What is my name?")] * 100]  # 500 messages

    truncated = apply_sliding_window(messages, max_tokens=2000)

    # The most recent user message must be in the truncated context
    assert any(m["content"] == "What is my name?" for m in truncated)

def test_truncation_notice_added_when_messages_dropped():
    """Verify that truncation adds a notice to the system prompt."""
    messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        *[{"role": "user", "content": "A" * 1000} for _ in range(20)]  # Many long messages
    ]

    result, was_truncated = build_context_with_truncation_notice(messages, max_tokens=4000)

    assert was_truncated
    system_msg = next(m for m in result if m["role"] == "system")
    assert "truncated" in system_msg["content"].lower() or "removed" in system_msg["content"].lower()

def test_context_within_token_budget():
    """Verify that context management never exceeds the token limit."""
    for _ in range(10):
        message_count = random.randint(10, 200)
        messages = [create_random_message() for _ in range(message_count)]
        truncated = apply_sliding_window(messages, max_tokens=4000)
        total_tokens = sum(estimate_tokens(m["content"]) for m in truncated)
        assert total_tokens <= 4000

Running these tests on every change to context management code catches bugs before they reach users. Context management failures are subtle enough that manual testing is unlikely to catch them consistently.

The engineers who design context management before users hit the context limit will build applications that degrade gracefully. The ones who discover the context limit from user complaints after launch will retrofit context management into a codebase that was not designed for it — and users will have already learned that the application breaks in long sessions.

Profiling context usage in production

Understanding where tokens are spent in a production application requires instrumentation. Token counts for different context segments — system prompt, conversation history, retrieved documents, response — should be tracked and graphed:

from dataclasses import dataclass
from typing import Optional

@dataclass
class ContextProfile:
    session_id: str
    turn_number: int
    system_prompt_tokens: int
    history_tokens: int
    retrieved_context_tokens: int
    current_message_tokens: int
    total_input_tokens: int
    estimated_output_tokens: int
    strategy_used: str  # "full", "sliding_window", "summarized", "rag_memory"
    messages_dropped: int

def profile_context_usage(
    session_id: str,
    turn_number: int,
    context_messages: List[dict],
    strategy_used: str,
    messages_dropped: int
) -> ContextProfile:
    system_tokens = sum(
        estimate_message_tokens(m) for m in context_messages if m["role"] == "system"
    )
    history_tokens = sum(
        estimate_message_tokens(m) for m in context_messages if m["role"] != "system"
    )

    profile = ContextProfile(
        session_id=session_id,
        turn_number=turn_number,
        system_prompt_tokens=system_tokens,
        history_tokens=history_tokens,
        retrieved_context_tokens=0,  # Set by RAG layer if applicable
        current_message_tokens=estimate_message_tokens(context_messages[-1]),
        total_input_tokens=system_tokens + history_tokens,
        estimated_output_tokens=500,  # Default estimate
        strategy_used=strategy_used,
        messages_dropped=messages_dropped
    )

    # Emit to monitoring
    metrics.record("llm.context.system_tokens", profile.system_prompt_tokens)
    metrics.record("llm.context.history_tokens", profile.history_tokens)
    metrics.record("llm.context.total_tokens", profile.total_input_tokens)
    metrics.record("llm.context.strategy", 1, tags={"strategy": strategy_used})
    if messages_dropped > 0:
        metrics.record("llm.context.messages_dropped", messages_dropped)

    return profile

Graphing these metrics over time reveals patterns: the average session length at which context management activates, the distribution of system prompt sizes across the user base, the percentage of sessions that hit the context limit. These measurements drive informed decisions — if 30% of sessions are hitting the context limit at turn 15, the sliding window threshold may need to be adjusted, or the system prompt may need to be trimmed.

Adapting context strategy to conversation phase

Different phases of a conversation have different context requirements. A coding assistant's early turns establish the codebase context, the user's goals, and the technical constraints. The middle turns are where the active work happens. Later turns often refer back to decisions made in the early turns.

A context strategy that weights the first few turns and the most recent turns more heavily than the middle turns can preserve the important framing from early in the conversation while maintaining recent context:

def prioritized_context_selection(
    messages: List[dict],
    max_tokens: int,
    anchor_turns: int = 4  # Number of early turns to always preserve
) -> List[dict]:
    """
    Selects messages for the context window by prioritizing:
    1. System messages (always included)
    2. The first N turns of the conversation (anchor context)
    3. The most recent messages (recent context)

    Middle messages are dropped when the limit is hit.
    """
    system_messages = [m for m in messages if m["role"] == "system"]
    conversation = [m for m in messages if m["role"] != "system"]

    # Pair messages into turns (user + assistant)
    turns = []
    i = 0
    while i < len(conversation):
        if i + 1 < len(conversation):
            turns.append(conversation[i:i+2])
            i += 2
        else:
            turns.append(conversation[i:i+1])
            i += 1

    # Anchor turns — always include
    anchor = turns[:anchor_turns]
    # Recent turns — include as many as fit
    recent = turns[anchor_turns:]

    system_tokens = sum(estimate_message_tokens(m) for m in system_messages)
    anchor_tokens = sum(estimate_message_tokens(m) for turn in anchor for m in turn)
    remaining = max_tokens - system_tokens - anchor_tokens

    selected_recent = []
    for turn in reversed(recent):
        turn_tokens = sum(estimate_message_tokens(m) for m in turn)
        if remaining - turn_tokens < 200:  # Reserve 200 tokens buffer
            break
        selected_recent.extend(turn)
        remaining -= turn_tokens

    # Reconstruct in chronological order
    anchor_messages = [m for turn in anchor for m in turn]
    recent_messages = list(reversed(selected_recent))
    return system_messages + anchor_messages + recent_messages

This strategy is more effective for long-horizon tasks — writing assistants, coding assistants, research assistants — than a pure sliding window. The user's initial goals and constraints remain in context even when the conversation is long, ensuring the assistant can reference them without requiring the user to repeat themselves.