← Articles

Multi-turn conversation memory patterns for LLM applications

By Kwesi Amponsah · 16 July 20260 views
#llm#memory#conversations#databases#architecture

The memory problem in LLM applications

Every language model has a context window — a hard ceiling on how many tokens of text it can consider at once. Early conversational applications handled this by feeding the entire chat history into every prompt, which works until the conversation grows long enough to hit the limit or become expensive to run. At that point, the naive approach either crashes with a context-length error or starts silently truncating the beginning of the conversation, making the assistant appear to forget things it was told directly.

The deeper problem is architectural. Even models with large context windows (some now support over a million tokens) should not receive the raw conversation transcript for every inference call. Token costs scale linearly with context length. A 100,000-token context on every query in a user session that runs dozens of queries per day quickly exceeds what most SaaS businesses can profitably absorb.

Production LLM applications need explicit memory architecture: strategies for deciding what to keep, what to summarise, what to retrieve, and how to assemble an effective context window for each new query. This article presents four patterns — sliding window, periodic summarisation, entity extraction, and retrieval-augmented memory — with TypeScript implementations and the database schemas each requires.

Pattern 1: Sliding window memory

The simplest durable memory pattern is a circular buffer of recent messages. Keep the last N turns (where N is calibrated to fit within your target context size) and discard older turns. This is the right starting point for most chatbots because it requires no LLM calls to maintain memory, adds no latency, and is trivially correct.

import { Pool } from "pg";
import Anthropic from "@anthropic-ai/sdk";

const db = new Pool({ connectionString: process.env.DATABASE_URL });
const anthropic = new Anthropic();

interface ConversationMessage {
  role: "user" | "assistant";
  content: string;
  createdAt: Date;
  tokenEstimate: number;
}

async function getRecentMessages(
  sessionId: string,
  maxTokens = 8000
): Promise<ConversationMessage[]> {
  const { rows } = await db.query<ConversationMessage>(
    `SELECT role, content, created_at AS "createdAt", token_estimate AS "tokenEstimate"
     FROM messages
     WHERE session_id = $1
     ORDER BY created_at DESC
     LIMIT 100`,
    [sessionId]
  );

  // Accumulate messages from newest to oldest until budget is exhausted
  const window: ConversationMessage[] = [];
  let tokenCount = 0;

  for (const msg of rows) {
    if (tokenCount + msg.tokenEstimate > maxTokens) break;
    window.unshift(msg); // prepend to maintain chronological order
    tokenCount += msg.tokenEstimate;
  }

  return window;
}

async function chat(
  sessionId: string,
  userMessage: string
): Promise<string> {
  // Save user message
  const userTokens = Math.ceil(userMessage.length / 4);
  await db.query(
    `INSERT INTO messages (session_id, role, content, token_estimate)
     VALUES ($1, 'user', $2, $3)`,
    [sessionId, userMessage, userTokens]
  );

  const history = await getRecentMessages(sessionId, 8000);

  const response = await anthropic.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    messages: history.map((m) => ({ role: m.role, content: m.content })),
  });

  const assistantText =
    response.content[0].type === "text" ? response.content[0].text : "";

  // Save assistant reply
  await db.query(
    `INSERT INTO messages (session_id, role, content, token_estimate)
     VALUES ($1, 'assistant', $2, $3)`,
    [sessionId, assistantText, Math.ceil(assistantText.length / 4)]
  );

  return assistantText;
}

The schema supporting this:

CREATE TABLE sessions (
  id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  user_id     UUID NOT NULL,
  created_at  TIMESTAMPTZ DEFAULT now(),
  last_active TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE messages (
  id             UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  session_id     UUID REFERENCES sessions(id) ON DELETE CASCADE,
  role           TEXT NOT NULL CHECK (role IN ('user', 'assistant', 'system')),
  content        TEXT NOT NULL,
  token_estimate INT NOT NULL DEFAULT 0,
  created_at     TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX ON messages (session_id, created_at DESC);

The sliding window breaks down when users ask about things said early in a long conversation. If someone sets their preferences in message 3 and asks "what did I say my budget was?" in message 150, those early messages will have been evicted from the window. That is the prompt for Pattern 2.

Pattern 2: Periodic summarisation

Summarisation compresses older conversation history into a compact summary block that is prepended to the current window. When the conversation exceeds a threshold (say, 6,000 tokens of history), generate a summary of the oldest half and replace those messages with the compressed summary. The result is that the context window always starts with a summary of "what happened earlier" followed by recent verbatim messages.

async function summariseMessages(
  messages: ConversationMessage[]
): Promise<string> {
  const transcript = messages
    .map((m) => `${m.role === "user" ? "User" : "Assistant"}: ${m.content}`)
    .join("\n\n");

  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5", // Smaller model is fine for summarisation
    max_tokens: 512,
    messages: [
      {
        role: "user",
        content: `Summarise the following conversation segment concisely. Preserve any specific facts, numbers, preferences, names, or commitments that the user mentioned. Use third person for the user ("the user said...").

Conversation:
${transcript}

Summary:`,
      },
    ],
  });

  return response.content[0].type === "text"
    ? response.content[0].text
    : "";
}

async function maybeCompressHistory(
  sessionId: string,
  compressionThreshold = 6000
): Promise<void> {
  const { rows } = await db.query(
    `SELECT SUM(token_estimate) AS total
     FROM messages
     WHERE session_id = $1 AND is_summary = false`,
    [sessionId]
  );

  const totalTokens = parseInt(rows[0].total ?? "0");

  if (totalTokens < compressionThreshold) return;

  // Get the oldest half of non-summary messages
  const { rows: messages } = await db.query<ConversationMessage & { id: string }>(
    `SELECT id, role, content, token_estimate AS "tokenEstimate"
     FROM messages
     WHERE session_id = $1 AND is_summary = false
     ORDER BY created_at ASC
     LIMIT (
       SELECT COUNT(*) / 2 FROM messages WHERE session_id = $1 AND is_summary = false
     )`,
    [sessionId]
  );

  if (messages.length < 4) return; // Don't summarise very short segments

  const summary = await summariseMessages(messages);
  const summaryTokens = Math.ceil(summary.length / 4);

  // Atomic replacement: delete old messages, insert summary
  await db.query("BEGIN");
  try {
    const ids = messages.map((m) => m.id);
    await db.query(
      `DELETE FROM messages WHERE id = ANY($1::uuid[])`,
      [ids]
    );

    await db.query(
      `INSERT INTO messages (session_id, role, content, token_estimate, is_summary)
       VALUES ($1, 'system', $2, $3, true)`,
      [sessionId, `[Earlier conversation summary]\n${summary}`, summaryTokens]
    );

    await db.query("COMMIT");
  } catch (err) {
    await db.query("ROLLBACK");
    throw err;
  }
}

Add the is_summary column to the messages table:

ALTER TABLE messages ADD COLUMN is_summary BOOLEAN NOT NULL DEFAULT false;
CREATE INDEX ON messages (session_id, is_summary, created_at DESC);

Call maybeCompressHistory at the start of each chat turn, before retrieving the window. This keeps the effective history length bounded while preserving the semantic content of older exchanges.

Pattern 3: Entity extraction memory

Summarisation preserves context but is lossy — the summary writer decides what matters, and it may not match what the user eventually asks about. Entity extraction takes a different approach: parse every message in real time for named entities and facts, store them in a structured key-value memory, and inject the relevant subset into every prompt.

This pattern shines for assistant use cases where the user shares factual information about themselves — their name, preferences, location, project details — that should influence every future response.

interface MemoryFact {
  key: string;
  value: string;
  confidence: number;
  sourceMessageId: string;
  updatedAt: Date;
}

const extractionSchema = {
  type: "object" as const,
  properties: {
    facts: {
      type: "array",
      items: {
        type: "object",
        properties: {
          key: { type: "string", description: "Canonical fact key, e.g. 'user.name', 'user.budget', 'project.language'" },
          value: { type: "string", description: "The extracted value" },
          confidence: { type: "number", description: "0-1 confidence in extraction accuracy" },
        },
        required: ["key", "value", "confidence"],
      },
    },
  },
  required: ["facts"],
};

async function extractFacts(
  sessionId: string,
  messageId: string,
  messageText: string
): Promise<void> {
  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5",
    max_tokens: 512,
    tools: [
      {
        name: "store_facts",
        description: "Store extracted facts from the user message",
        input_schema: extractionSchema,
      },
    ],
    tool_choice: { type: "tool", name: "store_facts" },
    messages: [
      {
        role: "user",
        content: `Extract any factual information the user shares about themselves, their project, preferences, or goals. Only extract clear, stated facts — not inferences.

User message: ${messageText}`,
      },
    ],
  });

  const toolUse = response.content.find((b) => b.type === "tool_use");
  if (!toolUse || toolUse.type !== "tool_use") return;

  const { facts } = toolUse.input as { facts: Array<{ key: string; value: string; confidence: number }> };

  for (const fact of facts) {
    if (fact.confidence < 0.7) continue; // Skip low-confidence extractions

    await db.query(
      `INSERT INTO memory_facts (session_id, key, value, confidence, source_message_id)
       VALUES ($1, $2, $3, $4, $5)
       ON CONFLICT (session_id, key) DO UPDATE
       SET value = $3, confidence = $4, source_message_id = $5, updated_at = now()`,
      [sessionId, fact.key, fact.value, fact.confidence, messageId]
    );
  }
}

async function buildMemoryContext(sessionId: string): Promise<string> {
  const { rows } = await db.query<MemoryFact>(
    `SELECT key, value, confidence
     FROM memory_facts
     WHERE session_id = $1
     ORDER BY updated_at DESC`,
    [sessionId]
  );

  if (rows.length === 0) return "";

  const lines = rows.map(
    (f) => `- ${f.key}: ${f.value}`
  );

  return `Known facts about the user and their context:\n${lines.join("\n")}`;
}

Schema for the memory facts table:

CREATE TABLE memory_facts (
  id                UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  session_id        UUID REFERENCES sessions(id) ON DELETE CASCADE,
  key               TEXT NOT NULL,
  value             TEXT NOT NULL,
  confidence        NUMERIC(3,2) NOT NULL,
  source_message_id UUID REFERENCES messages(id),
  updated_at        TIMESTAMPTZ DEFAULT now(),
  UNIQUE (session_id, key)
);

CREATE INDEX ON memory_facts (session_id);

The UNIQUE (session_id, key) constraint combined with the ON CONFLICT DO UPDATE upsert means newer mentions of the same fact automatically supersede older ones. If the user changes their budget, the updated value replaces the old one rather than creating a duplicate.

Pattern 4: Retrieval-augmented memory

For very long-running assistants — agents that a user returns to over weeks or months — even periodic summarisation eventually accumulates more summary text than fits in the context window. Retrieval-augmented memory solves this by storing memory as a vector database of fact chunks and retrieving only the chunks relevant to each new query.

interface MemoryChunk {
  id: string;
  sessionId: string;
  text: string;
  embedding: number[];
  importanceScore: number;
  createdAt: Date;
}

async function storeMemoryChunk(
  sessionId: string,
  text: string,
  importanceScore: number
): Promise<void> {
  const embeddingResponse = await anthropic.embeddings.create({
    model: "voyage-3",
    input: text,
  });
  const embedding = embeddingResponse.data[0].embedding;

  await db.query(
    `INSERT INTO memory_chunks (session_id, text, embedding, importance_score)
     VALUES ($1, $2, $3, $4)`,
    [sessionId, text, JSON.stringify(embedding), importanceScore]
  );
}

async function retrieveRelevantMemories(
  sessionId: string,
  query: string,
  topK = 5
): Promise<string[]> {
  const embeddingResponse = await anthropic.embeddings.create({
    model: "voyage-3",
    input: query,
  });
  const queryEmbedding = embeddingResponse.data[0].embedding;

  const { rows } = await db.query<{ text: string; relevance: number }>(
    `SELECT text, 
            (1 - (embedding <=> $2::vector)) * importance_score AS relevance
     FROM memory_chunks
     WHERE session_id = $1
     ORDER BY (1 - (embedding <=> $2::vector)) * importance_score DESC
     LIMIT $3`,
    [sessionId, JSON.stringify(queryEmbedding), topK]
  );

  return rows.map((r) => r.text);
}

The relevance score multiplies vector similarity by an importance_score, so memories the user marked as important or that were derived from high-signal events (e.g., explicit user corrections, confirmed decisions) rank higher than routine conversational noise.

Combining patterns in production

The four patterns are not mutually exclusive. Production systems typically use a layered approach:

  1. Sliding window (last 10-20 messages verbatim) for conversational flow
  2. Entity extraction running in the background on every user message to maintain a live fact store
  3. Retrieval-augmented memory for long-term recall of relevant past exchanges
  4. Periodic summarisation as a cost-control valve when daily active usage is high

Assemble the context window by concatenating: retrieved long-term memories, entity facts, summary of older recent history, and the raw recent messages window. Keep a token budget for each layer so one component cannot crowd out the others:

async function assembleContext(
  sessionId: string,
  query: string
): Promise<Anthropic.MessageParam[]> {
  const [memories, facts, recentMessages] = await Promise.all([
    retrieveRelevantMemories(sessionId, query, 5),
    buildMemoryContext(sessionId),
    getRecentMessages(sessionId, 4000),
  ]);

  const systemParts: string[] = [];

  if (memories.length > 0) {
    systemParts.push(
      `Relevant past context:\n${memories.map((m) => `- ${m}`).join("\n")}`
    );
  }

  if (facts) {
    systemParts.push(facts);
  }

  const systemContent = systemParts.join("\n\n");

  const messages: Anthropic.MessageParam[] = [];

  if (recentMessages.length > 0) {
    messages.push(
      ...recentMessages.map((m) => ({
        role: m.role as "user" | "assistant",
        content: m.content,
      }))
    );
  }

  return messages;
}

Monitoring memory health in production

Track these metrics per session to detect memory degradation before users notice:

  • Effective context utilisation: the fraction of the context window filled by actual conversation vs. overhead. Drop below 40% and you are wasting capacity; exceed 90% and you risk truncation.
  • Memory retrieval hit rate: the fraction of queries where the retrieved memory chunks were cited in the response. Low hit rate suggests your embedding model is poor at matching query semantics to stored memories.
  • Fact extraction accuracy: manually audit a sample of extracted facts against the source messages monthly. Degradation here usually means the user is phrasing things in new ways the extraction prompt does not handle.
  • Summarisation quality score: run your summarisation output through a secondary LLM call that scores completeness and accuracy on a 1-5 scale. Alert if the average drops below 3.5.

Memory architecture is the most consequential design decision for long-running AI assistants. The patterns above give you a toolkit that scales from a simple weekend project to a production system serving millions of conversation turns per day.

Comments

No comments yet. Be the first!

Sign in to leave a comment.