← Articles

Prompt caching with the Anthropic API: when and how to use it

By Sophia Papadopoulos · 15 July 20260 views
#prompt-caching#anthropic#claude#performance#cost-optimisation

The cost structure of large language model APIs

Every call to an LLM API involves two kinds of work: processing the input tokens (prefill) and generating the output tokens (decode). Prefill scales linearly with context length — doubling your system prompt doubles your input processing cost. For applications that send thousands of requests per hour against the same 10,000-token system prompt, input costs dominate the bill.

Prompt caching addresses this directly. When you mark content with a cache control annotation, the API computes and stores the KV (key-value) attention state for that content after the first request. Subsequent requests that share the same cached prefix skip the prefill computation for those tokens. You pay 10% of the normal input token price for cache hits, and time-to-first-token drops substantially because prefill is the dominant latency contributor for long prompts.

This sounds straightforward, but prompt caching has non-obvious semantics around what constitutes a "matching" prefix, where you can place cache boundaries, and what causes cache misses that look like hits.

What gets cached and what does not

Prompt caching operates at the prefix level. The API caches the KV state at the point where your cache boundary marker appears. For a subsequent request to hit the cache, every token from the beginning of the message list up to (and including) the cache boundary must be byte-for-byte identical to the original cached request.

A single character difference anywhere in that prefix invalidates the cache. This has implications:

  • Dynamic content (timestamps, user IDs, session tokens) placed before the cache boundary will always miss.
  • Whitespace differences matter. A trailing newline that was present in one request but not another is a miss.
  • The order of messages matters. Inserting a new message before the cache boundary is a miss.

The cache TTL is 5 minutes by default. If your traffic rate drops such that 5 minutes pass between requests hitting the same cache, the cached state is evicted and the next request pays the full creation cost. Anthropic offers an extended TTL option (type: "extended" in cache_control) that increases the TTL to 1 hour, useful for development workflows where request frequency is low.

Where you can place cache boundaries

Cache boundaries can appear in three locations:

  1. Within the system block, at the end of any content block
  2. In the messages array, at the end of a user turn content block
  3. Within tool definitions

You cannot place a cache boundary inside a string mid-sentence. The boundary is always at the end of a complete content block.

The most common pattern is a single boundary at the end of the system prompt:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const response = await client.messages.create({
  model: "claude-opus-4-5",
  max_tokens: 1024,
  system: [
    {
      type: "text",
      text: LARGE_SYSTEM_PROMPT, // e.g., 5000 tokens of instructions
      cache_control: { type: "ephemeral" },
    },
  ],
  messages: [
    { role: "user", content: "What is the refund policy for digital downloads?" },
  ],
});

The system prompt is cached after the first request. The user message is not — it changes with every request.

The minimum token threshold

Cache creation only triggers when the prefix to be cached is at least 1024 tokens for Claude Haiku models and 2048 tokens for Sonnet and Opus models. Below the threshold, the API processes the request normally without writing to the cache, and cache_creation_input_tokens will be 0.

Check the actual threshold behaviour in your usage response:

const usage = response.usage;
console.log({
  inputTokens: usage.input_tokens,
  cacheCreationTokens: usage.cache_creation_input_tokens ?? 0,
  cacheReadTokens: usage.cache_read_input_tokens ?? 0,
  outputTokens: usage.output_tokens,
});

If cacheCreationTokens is 0 on the first request and the prefix is below the threshold, you need to expand it. This is common when you're experimenting with short system prompts — add detailed instructions, few-shot examples, or the full set of tool definitions to push past the threshold.

Calculating the break-even point

Before instrumenting caching, verify the economics make sense for your traffic pattern.

Let token costs be:

  • Normal input: $P_{in}$ per MTok
  • Cache write: $P_{write}$ per MTok (typically $1.25 \times P_{in}$)
  • Cache read: $P_{read}$ per MTok (typically $0.1 \times P_{in}$)

For a prefix of $N$ tokens repeated across $R$ requests within the cache TTL, cost without caching is:

$$C_{no_cache} = R \times N \times P_{in}$$

Cost with caching (first request creates, remainder read):

$$C_{cache} = N \times P_{write} + (R - 1) \times N \times P_{read}$$

Break-even at $R^*$ requests:

$$R^* = \frac{P_{write} - P_{read}}{P_{in} - P_{read}} \approx \frac{1.25 - 0.1}{1.0 - 0.1} \approx 1.28$$

With these typical prices, you break even after just two requests. From the third request onward, every cache hit saves approximately 90% of the input token cost for that prefix. For any application that sends more than a handful of requests per hour against a shared prefix, caching pays off significantly.

The calculation changes if your cache TTL is frequently expiring. If R (requests per TTL window) is close to 1, caching adds cost instead of saving it.

Multiple cache breakpoints

You can place up to four cache breakpoints per request. This is useful when you have layered stable content:

const systemBlocks: Anthropic.TextBlockParam[] = [
  {
    type: "text",
    // Layer 1: Permanent company context (rarely changes)
    text: COMPANY_CONTEXT_PROMPT,
    cache_control: { type: "ephemeral" },
  },
  {
    type: "text",
    // Layer 2: Per-user subscription features (changes daily at most)
    text: buildUserFeaturePrompt(user.subscriptionTier),
    cache_control: { type: "ephemeral" },
  },
];

const response = await client.messages.create({
  model: "claude-opus-4-5",
  max_tokens: 1024,
  system: systemBlocks,
  messages: [{ role: "user", content: userMessage }],
});

With two breakpoints, the API caches the company context after the first request from any user, and caches the per-user feature prompt after the first request from each user tier. Requests from users on the same tier but with different messages can hit both caches.

The key constraint: cache breakpoints are evaluated from the beginning of the request. If layer 1 changes, layer 2's cache is also invalidated (because layer 2 sits after layer 1 in the prefix). Structure your breakpoints with the most stable content first and the most dynamic content last.

Caching conversation history

Multi-turn conversations are a natural fit for caching because the history grows monotonically and earlier turns are immutable:

type ConversationTurn = Anthropic.MessageParam;

async function continueConversation(
  history: ConversationTurn[],
  newMessage: string,
  systemPrompt: string
): Promise<string> {
  // Mark the last user turn in history with a cache breakpoint
  const cachedHistory = history.map((turn, index) => {
    if (index === history.length - 1 && turn.role === "user") {
      // Cache up to (but not including) the new message
      const content = Array.isArray(turn.content)
        ? turn.content.map((block, bi) => ({
            ...block,
            ...(bi === turn.content.length - 1
              ? { cache_control: { type: "ephemeral" as const } }
              : {}),
          }))
        : [
            {
              type: "text" as const,
              text: turn.content as string,
              cache_control: { type: "ephemeral" as const },
            },
          ];
      return { ...turn, content };
    }
    return turn;
  });

  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 2048,
    system: [
      {
        type: "text",
        text: systemPrompt,
        cache_control: { type: "ephemeral" },
      },
    ],
    messages: [...cachedHistory, { role: "user", content: newMessage }],
  });

  const textBlock = response.content.find((b) => b.type === "text");
  return textBlock?.type === "text" ? textBlock.text : "";
}

Each new turn extends the prefix. The previous-turn cache remains valid, so you pay full input price only for the new turn's tokens, not for the entire history. In a 20-turn conversation with 200 tokens per turn, by turn 10 you've accumulated 2000 tokens of cacheable history. Without caching, turn 20 would cost 4000 input tokens. With caching, it costs 200 (just the new user turn) plus a cache read of the prior 3800 tokens at 10% cost — an 82% reduction.

Caching tool definitions

If your application sends the same set of tool definitions on every request, cache them too. Tool definitions can be large — each tool with a detailed description and a complex input_schema may be 200–500 tokens. A set of 10 such tools is 2000–5000 tokens of overhead on every request:

const tools: Anthropic.Tool[] = [
  /* ... your tool definitions ... */
];

// Note: cache_control on tools uses a slightly different structure
// as of the current SDK version; check the latest API docs for the
// exact field placement, which may vary by SDK version
const response = await client.messages.create({
  model: "claude-opus-4-5",
  max_tokens: 1024,
  system: [
    { type: "text", text: systemPrompt, cache_control: { type: "ephemeral" } },
  ],
  tools,
  messages: [{ role: "user", content: userMessage }],
});

When tool definitions are stable (which they usually are in production), caching them alongside the system prompt captures the full static overhead of each request.

Common mistakes that silently disable caching

Several mistakes cause caching to silently fail — you think you're hitting the cache but you're not.

Dynamic content before the cache boundary. If you include Date.now() or a request UUID in the system prompt before the cache_control marker, every request is a cache miss:

// Wrong — includes timestamp before cache boundary
const system = [
  {
    type: "text" as const,
    // This changes every second, so every request is a miss
    text: `System time: ${new Date().toISOString()}\n\n${STABLE_PROMPT}`,
    cache_control: { type: "ephemeral" as const },
  },
];

// Correct — dynamic content goes after the cache boundary
const system2: Anthropic.TextBlockParam[] = [
  {
    type: "text",
    text: STABLE_PROMPT,
    cache_control: { type: "ephemeral" },
  },
  {
    type: "text",
    // This block has no cache_control — it's after the boundary
    text: `Current time: ${new Date().toISOString()}`,
  },
];

Serialisation instability. JSON.stringify() does not guarantee stable key ordering across JavaScript engines or versions. If you construct the system prompt by serialising a JavaScript object, add a stable serialiser:

import stableStringify from "json-stable-stringify";

const contextJson = stableStringify(userContext, { space: 2 });
const systemPrompt = `${STABLE_INSTRUCTIONS}\n\nContext:\n${contextJson}`;

Whitespace normalisation. A common framework-level mistake is a middleware that strips trailing whitespace or normalises newlines. If STABLE_PROMPT passes through such middleware inconsistently, you'll get cache misses.

Model version changes. Caches are per model version. If you upgrade from claude-opus-4-5 to claude-opus-4-5-20251001, all cached prefixes for the old version are invalid.

Measuring cache effectiveness in production

Add cache metrics to your LLM observability pipeline:

interface LLMMetrics {
  model: string;
  inputTokens: number;
  cacheCreateTokens: number;
  cacheReadTokens: number;
  outputTokens: number;
  durationMs: number;
}

function recordMetrics(response: Anthropic.Message, durationMs: number): LLMMetrics {
  const usage = response.usage;
  const metrics: LLMMetrics = {
    model: response.model,
    inputTokens: usage.input_tokens,
    cacheCreateTokens: usage.cache_creation_input_tokens ?? 0,
    cacheReadTokens: usage.cache_read_input_tokens ?? 0,
    outputTokens: usage.output_tokens,
    durationMs,
  };

  // Track cache hit rate
  const totalCacheableTokens = metrics.cacheCreateTokens + metrics.cacheReadTokens;
  const cacheHitRate =
    totalCacheableTokens > 0
      ? metrics.cacheReadTokens / totalCacheableTokens
      : 0;

  // Estimate cost savings
  const INPUT_COST_PER_MTOK = 3.0; // example for Opus
  const CACHE_READ_COST_PER_MTOK = 0.30;
  const savedCost =
    (metrics.cacheReadTokens / 1_000_000) *
    (INPUT_COST_PER_MTOK - CACHE_READ_COST_PER_MTOK);

  console.log(JSON.stringify({
    ...metrics,
    cacheHitRate: cacheHitRate.toFixed(3),
    estimatedSavedDollars: savedCost.toFixed(6),
  }));

  return metrics;
}

Track cacheHitRate over time. A healthy production application with stable prefixes should see hit rates above 80%. If you're seeing hit rates below 50%, investigate prefix instability or TTL exhaustion.

When caching does not help

Caching is not a universal improvement. It adds complexity and a small overhead for cache misses. Avoid caching when:

  • Your system prompt is below the minimum token threshold and you cannot expand it meaningfully.
  • Every request has a unique prefix that cannot be split into a stable and dynamic portion.
  • Your application sends very low request volumes (under ~10 per hour) against the same prefix — the cache will consistently evict between requests.
  • You are doing one-off offline batch processing where latency is irrelevant and each document is processed once.

For everything else — interactive applications, high-throughput inference, conversational agents, RAG systems with stable context — prompt caching is one of the highest-leverage optimisations available, and it requires no changes to model or output quality.

Comments

No comments yet. Be the first!

Sign in to leave a comment.