← Articles

LLM latency optimisation: batching, caching, and model selection

By Chen Wei · 16 July 20260 views
#llm#latency#performance#caching#batching

Slow LLM responses destroy user experience. A two-second delay on a search result is tolerable; a two-second delay on every keystroke in a chat interface is not. Optimising LLM latency is distinct from optimising traditional API latency because the bottleneck is almost entirely on the model inference side — and that bottleneck has specific, exploitable structure.

This article covers the techniques that consistently produce the largest gains, in order of typical impact, with concrete implementation patterns for each.

Understanding where time goes

Before optimising, measure. LLM latency has two distinct components that need separate instrumentation:

Time to first token (TTFT) — how long before streaming begins. This is what users perceive as "lag" in interactive applications. It includes request serialisation, network round-trip, prompt processing (prefill), and sampling setup.

Inter-token latency (ITL) — time between successive tokens during streaming. At typical generation speeds (50–100 tokens/second), this adds roughly 10–20ms per token, so a 500-token response adds 5–10 seconds of streaming time on top of TTFT.

interface LLMLatencyMetrics {
  requestSentAt: number;
  firstTokenAt: number | null;
  lastTokenAt: number | null;
  inputTokens: number;
  outputTokens: number;
}

async function* streamWithMetrics(
  params: Anthropic.MessageCreateParamsStreaming,
  client: Anthropic
): AsyncGenerator<string, LLMLatencyMetrics> {
  const metrics: LLMLatencyMetrics = {
    requestSentAt: Date.now(),
    firstTokenAt: null,
    lastTokenAt: null,
    inputTokens: 0,
    outputTokens: 0,
  };

  const stream = client.messages.stream(params);

  for await (const event of stream) {
    if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
      if (metrics.firstTokenAt === null) {
        metrics.firstTokenAt = Date.now();
      }
      metrics.lastTokenAt = Date.now();
      yield event.delta.text;
    }

    if (event.type === "message_delta" && event.usage) {
      metrics.outputTokens = event.usage.output_tokens;
    }

    if (event.type === "message_start" && event.message.usage) {
      metrics.inputTokens = event.message.usage.input_tokens;
    }
  }

  return metrics;
}

Track TTFT and ITL separately in your metrics. You may find TTFT is fine but users are complaining about slow responses — that points to a long output, not a slow model. Shortening output constraints is often the highest-leverage fix.

Prompt caching: the highest-ROI optimisation

Prompt caching allows you to store a processed prefix on the model provider's infrastructure and skip reprocessing it on subsequent requests. On the Anthropic API, cached tokens cost 10% of regular input token price and skip the prefill computation entirely, which directly reduces TTFT.

The canonical use case is a large, stable system prompt. If your system prompt is 4,000 tokens and you pay $3 per million input tokens, each request costs $0.012 in system prompt tokens alone. With caching at $0.30 per million, that drops to $0.0012. At 100,000 daily requests, that is $1,080/month saved on the system prompt alone.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// Large stable context — policy docs, codebase, knowledge base
const LARGE_STABLE_CONTEXT = `
[Your 10,000-token system context here — product docs, policy, examples]
`;

async function queryWithCaching(
  userMessage: string,
  conversationHistory: Anthropic.MessageParam[]
) {
  const response = await client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    system: [
      {
        type: "text",
        text: LARGE_STABLE_CONTEXT,
        // This prefix will be cached after the first call
        cache_control: { type: "ephemeral" },
      },
      {
        type: "text",
        // Dynamic per-session context is NOT cached (not cost-effective)
        text: `Current user session: ${new Date().toISOString()}`,
      },
    ],
    messages: [
      ...conversationHistory,
      { role: "user", content: userMessage },
    ],
  });

  // Check if cache was hit
  const usage = response.usage as Anthropic.Usage & {
    cache_read_input_tokens?: number;
    cache_creation_input_tokens?: number;
  };

  const cacheHit = (usage.cache_read_input_tokens ?? 0) > 0;
  console.log(`Cache hit: ${cacheHit}, tokens: ${JSON.stringify(usage)}`);

  return response;
}

Cache design rules:

  • Place the cache breakpoint (cache_control) after stable content and before dynamic content
  • The cache is per-model, per-account, and keyed on the exact byte sequence — even a one-token change invalidates it
  • Cache entries persist for roughly 5 minutes of inactivity; keep-warm requests every 4 minutes are cheap if your cache is large
  • Do not cache conversation history — it changes every turn

Model selection and routing

Different models have dramatically different latency profiles. Claude Haiku 3 returns first tokens roughly 5x faster than Sonnet for similar prompts. The trick is routing requests to the right model based on task complexity.

type TaskComplexity = "simple" | "moderate" | "complex";

interface RoutingCriteria {
  maxInputTokens: number;
  requiresReasoning: boolean;
  requiresCodeGeneration: boolean;
  latencySensitive: boolean;
  acceptableOutputQuality: "draft" | "production" | "high-stakes";
}

function selectModel(criteria: RoutingCriteria): string {
  // High-stakes decisions: always use best model regardless of speed
  if (criteria.acceptableOutputQuality === "high-stakes") {
    return "claude-opus-4-5";
  }

  // Complex reasoning or code: Sonnet is the floor
  if (criteria.requiresReasoning || criteria.requiresCodeGeneration) {
    return "claude-sonnet-4-5";
  }

  // Long inputs benefit from Sonnet (Haiku degrades on long context)
  if (criteria.maxInputTokens > 8000) {
    return "claude-sonnet-4-5";
  }

  // Latency-sensitive + simple task = Haiku
  if (criteria.latencySensitive && criteria.acceptableOutputQuality === "draft") {
    return "claude-haiku-3-5";
  }

  return "claude-sonnet-4-5"; // default
}

// Example: autocomplete suggestions → Haiku
const autocompleteModel = selectModel({
  maxInputTokens: 500,
  requiresReasoning: false,
  requiresCodeGeneration: false,
  latencySensitive: true,
  acceptableOutputQuality: "draft",
});

// Example: contract analysis → Opus
const contractModel = selectModel({
  maxInputTokens: 20000,
  requiresReasoning: true,
  requiresCodeGeneration: false,
  latencySensitive: false,
  acceptableOutputQuality: "high-stakes",
});

A/B test model selection decisions. In our experience, Haiku handles 40–60% of most application's requests acceptably, which reduces average latency significantly without user-visible quality degradation.

Batching non-interactive requests

For workloads that can tolerate latency — document processing, nightly analysis, content indexing, report generation — use the batch API. Batch requests are processed asynchronously and cost 50% less than synchronous calls.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

interface BatchItem {
  customId: string;
  document: string;
  taskDescription: string;
}

async function submitBatch(items: BatchItem[]) {
  const requests: Anthropic.MessageCreateParamsNonStreaming[] = items.map((item) => ({
    custom_id: item.customId,
    params: {
      model: "claude-haiku-3-5",
      max_tokens: 256,
      messages: [
        {
          role: "user",
          content: `${item.taskDescription}\n\n${item.document}`,
        },
      ],
    },
  }));

  const batch = await client.batches.create({ requests });
  console.log(`Batch ${batch.id} submitted with ${items.length} items`);
  return batch.id;
}

async function pollBatchResults(batchId: string): Promise<Map<string, string>> {
  const results = new Map<string, string>();

  while (true) {
    const batch = await client.batches.retrieve(batchId);

    if (batch.processing_status === "ended") {
      // Fetch results
      for await (const result of await client.batches.results(batchId)) {
        if (result.result.type === "succeeded") {
          const text = result.result.message.content
            .filter((b) => b.type === "text")
            .map((b) => (b as Anthropic.TextBlock).text)
            .join("");
          results.set(result.custom_id, text);
        } else {
          console.error(`Failed: ${result.custom_id}`, result.result.error);
        }
      }
      break;
    }

    console.log(
      `Batch status: ${batch.processing_status}, ` +
        `completed: ${batch.request_counts.succeeded}/${batch.request_counts.processing}`
    );
    await new Promise((r) => setTimeout(r, 30_000)); // Poll every 30s
  }

  return results;
}

Separate your traffic into two streams: interactive (synchronous, optimise for TTFT) and background (batch, optimise for cost). Most applications have more background work than they realise.

Response caching at the application layer

LLM responses for identical inputs are deterministic at temperature=0. Cache them.

import { createClient } from "redis";
import crypto from "crypto";

const redis = createClient({ url: process.env.REDIS_URL });

interface CacheConfig {
  ttlSeconds: number;
  model: string;
  systemPromptVersion: string;
}

function buildCacheKey(
  userMessage: string,
  conversationHistory: Anthropic.MessageParam[],
  config: CacheConfig
): string {
  const payload = JSON.stringify({
    model: config.model,
    systemVersion: config.systemPromptVersion,
    history: conversationHistory,
    message: userMessage,
  });
  return `llm:${crypto.createHash("sha256").update(payload).digest("hex")}`;
}

async function cachedQuery(
  userMessage: string,
  conversationHistory: Anthropic.MessageParam[],
  config: CacheConfig,
  queryFn: () => Promise<string>
): Promise<{ text: string; fromCache: boolean }> {
  const key = buildCacheKey(userMessage, conversationHistory, config);

  const cached = await redis.get(key);
  if (cached) {
    return { text: cached, fromCache: true };
  }

  const result = await queryFn();
  await redis.setEx(key, config.ttlSeconds, result);

  return { text: result, fromCache: false };
}

What to cache and for how long:

  • FAQ responses, documentation queries — 24 hours; the underlying documents change infrequently
  • Classification results for identical text — 1 hour; re-classify after policy updates
  • Conversational responses — generally do not cache; context changes per user
  • Code generation for the same prompt — 4 hours; useful for documentation generation pipelines

Never cache at temperature > 0.3. The non-determinism defeats the cache.

Streaming and perceived latency

Even if TTFT is fast, users will feel the application is slow if they see a blank screen while waiting for a complete response. Stream everything interactive.

// Next.js Route Handler with streaming
import { NextRequest } from "next/server";
import Anthropic from "@anthropic-ai/sdk";

export const runtime = "edge";

const client = new Anthropic();

export async function POST(req: NextRequest) {
  const { message, history } = await req.json();

  const encoder = new TextEncoder();

  const stream = new ReadableStream({
    async start(controller) {
      try {
        const anthropicStream = client.messages.stream({
          model: "claude-haiku-3-5",
          max_tokens: 1024,
          messages: [...history, { role: "user", content: message }],
        });

        for await (const event of anthropicStream) {
          if (
            event.type === "content_block_delta" &&
            event.delta.type === "text_delta"
          ) {
            controller.enqueue(encoder.encode(`data: ${JSON.stringify({ text: event.delta.text })}\n\n`));
          }
        }

        controller.enqueue(encoder.encode("data: [DONE]\n\n"));
      } catch (err) {
        controller.error(err);
      } finally {
        controller.close();
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      Connection: "keep-alive",
    },
  });
}

On the client side, render tokens as they arrive. React state batching in React 18+ batches rapid state updates — use a ref to accumulate and a low-frequency interval to flush to state:

function useStreamingText(streamUrl: string) {
  const [text, setText] = useState("");
  const bufferRef = useRef("");

  const flush = useCallback(() => {
    if (bufferRef.current !== "") {
      setText((prev) => prev + bufferRef.current);
      bufferRef.current = "";
    }
  }, []);

  useEffect(() => {
    const interval = setInterval(flush, 16); // ~60fps
    return () => clearInterval(interval);
  }, [flush]);

  const startStream = useCallback(async (message: string) => {
    setText("");
    bufferRef.current = "";

    const eventSource = new EventSource(streamUrl);
    eventSource.onmessage = (e) => {
      if (e.data === "[DONE]") {
        flush();
        eventSource.close();
        return;
      }
      const { text } = JSON.parse(e.data);
      bufferRef.current += text;
    };
  }, [streamUrl, flush]);

  return { text, startStream };
}

Output length control

The single fastest way to reduce total response time is to generate fewer tokens. Output length is entirely within your control via the prompt and max_tokens.

Prompts that produce long outputs:

  • "Explain..." — open-ended, models fill space
  • "Write a detailed..." — explicitly requests length
  • No specified format — models default to prose with padding

Prompts that produce short outputs:

  • "Answer in one sentence: ..."
  • "Extract only the relevant fields as JSON: ..."
  • "Rate on a scale of 1-10 with a two-sentence justification: ..."

For classification tasks, instruct the model to return only the structured output:

// Before: model writes prose then JSON — ~300 tokens
const verbosePrompt = `Analyze this text and classify its sentiment. 
Consider the overall tone, specific word choices, and context before 
providing your final answer in JSON format.`;

// After: model returns JSON directly — ~30 tokens
const concisePrompt = `Classify the sentiment of the following text.
Respond with only a JSON object: {"sentiment": "positive"|"negative"|"neutral", "confidence": 0.0-1.0}`;

The difference is 10x in output tokens and roughly 3–5x in total wall-clock time for classification tasks.

Parallel requests and speculative prefetching

When you know what the next request will likely be, prefetch it in parallel with rendering the current response.

async function handleUserTurn(
  userMessage: string,
  conversationHistory: Anthropic.MessageParam[],
  client: Anthropic
) {
  // Primary response
  const primaryPromise = client.messages.create({
    model: "claude-sonnet-4-5",
    max_tokens: 1024,
    messages: [...conversationHistory, { role: "user", content: userMessage }],
  });

  // Speculative: generate follow-up suggestions in parallel
  // (used only if user doesn't type immediately)
  const followUpPromise = client.messages.create({
    model: "claude-haiku-3-5",
    max_tokens: 100,
    messages: [
      ...conversationHistory,
      { role: "user", content: userMessage },
      // Guess the assistant's response for context
      {
        role: "assistant",
        content: "I'll help you with that. ",
      },
      {
        role: "user",
        content:
          "Based on this conversation, give me 3 short follow-up questions the user might ask next. JSON array only.",
      },
    ],
  });

  const [primary, followUps] = await Promise.allSettled([
    primaryPromise,
    followUpPromise,
  ]);

  return { primary, followUps };
}

This technique works well in chat interfaces where displaying suggested follow-ups is a product feature anyway. You get the suggestions "for free" by overlapping their generation with the primary response.

Infrastructure choices

Beyond API-level optimisations, infrastructure placement matters significantly:

Deploy close to the API endpoint. Anthropic's API is served from US-East. A request from us-east-1 has ~5ms round-trip; from us-west-2, ~80ms; from eu-west-1, ~150ms. For high-frequency interactive applications, this is material.

Connection pooling. HTTPS connection establishment adds 50–150ms for cold connections. Keep-alive with connection pooling eliminates this:

import https from "https";
import Anthropic from "@anthropic-ai/sdk";

const agent = new https.Agent({
  keepAlive: true,
  keepAliveMsecs: 30000,
  maxSockets: 50,
});

const client = new Anthropic({
  httpAgent: agent, // reuse connections across requests
});

Edge functions for static prompt pre-processing. If your prompt construction involves static operations (template filling, JSON serialisation), do this in an edge function close to the user, then forward to a regional function that makes the API call. Reduces the user-perceived TTFT by the client-to-edge round-trip.

The compound effect of these optimisations — prompt caching, model routing, response caching, short outputs, streaming, and connection pooling — consistently reduces p50 TTFT by 50–70% and p99 by 60–80% compared to a naive implementation. Measure each independently to understand which lever matters most for your specific workload.

Comments

No comments yet. Be the first!

Sign in to leave a comment.