Caching LLM Responses Safely: When it Helps and When it Hurts

Introduction: The Architecture of LLM-Integrated Applications

As developers, we are currently witnessing a shift where Large Language Models (LLMs) are becoming first-class citizens in our application architectures. However, integrating models like GPT-4 or Claude is rarely as simple as an API call. We are constrained by latency, expensive token costs, and rate limits. When building high-traffic applications that rely on these models, the instinct is often to cache everything. As someone who has spent years helping teams navigate Firestore data modelling, I can tell you that caching LLM responses is a classic 'denormalisation' problem. It is a decision that requires a rigorous framework—one that prioritizes consistency and user experience over the naive convenience of a cache-all approach.

In this article, we will dissect the decision-making process for caching LLM outputs. We will look at how to model this in Firestore, analyze the trade-offs of read/write ratios, and ultimately decide when caching helps your application scale and when it introduces technical debt that could break your product’s reliability.

The Denormalisation Decision Framework for AI Responses

When you decide to store an LLM response, you are effectively creating a 'denormalized' copy of ephemeral computation. My golden rule of denormalisation remains the starting point for every conversation: Do you optimize for the read path or the write path?

For LLM responses, the read-to-write ratio is almost always heavily skewed towards reads. If a user asks a question, they might refresh that page or view that response multiple times. More importantly, if multiple users ask the same question, caching allows you to serve the result instantly without incurring the latency of a second inference pass. However, unlike a static user profile, an LLM response is 'heavy' data. Storing these strings in Firestore documents changes your document size distribution.

To decide whether to cache, apply this framework:

1. The Determinism Test: Is the prompt deterministic (e.g., "Explain this legal clause") or context-dependent (e.g., "What should I do now, given my current location and heart rate?")? If the context is dynamic, a cache is a trap.
2. The Cost-Benefit Analysis: If the model costs $0.05 per request and the document read costs $0.0000003, you are saving money. But if the caching logic leads to "stale" answers that mislead the user, what is the cost of that user churn?
3. The Data Fan-out Potential: How many times will this specific prompt-response pair be accessed? If it's a one-off query, caching is just storage bloat.

Designing the Firestore Schema for Cached Responses

When we decide to cache, we must choose between two primary modelling patterns: the 'Embedded Response' or the 'Dedicated Response Collection.'

If the response is strictly tied to a single user session or task document, embedding it as a field can be tempting. However, for LLM applications, I advocate for a separate llm_responses collection. This allows you to handle TTL (Time-To-Live) policies via Firebase Extensions or Cloud Functions more cleanly without polluting your primary application state documents.

Consider this schema implementation in Kotlin/Android, representing a cache-first retrieval pattern:

data class LlmCacheEntry(
    val promptHash: String,
    val responseBody: String,
    val modelVersion: String,
    val createdAt: Timestamp,
    val hitCount: Int = 0
)

// Logic to retrieve or fetch
suspend fun getOrGenerateResponse(prompt: String): String {
    val hash = hashPrompt(prompt)
    val cacheDoc = firestore.collection("llm_cache").document(hash).get().await()
    
    return if (cacheDoc.exists()) {
        cacheDoc.toObject(LlmCacheEntry::class.java)?.responseBody ?: generateNew(prompt)
    } else {
        val newResponse = generateNew(prompt)
        saveToCache(hash, newResponse)
        newResponse
    }
}

The use of a promptHash as the document ID is crucial. By hashing the prompt, you create a natural index that allows for O(1) lookups. This pattern ensures that your read performance is decoupled from the complexity of the LLM generation itself.

The Hidden Risks: Consistency and the 'Stale Truth'

The biggest trap for beginners is ignoring the 'consistency trade-off.' When you cache, you are making a value judgment that the past is a perfect predictor of the future.

In many applications, the user's intent evolves. If a user asks a model to "summarize my project" and then updates their project details, a cached response is now factually incorrect. In a traditional database, you would perform a write to update the document. In a caching scenario, you have introduced a consistency hurdle: cache invalidation.

There are three levels of cache invalidation you must account for:

1. Time-based Invalidation: Simple, but blunt. After 24 hours, the cache is deleted. Great for general knowledge queries, terrible for personalized data.
2. Event-based Invalidation: The most robust. When the source data (the project document) is updated, you trigger a Cloud Function to delete the corresponding entry in the llm_cache collection.
3. Version-based Invalidation: If you update your system prompt or switch model versions, you must invalidate the entire cache. Always include a modelVersion field in your cache document to prevent serving legacy, low-quality responses after a model update.

Scaling Performance: Firestore as a Vector Cache

For advanced high-traffic applications, simple key-value caching (the hash approach) might not be enough. What if the user asks a slightly different question that has the same semantic meaning? For example, "Tell me about the tax law" vs "What is the tax regulation?"

Here, the standard hash-based cache fails. To truly optimize performance, you move into the realm of 'Semantic Caching.' You would store the embedding of the prompt in a vector database (like Pinecone or Firestore’s native Vector Search capability) and query for similar prompts before hitting the LLM API. This adds latency to the retrieval path, but it saves significantly more on the backend inference costs.

At scale, your Firestore data modelling must support this. You are no longer just storing text; you are storing metadata about the generation.

# Example of a structured cache document in Firestore
cache_id: "hash_12345"
prompt: "Explain the project status"
response: "The project is 80% complete..."
embedding_vector: [0.12, -0.05, 0.88, ...]
metadata:
  model: "gpt-4-turbo"
  temperature: 0.7
  tokens_used: 145
  last_updated: 2023-10-27T10:00:00Z

This structure allows you to perform analytics on your cache. You can see which prompts are 'expensive' (high token usage) and prioritize those for aggressive caching. It turns your cache into a business intelligence asset.

Final Recommendations for the Architect

Let’s summarize the framework for your next implementation:

1. Do not cache by default. Start by tracking your LLM costs and latency. If the cost is negligible and the model is fast, the complexity of managing a cache is actually a liability. Complexity is the silent killer of high-traffic applications.
2. Implement observability before caching. Before you cache, you need to know how often identical requests hit your API. Use Firestore counters or a simple analytics middleware to track 'duplicate request frequency.' If it’s under 5%, caching is likely premature optimization.
3. Favor explicit invalidation. If you must cache, create a clear path for invalidation. Do not rely solely on TTL. If your users rely on the accuracy of the LLM, a stale result is worse than a slightly slower, fresh result.
4. Monitor document size. When caching large LLM responses, remember that Firestore documents have a 1MB limit. If your LLM is generating long-form content or code, you might hit this limit faster than expected. Use a separate collection for these payloads and store only a reference in your primary user session documents.

By following this approach, you treat your cache not as a magic performance bullet, but as a deliberate architectural decision. You move from "caching to be fast" to "caching to be efficient and reliable." In my experience with high-traffic consultancy projects, the latter is the only way to build software that lasts. The goal is never to build the fastest application—it's to build one where you understand exactly where your data is, why it's there, and when it needs to be updated.

Keep your schemas lean, keep your consistency requirements explicit, and never underestimate the power of a well-indexed promptHash. Happy coding, and may your read/write ratios always work in your favor.