When Event-Driven Architecture Creates More Problems Than It Solves

The Allure of the 'Clean' Event-Driven Rewrite

There is a pervasive myth in our industry that as a system grows, it must inevitably transform into a sprawling, event-driven mesh of microservices. We are told that monoliths—or even well-structured stateful systems—are debt that must be paid off with a 'big-bang' rewrite to Kafka, Pub/Sub, or a web of decoupled Cloud Functions. I’ve seen this script play out time and again. A team manages 10,000 concurrent users on a direct request-response pattern using Firestore and simple triggers. When they hit 50,000, they panic. They perceive the latency spikes as a fundamental failure of their data model. They start drafting architecture diagrams filled with event buses and asynchronous choreographies.

I am here to tell you that this is the path to technical bankruptcy. My team and I scaled our real-time comments and reactions platform from 10,000 to 2,000,000 concurrent users without once performing a total architectural overhaul. We kept our core data structures, we kept our Firestore-centric logic, and we resisted the urge to 'event-source' our way out of load issues. Scaling is an engineering discipline, not a reason to discard perfectly functional business logic. The push toward pure event-driven architecture often creates a distributed monolith of hidden state, debugging nightmares, and unnecessary operational overhead.

The Bottleneck: When Hot Documents Are Misunderstood

The primary culprit for the 'scale-up panic' is almost always the Firestore 1-write-per-second-per-document limit. When you have a viral news article in our media stack, the reactions count—the 'like' button—becomes a hotspot. If you follow the traditional event-driven advice, you might be tempted to fire a Cloud Function for every single click, pushing that event into a message queue to be processed later. You trade the Firestore write limit for a system of queues, retry policies, dead-letter topics, and delayed consistency.

What happens when that queue backs up? What happens when you have to debug why a user's like didn't increment the count, only to find the message stuck in a partition that nobody is monitoring? Instead of shifting to an event-driven system, we stayed within the bounds of Firestore by implementing client-side 'sharding' of the counter. We didn't change the data model; we changed how we interacted with the document. By creating a sub-collection of 'shards' and periodically aggregating them, we bypassed the 1-write-per-second limit without introducing the complexity of an external message broker. We kept the state where it lived, solved the constraint, and moved on. The system remained consistent, observable, and, most importantly, debuggable.

Incremental Scaling: The Power of Targeted Optimization

When you reach 100,000 concurrent users, you will see your Cloud Function execution times climb. This is usually due to resource contention or inefficient fan-out patterns. A common mistake is to attempt a complete decoupling of the system using pub/sub topics for every user interaction. This leads to 'event explosion,' where you lose visibility into the request path.

Instead of rewriting, look at the granular constraints. Is your Cloud Function waiting on too many downstream network calls? Are you initializing your database connections inside the execution scope? In our environment, we reduced latency by over 60% simply by moving to global persistent connection pools and optimizing our Firestore document reads through index refinement rather than moving to a different messaging infrastructure.

Consider this Kotlin example for a high-throughput update pattern that avoids the 'event-everything' trap:

// Optimized counter increment using sharded sub-collection
fun incrementReactionCount(documentId: String, shardCount: Int) {
    val shardId = (0 until shardCount).random()
    val shardRef = db.collection("posts")
        .document(documentId)
        .collection("shards")
        .document(shardId.toString())

    db.runTransaction { transaction ->
        val snapshot = transaction.get(shardRef)
        val count = snapshot.getLong("count") ?: 0
        transaction.update(shardRef, "count", count + 1)
    }
}

By staying within the transaction boundaries of Firestore, we ensure consistency. When you use asynchronous events for simple operations, you are choosing 'eventual consistency' as a default, and you will eventually pay the price in support tickets and data reconciliation scripts.

The Operational Cost of Distributed Choreography

Event-driven architectures often introduce 'hidden state.' When you have five different Cloud Functions reacting to a single event, tracing a request flow becomes an exercise in archaeology. In our 2-million-user system, we prioritize synchronous visibility. If a user comments, the Firestore trigger fires, updates the document, and the client listens to the real-time stream. It’s direct, it’s fast, and it’s observable.

If we had opted for an event-driven rewrite, we would have needed a full tracing suite just to understand the state of a single user action. We would have needed to manage the throughput limits of Pub/Sub, the costs of cross-service invocation, and the nightmare of partial failures. By remaining within the Firestore ecosystem, our 'distributed' state is managed by the cloud provider’s highly available infrastructure. We aren't responsible for partitioning messages or managing queue lag; we are responsible for the business logic. That is the leverage you want as a staff engineer. Don't fight the platform; work within its guardrails.

When to Actually Evolve (and When to Stop)

There is a time for architectural evolution, but it should be driven by functional requirements, not by the number of concurrent users. If you need to perform complex data analysis across millions of rows, or if you need to integrate disparate services that weren't built in the same environment, then and only then do you introduce an event bus. But never migrate your 'hot' path to an event-driven model just to gain throughput. You will almost always find that your existing stack has knobs and levers you haven't turned yet.

To manage our scale, we moved to Firestore’s TTL policies for ephemeral data and utilized Cloud Tasks for heavy-duty background jobs—but we kept the real-time path as direct as possible. We didn't change our schema definition (in YAML format, for example) to accommodate an event-first world:

# Keeping our schema consistent while scaling horizontally
firestore_config:
  retries: 5
  backoff_strategy: exponential
  shard_limit: 20
  cache_ttl_seconds: 300
  # We prefer local state management over distributed events
  persistence: true

Every time you introduce a new asynchronous layer, you add a 'tax' to your system. That tax is paid in debugging time, monitoring complexity, and the constant fear of race conditions. Scaling from 10k to 2M users taught me that the best architecture is the one that stays boring while the load increases. If you can handle the load by adjusting your concurrency limits, sharding your data, or refining your indexes, do that. Don't fall for the hype of 'infinite scalability' through microservice fragmentation. The most scalable system is the one that minimizes the number of moving parts between the user's intent and the state transition.

Keep your data close to the compute, keep your state centralized where it makes sense, and treat 'scaling' as an operation, not a rewrite. Your future self—who has to debug this system at 3 AM—will thank you for it. We reached 2M concurrents by being disciplined with what we had, not by chasing the next shiny architecture pattern. Success is found in the refinement of the existing, not in the destruction of the functional.