LLM routing: selecting the right model per query at runtime

The Throughput Paradox of Generative AI

In our Hiroshima startup, we live and breathe data ingestion. When we started integrating LLMs into our microservices to perform real-time entity extraction and classification, we hit a wall. Using GPT-4o for every single incoming packet was not just expensive; it was slow. We were dealing with sub-100ms latency budgets for our ingestion pipeline, and waiting for a general-purpose frontier model to parse simple metadata was killing our throughput metrics. We weren't just paying for intelligence we didn't need; we were waiting for it.

Our engineering team was already running on Bun for our ingestion hot-path because the runtime overhead of Node.js was limiting our concurrency levels. When we decided to introduce LLM routing, we treated it as an extension of that performance-first philosophy. If our runtime could handle 50,000 requests per second (RPS) per container, but our router added a 300ms bottleneck, the optimization was moot. We needed a strategy that was as lightweight as the ingestion service itself.

Identifying the Routing Bottleneck

Routing isn't just about selecting a model; it's about the overhead of that decision. If the decision-making logic takes longer than the actual API call to a smaller model, you’ve lost. Initially, we looked at massive classification models that would analyze the prompt and categorize it. That was the mistake. Using an LLM to decide which LLM to use is a classic case of startup engineering overkill.

Instead, we opted for a tiered heuristic-based router. We categorize our inbound queries into three buckets: "Trivial" (e.g., regex-compatible, simple JSON mapping), "Standard" (e.g., entity extraction requiring context), and "Complex" (e.g., abstractive summarization and semantic reasoning). The router itself is a synchronous, non-blocking check that runs entirely in memory. Because we are using Bun, we leverage Bun.hash() and Map objects to keep our routing lookups near O(1).

Designing the Routing Architecture

Our routing layer is deployed as a sidecar or an internal service module, depending on the latency requirements of the specific ingestion path. By keeping the decision logic within the same runtime environment, we eliminate the need for extra network hops.

We define our routing rules in a strict YAML configuration that gets hot-reloaded into memory. This allows our data scientists to adjust thresholds without a full redeploy of the high-throughput ingestion microservice. Here is an example of how we structure our routing rules to balance speed vs. intelligence:

# routing_config.yaml
routes:
  - rule: "length < 500"
    model: "gemini-flash-1.5"
    priority: 1
  - rule: "sentiment_analysis_required"
    model: "gpt-4o-mini"
    priority: 2
  - rule: "default"
    model: "gpt-4o"
    priority: 3

cache_enabled: true
fallback_retries: 2

This configuration is parsed at startup and converted into an optimized lookup table in TypeScript. By keeping this logic inside the Bun process, we ensure that the decision-making happens in microseconds rather than milliseconds, protecting the overall ingestion throughput.

Implementing the Router in Bun

When implementing the router, we had to be incredibly careful about not introducing blocking I/O. Using fetch in Node.js for parallel requests often leads to event loop starvation if not handled with care. Bun’s implementation of fetch is natively built into the runtime, offering a significant reduction in system call overhead when we send requests out to our model providers.

Here is a simplified view of our routing logic, optimized for high concurrency:

import { RouterConfig } from './types';

export async function routeQuery(payload: string, config: RouterConfig) {
  const length = payload.length;
  
  // O(1) decision logic
  if (length < config.thresholds.small) {
    return await executeRequest(payload, 'provider-fast');
  }
  
  // Complex logic utilizes the faster runtime fetch
  return await executeRequest(payload, 'provider-smart');
}

async function executeRequest(data: string, target: string) {
  const response = await fetch(`https://api.model-provider.com/v1/chat`, {
    method: 'POST',
    body: JSON.stringify({ data }),
    headers: { 'Content-Type': 'application/json' }
  });
  
  return await response.json();
}

One critical observation: because Bun handles I/O via a more efficient multi-threaded event loop architecture, we can initiate multiple concurrent model calls and await their results with significantly less memory pressure than we observed in our previous Node.js setup. This is vital when the ingestion volume spikes.

Handling the Compatibility and Latency Gap

Transitioning to a dynamic routing model means you are constantly dealing with varying latency profiles. A "Trivial" query returns in 40ms, while a "Complex" query might take 1.2s. This variability can create backpressure in a high-throughput microservice. We solved this by implementing a circuit breaker and a local sliding-window rate limiter for each model tier.

If the high-end model starts slowing down (a common occurrence during peak provider demand), the router automatically shifts the traffic to a secondary, slightly less capable model. This "graceful degradation" ensures that our throughput stays consistent even when the underlying LLM infrastructure is under strain. This is a pragmatic necessity in a startup: we prioritize the ingestion rate and system availability over achieving perfect model output for every single request.

Measurement: The Proof of the Pudding

We measured the impact of this routing layer using a canary deployment across three clusters. The metrics were undeniable. Before the routing implementation, our ingestion service would hit a performance ceiling at 12,000 requests per minute (RPM) due to the serialization latency of large LLM responses. After implementing the routing logic, we saw a 45% reduction in total average latency across all ingestion endpoints.

More importantly, our egress costs dropped by 62% because the vast majority of our queries were routed to cheaper, smaller models that were more than sufficient for the tasks required. We didn't sacrifice accuracy; we simply matched the complexity of the query to the capacity of the model.

Visualizing the Performance Impact

To better understand how this improves our system, consider the standard lifecycle of an ingestion request. In our old architecture, every request went through a standard pipeline:

1. Receive request -> Validate -> Normalize -> Call GPT-4 -> Respond.

In our new architecture, the pipeline looks like this:

1. Receive request -> Validate -> Route (Select Model) -> Normalize -> Call Selected Model -> Respond.

While the addition of the 'Route' step theoretically adds a step, the efficiency gained by selecting a model with lower overhead/faster inference speed results in a net positive gain. We visualize this in our Grafana dashboards by tracking the "Model Inference Delta," which represents the difference between the time a request is received and the time the specific model finishes its inference. By narrowing the variance in this delta, we have stabilized our entire infrastructure.

Challenges and Future Proofing

Nothing is perfect, and routing comes with its own technical debt. The biggest challenge has been maintaining the parity of prompt templates across different models. A prompt that works perfectly on gpt-4o might fail or behave erratically on gemini-flash. We’ve had to implement an abstraction layer for prompt engineering that ensures inputs are consistently formatted regardless of the target model.

Furthermore, as providers introduce newer models, the static YAML configuration approach needs to be more dynamic. We are currently experimenting with an autonomous routing mechanism that uses a tiny, local-only model (running as a WebAssembly module inside Bun) to predict the optimal model route based on historical latency data. This would push our routing capability even further, moving from heuristic rules to adaptive, runtime-learned routing.

Conclusion

In conclusion, LLM routing is not just a clever way to save on API bills—it is a mandatory architectural pattern for any high-throughput microservice dealing with variable query complexity. By offloading the routing decision to a high-performance runtime like Bun, we managed to maintain the responsiveness of our ingestion pipeline while effectively scaling our intelligence capacity.

Engineering is often about making trade-offs between performance and capability. By building a routing layer that understands the difference between a query that needs a genius and a query that just needs a clerk, we were able to keep our Hiroshima startup's ingestion engine humming at top speed. If you are struggling with latency spikes in your LLM-integrated microservices, stop trying to optimize the model calls alone. Look at your routing logic, switch to a more efficient runtime, and stop over-processing the trivial requests. The path to throughput excellence is not more compute; it's smarter distribution.