Scaling AI Writing Assistants: Optimizing Throughput with Bun and TypeScript

The Throughput Challenge: Engineering for Generative AI

At our Hiroshima-based startup, building an AI writing assistant isn't just about prompt engineering; it is a battle against system overhead. When you are processing thousands of concurrent text transformation requests, the choice of runtime becomes a critical throughput bottleneck. We recently transitioned our core writing assistant engine from a traditional Node.js environment to Bun, resulting in a dramatic reduction in request latency and a significant increase in concurrent throughput.

For those new to the field, an AI writing assistant acts as a bridge between a user’s draft and an LLM, injecting style and tone controls via structured prompt augmentation. While the heavy lifting happens in the LLM, the orchestrator service—the piece that handles authentication, prompt construction, and result streaming—often becomes the bottleneck due to its I/O-heavy nature. In Node.js, the overhead of the event loop and the way it handles incoming HTTP streams under heavy load meant we were frequently hitting limits that had nothing to do with the model's response time, but rather our infrastructure's inability to queue connections effectively.

Identifying the Node.js Bottleneck in the Hot Path

Our initial architecture utilized Express.js on Node.js. While reliable, we noticed that under high traffic spikes, the event loop latency increased disproportionately. When an AI writing assistant operates, it is essentially a high-concurrency pipe. You receive a block of text, you decorate it with system instructions regarding tone (e.g., 'professional', 'casual', 'empathetic'), and you pipe it to an external inference API.

Node.js, while excellent, introduces non-trivial overhead when managing thousands of open TCP connections. We found that the internal buffering mechanisms in our Node.js middleware were consuming memory and CPU cycles that could have been better spent on JSON parsing and payload management. When we profiled the system, the 'hot path'—the path from receiving the POST request to dispatching the request to the LLM backend—was consistently taking 40-60ms in overhead, excluding LLM inference time. In a system where we aimed for sub-200ms TTFT (Time To First Token), losing 60ms to runtime overhead was unacceptable. We needed a runtime that offered a more efficient I/O model and a native approach to HTTP servers.

Migrating to Bun: Why the Runtime Matters

Bun represents a paradigm shift for TypeScript-based microservices. It is not merely a faster Node.js; it is a fundamental re-implementation of the JavaScript runtime, specifically built with modern hardware and high-throughput networking in mind. Its native implementation of the Fetch API and the HTTP server meant that we could eliminate Express.js entirely.

When we migrated, the primary goal was to replace the standard library overhead with Bun’s built-in, highly optimized primitives. By using Bun.serve, we shifted the burden of HTTP connection management from the application layer to the runtime layer, which is written in C++ and Zig. This change alone allowed us to handle roughly double the concurrent requests per container compared to our old setup, all without touching the business logic surrounding the style and tone control algorithms. Below is a simplified look at how we implemented the high-throughput server:

# docker-compose.yml configuration for Bun service
version: '3.8'
services:
  writing-assistant:
    image: oven/bun:latest
    command: ["bun", "run", "src/server.ts"]
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=production
      - LLM_API_KEY=${LLM_API_KEY}
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 1G

This simple migration to Bun meant that our TypeScript services could leverage optimized memory management and faster cold starts. For a startup, cold starts are not just a technical vanity metric; they directly affect our ability to scale out during traffic surges. When our monitoring alerts spike, we can spin up new instances in milliseconds rather than seconds.

Managing Compatibility Gaps and Type Safety

Migrating to Bun is usually straightforward, but the 'compatibility layer' is where the real work happens. Because our system relies on various internal packages that were heavily tied to Node.js APIs (like fs or path), we had to ensure that our transition didn't break our build process. Fortunately, Bun’s support for Node.js compatibility is robust. We didn't have to rewrite our utility functions, but we did have to be mindful of how we imported specific low-level modules.

For an AI writing assistant, type safety is paramount. When injecting tone controls, we use Zod for schema validation. Using Bun with TypeScript meant we could leverage bun build to bundle our services into single files, significantly reducing the node_modules overhead in our deployment artifacts. This results in cleaner CI/CD pipelines and faster rollouts. Here is an example of how we structure our tone control handler with TypeScript:

// Writing assistant logic using TypeScript and Zod
import { z } from "zod";

const ToneSchema = z.object({
  originalText: z.string().min(1),
  tone: z.enum(["professional", "casual", "witty", "empathetic"]),
  maxLength: z.number().optional()
});

export const transformPrompt = (input: unknown) => {
  const data = ToneSchema.parse(input);
  const styleInstruction = `Rewrite the following text in a ${data.tone} tone.`;
  return `${styleInstruction}\n\nText: ${data.originalText}`;
};

The ability to bundle these types directly into the runtime environment means our microservice is extremely lean. The startup overhead—the time it takes for our service to be 'ready'—dropped by 75% after the migration. This is critical for our autoscaling group in Hiroshima, where we often run on smaller, spot-instance clusters to keep costs low.

The Results: Metrics and Future Outlook

After moving our hot-path ingestion service to Bun, the results were conclusive. We achieved a 2x improvement in throughput. Our latency overhead dropped from the previous 40-60ms range to under 15ms. This 45ms gain is essentially 'free' performance; we didn't change the algorithm that determines how text is rewritten, nor did we change the underlying LLM provider. We simply removed the runtime tax.

Pragmatism is the key to startup engineering. We didn't migrate to Bun because it was the 'newest' tool; we migrated because our throughput requirements were outpacing our architectural ability to handle them. The lesson here for developers building AI-based tools is simple: do not assume that slow performance is a function of your LLM calls. Often, the bottleneck is closer to home.

Looking ahead, we are exploring Bun’s native Bun.file and advanced stream manipulation to handle long-form document processing. Since we are already running on Bun, the transition to these lower-level APIs is seamless. By stripping away unnecessary layers of abstraction, we allow our TypeScript code to do what it does best: orchestrate data efficiently. We will continue to scale our writing assistant by focusing on the runtime layer, ensuring that our AI features stay responsive regardless of how high our traffic scales. The move to Bun was not just a performance upgrade—it was a strategic decision that gave us the headroom to innovate on the feature set while keeping our infrastructure lean and mean. Moving forward, the focus remains on throughput, type safety, and keeping our deployment footprint as small as possible.