Streaming LLM responses to the browser with Server-Sent Events
Why SSE is the right choice for LLM streaming
WebSockets are the standard answer for real-time browser communication, but they are bidirectional. For LLM streaming, the communication is fundamentally one-directional: the user sends a message, and the server streams back a response. You do not need a persistent bidirectional connection.
Server-Sent Events (SSE) are an HTTP-based push mechanism built for exactly this pattern. The browser opens a standard HTTP request with Accept: text/event-stream, and the server holds the connection open, writing data frames as they become available. When the stream ends, the connection closes. If it disconnects unexpectedly, the browser automatically reconnects.
Compared to WebSockets, SSE is simpler to implement (it works over plain HTTP without protocol upgrade), simpler to proxy (it goes through standard HTTP infrastructure including Cloudflare and Vercel edge networks), and simpler to debug (you can inspect it in the Network tab like any HTTP request). The tradeoff is that SSE is unidirectional — but for LLM response streaming, that is exactly what you need.
The SSE wire format
The SSE format is a text stream where each event is one or more field: value lines followed by a blank line. The key field for sending data is data:.
data: {"token": "Hello"}\n\n
data: {"token": " world"}\n\n
data: [DONE]\n\n
The browser's EventSource API fires a message event for each data: line. The special data: [DONE] sentinel signals stream completion. This is the convention used by the OpenAI API and most LLM APIs, which makes it familiar and easy to handle consistently.
You can also name events for different message types:
event: token\ndata: {"text": "Hello"}\n\n
event: metadata\ndata: {"model": "claude-opus-4-5", "inputTokens": 12}\n\n
event: done\ndata: {}\n\n
Named events let the client react differently to different stream payloads — useful for sending metadata or error information alongside tokens.
Building the Next.js Route Handler
The Route Handler is the server side of the streaming implementation. It receives the user's message, calls the LLM API with streaming enabled, and forwards each token as an SSE frame.
// app/api/chat/route.ts
import Anthropic from "@anthropic-ai/sdk";
import { NextRequest } from "next/server";
const client = new Anthropic();
export async function POST(req: NextRequest) {
const { messages } = await req.json() as {
messages: Array<{ role: "user" | "assistant"; content: string }>;
};
if (!messages || messages.length === 0) {
return new Response("Missing messages", { status: 400 });
}
const encoder = new TextEncoder();
const stream = new ReadableStream({
async start(controller) {
const send = (data: string) => {
controller.enqueue(encoder.encode(`data: ${data}\n\n`));
};
try {
// Create a streaming message from Anthropic
const anthropicStream = await client.messages.stream({
model: "claude-opus-4-5",
max_tokens: 1024,
messages,
});
// Forward each text delta as an SSE frame
for await (const event of anthropicStream) {
if (
event.type === "content_block_delta" &&
event.delta.type === "text_delta"
) {
send(JSON.stringify({ token: event.delta.text }));
}
}
// Get final message for usage statistics
const finalMessage = await anthropicStream.finalMessage();
send(
JSON.stringify({
done: true,
usage: {
inputTokens: finalMessage.usage.input_tokens,
outputTokens: finalMessage.usage.output_tokens,
},
})
);
send("[DONE]");
} catch (err) {
send(
JSON.stringify({
error: err instanceof Error ? err.message : "Stream error",
})
);
} finally {
controller.close();
}
},
});
return new Response(stream, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
Connection: "keep-alive",
// Required for Cloudflare and some proxies to not buffer the stream
"X-Accel-Buffering": "no",
},
});
}
The X-Accel-Buffering: no header is important for deployments behind nginx or Cloudflare — without it, the proxy may buffer the entire response before forwarding it, defeating the purpose of streaming.
Reading the stream in a React component
The browser side uses the Fetch API with a ReadableStream reader rather than EventSource. This gives you more control over request headers (including Authorization) that EventSource cannot set.
// hooks/useChat.ts
import { useState, useCallback } from "react";
interface Message {
role: "user" | "assistant";
content: string;
}
interface UseChatResult {
messages: Message[];
isStreaming: boolean;
error: string | null;
sendMessage: (content: string) => Promise<void>;
}
export function useChat(): UseChatResult {
const [messages, setMessages] = useState<Message[]>([]);
const [isStreaming, setIsStreaming] = useState(false);
const [error, setError] = useState<string | null>(null);
const sendMessage = useCallback(
async (content: string) => {
setError(null);
const userMessage: Message = { role: "user", content };
const updatedMessages = [...messages, userMessage];
setMessages(updatedMessages);
setIsStreaming(true);
// Add an empty assistant message that we will fill as tokens arrive
setMessages((prev) => [...prev, { role: "assistant", content: "" }]);
try {
const response = await fetch("/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ messages: updatedMessages }),
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const reader = response.body!.getReader();
const decoder = new TextDecoder();
let buffer = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
buffer += decoder.decode(value, { stream: true });
const lines = buffer.split("\n");
buffer = lines.pop() ?? ""; // Keep incomplete line in buffer
for (const line of lines) {
if (!line.startsWith("data: ")) continue;
const payload = line.slice(6).trim();
if (payload === "[DONE]") break;
try {
const parsed = JSON.parse(payload) as {
token?: string;
done?: boolean;
error?: string;
};
if (parsed.error) {
throw new Error(parsed.error);
}
if (parsed.token) {
// Append token to the last assistant message
setMessages((prev) => {
const updated = [...prev];
const last = updated[updated.length - 1];
if (last.role === "assistant") {
updated[updated.length - 1] = {
...last,
content: last.content + parsed.token,
};
}
return updated;
});
}
} catch (parseErr) {
console.warn("Failed to parse SSE line:", line, parseErr);
}
}
}
} catch (err) {
setError(err instanceof Error ? err.message : "Unknown error");
// Remove the incomplete assistant message on error
setMessages((prev) => prev.slice(0, -1));
} finally {
setIsStreaming(false);
}
},
[messages]
);
return { messages, isStreaming, error, sendMessage };
}
The key implementation detail is handling the SSE buffer correctly. The ReadableStream reader delivers data in chunks that do not necessarily align with SSE line boundaries. The buffer accumulates raw text and processes complete lines, keeping any incomplete line for the next chunk. Skipping this buffer management leads to parsing errors on busy streams.
Building the chat UI component
// components/Chat.tsx
"use client";
import { useRef, useEffect, FormEvent, useState } from "react";
import { useChat } from "@/hooks/useChat";
export function Chat() {
const { messages, isStreaming, error, sendMessage } = useChat();
const [input, setInput] = useState("");
const bottomRef = useRef<HTMLDivElement>(null);
// Auto-scroll to bottom as new tokens arrive
useEffect(() => {
bottomRef.current?.scrollIntoView({ behavior: "smooth" });
}, [messages]);
async function handleSubmit(e: FormEvent) {
e.preventDefault();
const content = input.trim();
if (!content || isStreaming) return;
setInput("");
await sendMessage(content);
}
return (
<div className="flex flex-col h-screen max-w-2xl mx-auto p-4">
<div className="flex-1 overflow-y-auto space-y-4 pb-4">
{messages.map((msg, i) => (
<div
key={i}
className={`flex ${msg.role === "user" ? "justify-end" : "justify-start"}`}
>
<div
className={`max-w-[80%] rounded-lg px-4 py-2 ${
msg.role === "user"
? "bg-blue-600 text-white"
: "bg-gray-100 text-gray-900"
}`}
>
<p className="whitespace-pre-wrap">{msg.content}</p>
{/* Streaming cursor on the last assistant message */}
{isStreaming &&
i === messages.length - 1 &&
msg.role === "assistant" && (
<span className="inline-block w-2 h-4 bg-gray-500 animate-pulse ml-0.5 align-middle" />
)}
</div>
</div>
))}
{error && (
<div className="bg-red-50 border border-red-200 text-red-700 rounded-lg p-3">
Error: {error}
</div>
)}
<div ref={bottomRef} />
</div>
<form onSubmit={handleSubmit} className="flex gap-2 pt-4 border-t">
<input
type="text"
value={input}
onChange={(e) => setInput(e.target.value)}
placeholder="Type a message..."
disabled={isStreaming}
className="flex-1 rounded-lg border px-4 py-2 focus:outline-none focus:ring-2 focus:ring-blue-500 disabled:opacity-50"
/>
<button
type="submit"
disabled={isStreaming || !input.trim()}
className="rounded-lg bg-blue-600 text-white px-4 py-2 disabled:opacity-50 hover:bg-blue-700 transition-colors"
>
{isStreaming ? "Streaming..." : "Send"}
</button>
</form>
</div>
);
}
Handling abort and cleanup
Users stop mid-stream. When the component unmounts or the user navigates away, the ongoing fetch should be cancelled to avoid leaking connections and unnecessary LLM tokens.
// Updated useChat hook with abort support
export function useChat(): UseChatResult & { abort: () => void } {
const [messages, setMessages] = useState<Message[]>([]);
const [isStreaming, setIsStreaming] = useState(false);
const [error, setError] = useState<string | null>(null);
const abortControllerRef = useRef<AbortController | null>(null);
const abort = useCallback(() => {
abortControllerRef.current?.abort();
setIsStreaming(false);
}, []);
// Clean up on unmount
useEffect(() => {
return () => {
abortControllerRef.current?.abort();
};
}, []);
const sendMessage = useCallback(
async (content: string) => {
setError(null);
// Cancel any in-progress stream
abortControllerRef.current?.abort();
const controller = new AbortController();
abortControllerRef.current = controller;
const userMessage: Message = { role: "user", content };
const updatedMessages = [...messages, userMessage];
setMessages(updatedMessages);
setIsStreaming(true);
setMessages((prev) => [...prev, { role: "assistant", content: "" }]);
try {
const response = await fetch("/api/chat", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ messages: updatedMessages }),
signal: controller.signal, // Link the abort signal
});
// ... rest of stream reading logic
} catch (err) {
if ((err as Error).name === "AbortError") {
// User aborted — not an error worth showing
return;
}
setError(err instanceof Error ? err.message : "Unknown error");
setMessages((prev) => prev.slice(0, -1));
} finally {
setIsStreaming(false);
}
},
[messages]
);
return { messages, isStreaming, error, sendMessage, abort };
}
On the server side, Next.js Route Handlers detect client disconnection through the request's signal. When the client aborts, the for await loop over the Anthropic stream will throw an AbortError, which is caught and handled gracefully.
Streaming with tool use
When your LLM uses tools, the streaming gets more complex. The model may produce text, then request tool calls, then produce more text after receiving the tool results. The stream needs to handle all these event types.
// app/api/chat-with-tools/route.ts
export async function POST(req: NextRequest) {
const { messages } = await req.json();
const encoder = new TextEncoder();
const stream = new ReadableStream({
async start(controller) {
const send = (data: object) => {
controller.enqueue(
encoder.encode(`data: ${JSON.stringify(data)}\n\n`)
);
};
const conversationMessages = [...messages];
try {
// Agentic loop — may take multiple turns
for (let turn = 0; turn < 10; turn++) {
const anthropicStream = await client.messages.stream({
model: "claude-opus-4-5",
max_tokens: 1024,
tools: yourTools,
messages: conversationMessages,
});
let hasToolCalls = false;
const toolCallsInProgress: Record<string, { name: string; inputBuffer: string }> = {};
for await (const event of anthropicStream) {
if (
event.type === "content_block_delta" &&
event.delta.type === "text_delta"
) {
send({ type: "token", text: event.delta.text });
} else if (
event.type === "content_block_start" &&
event.content_block.type === "tool_use"
) {
hasToolCalls = true;
const block = event.content_block;
toolCallsInProgress[block.id] = { name: block.name, inputBuffer: "" };
send({ type: "tool_start", toolName: block.name });
} else if (
event.type === "content_block_delta" &&
event.delta.type === "input_json_delta"
) {
// Find which tool call this delta belongs to
const toolId = Object.keys(toolCallsInProgress).at(-1);
if (toolId) {
toolCallsInProgress[toolId].inputBuffer += event.delta.partial_json;
}
}
}
const finalMessage = await anthropicStream.finalMessage();
if (!hasToolCalls) {
send({ type: "done" });
break;
}
// Execute tool calls
conversationMessages.push({ role: "assistant", content: finalMessage.content });
const toolResults = await Promise.all(
finalMessage.content
.filter((b): b is Anthropic.ToolUseBlock => b.type === "tool_use")
.map(async (toolCall) => {
send({ type: "tool_executing", toolName: toolCall.name });
const result = await executeTool(toolCall.name, toolCall.input);
send({ type: "tool_done", toolName: toolCall.name });
return {
type: "tool_result" as const,
tool_use_id: toolCall.id,
content: JSON.stringify(result),
};
})
);
conversationMessages.push({ role: "user", content: toolResults });
}
} catch (err) {
send({ type: "error", message: err instanceof Error ? err.message : "Stream error" });
} finally {
controller.enqueue(encoder.encode("data: [DONE]\n\n"));
controller.close();
}
},
});
return new Response(stream, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"X-Accel-Buffering": "no",
},
});
}
The client receives typed events — token, tool_start, tool_executing, tool_done, done, error — and can render them differently. A common pattern is to show a progress indicator ("Searching knowledge base...") while tools are executing, then switch back to streaming text when the model resumes generating its response.
Edge deployment considerations
Deploying streaming Route Handlers to Vercel Edge or Cloudflare Pages requires the runtime directive:
// At the top of your route.ts file
export const runtime = "edge";
export const dynamic = "force-dynamic";
Edge runtimes support the Web Streams API natively, but they have constraints: no Node.js built-ins, 25-second maximum duration on Vercel's Edge, and 30-second CPU time limits on Cloudflare Workers.
For long-running LLM responses, these limits can be tight. Strategies for working within them include using smaller models with lower latency, setting max_tokens conservatively, and breaking long tasks into shorter independent requests. For tasks that genuinely need more time, run the LLM in a standard Node.js serverless function (with up to 5 minutes of execution time on Vercel) rather than an edge function.
Streaming LLM responses to the browser is now a table-stakes feature for any conversational AI product. The SSE approach is simple, reliable across all HTTP infrastructure, and requires no special client libraries — just the Fetch API that every browser has natively supported for years.