Building a document Q&A system with Firestore and Claude

Architecture overview

A document Q&A system has four distinct phases: ingestion, storage, retrieval, and generation. Each phase has independent failure modes and tuning parameters, so keeping them cleanly separated in the architecture makes the system easier to debug and improve independently.

The architecture we will build uses:

• Cloud Storage for raw document files (PDF, DOCX, TXT)
• Cloud Functions for document processing triggered by Storage events
• Firestore for document metadata, extracted text chunks, and their embeddings
• Voyage AI (via the embeddings API) for generating text embeddings
• Claude (via the Anthropic API) for answer generation with streamed responses
• Next.js for the frontend with a Route Handler that streams Claude's response to the browser

This architecture fits comfortably within Firebase's free tier for small document collections and scales to millions of chunks on the pay-as-you-go plan.

Firestore data model

Design the Firestore schema before writing any processing code. The schema drives the queries, and Firestore's query model is less flexible than SQL—you need to know your access patterns upfront.

/documents/{documentId}
  id: string
  userId: string
  filename: string
  storagePath: string
  status: "processing" | "ready" | "error"
  chunkCount: number
  createdAt: Timestamp
  updatedAt: Timestamp
  errorMessage?: string

/documents/{documentId}/chunks/{chunkId}
  id: string
  documentId: string
  userId: string
  text: string
  embedding: array<number>   // stored as Firestore array field
  chunkIndex: number
  tokenCount: number
  createdAt: Timestamp

/conversations/{conversationId}
  userId: string
  documentIds: array<string>  // documents in scope for this conversation
  createdAt: Timestamp
  updatedAt: Timestamp

/conversations/{conversationId}/messages/{messageId}
  role: "user" | "assistant"
  content: string
  retrievedChunkIds?: array<string>  // for traceability
  createdAt: Timestamp

The chunks subcollection stores embeddings as plain number arrays. Firestore does not have a native vector type or ANN index (unlike pgvector), so similarity search must be done in application code after fetching candidates. This means Firestore is appropriate for corpora up to tens of thousands of chunks—beyond that, replace the embedding storage and retrieval with a dedicated vector database while keeping Firestore for metadata and conversation state.

Document ingestion with Cloud Functions

When a user uploads a file to Cloud Storage, a Cloud Function handles text extraction, chunking, and embedding generation.

// functions/src/processDocument.ts
import * as functions from "firebase-functions/v2";
import * as admin from "firebase-admin";
import Anthropic from "@anthropic-ai/sdk";
import voyageai from "voyageai";

const db = admin.firestore();
const storage = admin.storage();
const voyage = new voyageai.Client({ apiKey: process.env.VOYAGE_API_KEY });

export const processDocument = functions.storage.onObjectFinalized(
  { memory: "1GiB", timeoutSeconds: 540 },
  async (event) => {
    const filePath = event.data.name;
    if (!filePath?.startsWith("documents/")) return;

    // Extract documentId from path: documents/{userId}/{documentId}/filename.pdf
    const parts = filePath.split("/");
    const userId = parts[1];
    const documentId = parts[2];

    const docRef = db.collection("documents").doc(documentId);

    try {
      await docRef.update({ status: "processing", updatedAt: admin.firestore.FieldValue.serverTimestamp() });

      // Download the file
      const bucket = storage.bucket(event.data.bucket);
      const [fileBuffer] = await bucket.file(filePath).download();

      // Extract text based on file type
      const text = await extractText(fileBuffer, event.data.contentType ?? "");

      // Chunk the text
      const chunks = chunkText(text, { maxTokens: 512, overlapTokens: 64 });

      // Generate embeddings in batches of 96 (Voyage AI limit)
      const batchSize = 96;
      const chunkTexts = chunks.map((c) => c.text);
      const allEmbeddings: number[][] = [];

      for (let i = 0; i < chunkTexts.length; i += batchSize) {
        const batch = chunkTexts.slice(i, i + batchSize);
        const result = await voyage.embed({
          input: batch,
          model: "voyage-3",
          inputType: "document",
        });
        allEmbeddings.push(...result.embeddings);
      }

      // Write chunks to Firestore in batches of 500
      const chunksRef = docRef.collection("chunks");
      const writeBatchSize = 500;

      for (let i = 0; i < chunks.length; i += writeBatchSize) {
        const batch = db.batch();
        for (let j = i; j < Math.min(i + writeBatchSize, chunks.length); j++) {
          const chunkRef = chunksRef.doc();
          batch.set(chunkRef, {
            id: chunkRef.id,
            documentId,
            userId,
            text: chunks[j].text,
            embedding: allEmbeddings[j],
            chunkIndex: j,
            tokenCount: chunks[j].tokenCount,
            createdAt: admin.firestore.FieldValue.serverTimestamp(),
          });
        }
        await batch.commit();
      }

      await docRef.update({
        status: "ready",
        chunkCount: chunks.length,
        updatedAt: admin.firestore.FieldValue.serverTimestamp(),
      });
    } catch (error) {
      console.error("Document processing failed:", error);
      await docRef.update({
        status: "error",
        errorMessage: error instanceof Error ? error.message : "Unknown error",
        updatedAt: admin.firestore.FieldValue.serverTimestamp(),
      });
    }
  },
);

The chunking function is one of the highest-leverage pieces of the entire system. Poor chunking degrades retrieval quality more than almost any other factor.

interface Chunk {
  text: string;
  tokenCount: number;
}

function chunkText(
  text: string,
  options: { maxTokens: number; overlapTokens: number },
): Chunk[] {
  const { maxTokens, overlapTokens } = options;

  // Approximate: 1 token ≈ 4 characters for English
  const charsPerToken = 4;
  const maxChars = maxTokens * charsPerToken;
  const overlapChars = overlapTokens * charsPerToken;

  // Split on paragraph boundaries first, then sentences
  const paragraphs = text.split(/\n\n+/);
  const chunks: Chunk[] = [];
  let currentChunk = "";

  for (const paragraph of paragraphs) {
    if ((currentChunk + "\n\n" + paragraph).length <= maxChars) {
      currentChunk = currentChunk ? currentChunk + "\n\n" + paragraph : paragraph;
    } else {
      if (currentChunk) {
        chunks.push({
          text: currentChunk.trim(),
          tokenCount: Math.ceil(currentChunk.length / charsPerToken),
        });
        // Carry over the overlap from the end of the current chunk
        const overlapText = currentChunk.slice(-overlapChars);
        currentChunk = overlapText + "\n\n" + paragraph;
      } else {
        // Single paragraph exceeds max — split by sentence
        const sentences = paragraph.split(/(?<=[.!?])\s+/);
        for (const sentence of sentences) {
          if ((currentChunk + " " + sentence).length <= maxChars) {
            currentChunk = currentChunk ? currentChunk + " " + sentence : sentence;
          } else {
            if (currentChunk) {
              chunks.push({
                text: currentChunk.trim(),
                tokenCount: Math.ceil(currentChunk.length / charsPerToken),
              });
            }
            currentChunk = sentence;
          }
        }
      }
    }
  }

  if (currentChunk.trim()) {
    chunks.push({
      text: currentChunk.trim(),
      tokenCount: Math.ceil(currentChunk.length / charsPerToken),
    });
  }

  return chunks;
}

Semantic retrieval from Firestore

Since Firestore lacks ANN indexing, retrieval works by fetching all chunks for a document and computing cosine similarity in memory. This is acceptable for documents up to several hundred chunks (~50–100 pages). For larger corpora, export embeddings to a vector database.

// lib/retrieval.ts
import * as admin from "firebase-admin";
import voyageai from "voyageai";

const db = admin.firestore();
const voyage = new voyageai.Client({ apiKey: process.env.VOYAGE_API_KEY });

function cosineSimilarity(a: number[], b: number[]): number {
  let dot = 0;
  let normA = 0;
  let normB = 0;
  for (let i = 0; i < a.length; i++) {
    dot += a[i] * b[i];
    normA += a[i] * a[i];
    normB += b[i] * b[i];
  }
  return dot / (Math.sqrt(normA) * Math.sqrt(normB) + 1e-8);
}

interface RetrievedChunk {
  id: string;
  text: string;
  score: number;
  documentId: string;
}

export async function retrieveRelevantChunks(
  query: string,
  documentIds: string[],
  topK: number = 5,
): Promise<RetrievedChunk[]> {
  // Embed the query
  const queryEmbedResult = await voyage.embed({
    input: [query],
    model: "voyage-3",
    inputType: "query",
  });
  const queryEmbedding = queryEmbedResult.embeddings[0];

  // Fetch all chunks for the relevant documents in parallel
  const chunkPromises = documentIds.map((docId) =>
    db.collection("documents").doc(docId).collection("chunks").get(),
  );
  const chunkSnapshots = await Promise.all(chunkPromises);

  // Score all chunks
  const scored: RetrievedChunk[] = [];
  for (const snapshot of chunkSnapshots) {
    for (const doc of snapshot.docs) {
      const data = doc.data();
      const embedding: number[] = data.embedding;
      const score = cosineSimilarity(queryEmbedding, embedding);
      scored.push({
        id: doc.id,
        text: data.text,
        score,
        documentId: data.documentId,
      });
    }
  }

  // Sort by score and return top K
  scored.sort((a, b) => b.score - a.score);
  return scored.slice(0, topK);
}

Streaming answer generation with Claude

The answer generation handler is a Next.js Route Handler that streams Claude's response back to the browser using Server-Sent Events. Streaming is important for perceived performance—users see the first tokens within 300–500ms rather than waiting for the complete response.

// app/api/ask/route.ts
import { NextRequest } from "next/server";
import Anthropic from "@anthropic-ai/sdk";
import { retrieveRelevantChunks } from "@/lib/retrieval";
import { db } from "@/lib/firebase-admin";
import { verifyAuthToken } from "@/lib/auth";

const anthropic = new Anthropic();

export async function POST(request: NextRequest) {
  const { token, question, documentIds, conversationId } = await request.json();

  // Verify the user owns the documents
  const userId = await verifyAuthToken(token);
  if (!userId) {
    return new Response("Unauthorized", { status: 401 });
  }

  // Retrieve relevant chunks
  const chunks = await retrieveRelevantChunks(question, documentIds, 5);

  if (chunks.length === 0) {
    return Response.json({ error: "No relevant content found" }, { status: 404 });
  }

  // Build the context string
  const context = chunks
    .map((c, i) => `[Source ${i + 1}]\n${c.text}`)
    .join("\n\n---\n\n");

  // Fetch conversation history for multi-turn support
  const messagesSnap = await db
    .collection("conversations")
    .doc(conversationId)
    .collection("messages")
    .orderBy("createdAt", "asc")
    .limit(10)
    .get();

  const history = messagesSnap.docs.map((doc) => ({
    role: doc.data().role as "user" | "assistant",
    content: doc.data().content as string,
  }));

  // Append the new question
  const messages = [
    ...history,
    { role: "user" as const, content: question },
  ];

  // Save the user message to Firestore
  const userMsgRef = db
    .collection("conversations")
    .doc(conversationId)
    .collection("messages")
    .doc();
  await userMsgRef.set({
    role: "user",
    content: question,
    retrievedChunkIds: chunks.map((c) => c.id),
    createdAt: new Date(),
  });

  // Stream the response
  const encoder = new TextEncoder();
  let fullResponse = "";

  const stream = new ReadableStream({
    async start(controller) {
      try {
        const claudeStream = await anthropic.messages.stream({
          model: "claude-sonnet-4-5",
          max_tokens: 2048,
          system: `You are a helpful assistant that answers questions about documents.
          
Answer the user's question using ONLY the information provided in the context below.
If the context does not contain enough information to answer the question, say so clearly.
Do not use knowledge from outside the provided context.
Cite sources by referring to [Source N] when using information from that source.

Context:
${context}`,
          messages,
        });

        for await (const chunk of claudeStream) {
          if (
            chunk.type === "content_block_delta" &&
            chunk.delta.type === "text_delta"
          ) {
            const text = chunk.delta.text;
            fullResponse += text;
            controller.enqueue(
              encoder.encode(`data: ${JSON.stringify({ text })}\n\n`),
            );
          }
        }

        // Save the complete assistant response
        const assistantMsgRef = db
          .collection("conversations")
          .doc(conversationId)
          .collection("messages")
          .doc();
        await assistantMsgRef.set({
          role: "assistant",
          content: fullResponse,
          createdAt: new Date(),
        });

        controller.enqueue(encoder.encode("data: [DONE]\n\n"));
        controller.close();
      } catch (error) {
        controller.error(error);
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      Connection: "keep-alive",
    },
  });
}

Frontend: streaming the response to the user

// components/QAInterface.tsx
"use client";
import { useState } from "react";

interface Message {
  role: "user" | "assistant";
  content: string;
}

export function QAInterface({
  documentIds,
  conversationId,
}: {
  documentIds: string[];
  conversationId: string;
}) {
  const [messages, setMessages] = useState<Message[]>([]);
  const [question, setQuestion] = useState("");
  const [isStreaming, setIsStreaming] = useState(false);

  async function handleSubmit(e: React.FormEvent) {
    e.preventDefault();
    if (!question.trim() || isStreaming) return;

    const userQuestion = question;
    setQuestion("");
    setMessages((prev) => [...prev, { role: "user", content: userQuestion }]);
    setIsStreaming(true);

    // Add empty assistant message that we'll fill in
    setMessages((prev) => [...prev, { role: "assistant", content: "" }]);

    try {
      const token = await getAuthToken(); // your auth implementation
      const response = await fetch("/api/ask", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({ token, question: userQuestion, documentIds, conversationId }),
      });

      if (!response.ok) throw new Error("Request failed");

      const reader = response.body!.getReader();
      const decoder = new TextDecoder();

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        const chunk = decoder.decode(value);
        const lines = chunk.split("\n");

        for (const line of lines) {
          if (!line.startsWith("data: ")) continue;
          const data = line.slice(6);
          if (data === "[DONE]") break;

          const { text } = JSON.parse(data);
          setMessages((prev) => {
            const updated = [...prev];
            updated[updated.length - 1] = {
              role: "assistant",
              content: updated[updated.length - 1].content + text,
            };
            return updated;
          });
        }
      }
    } finally {
      setIsStreaming(false);
    }
  }

  return (
    <div className="flex flex-col h-full">
      <div className="flex-1 overflow-y-auto p-4 space-y-4">
        {messages.map((msg, i) => (
          <div
            key={i}
            className={`p-3 rounded-lg ${
              msg.role === "user"
                ? "bg-blue-100 ml-8"
                : "bg-gray-100 mr-8"
            }`}
          >
            <p className="text-sm font-semibold mb-1">
              {msg.role === "user" ? "You" : "Assistant"}
            </p>
            <p className="whitespace-pre-wrap">{msg.content}</p>
            {isStreaming && i === messages.length - 1 && msg.role === "assistant" && (
              <span className="inline-block w-1 h-4 bg-gray-500 animate-pulse ml-1" />
            )}
          </div>
        ))}
      </div>

      <form onSubmit={handleSubmit} className="p-4 border-t">
        <div className="flex gap-2">
          <input
            type="text"
            value={question}
            onChange={(e) => setQuestion(e.target.value)}
            placeholder="Ask a question about your documents..."
            className="flex-1 border rounded-lg px-3 py-2"
            disabled={isStreaming}
          />
          <button
            type="submit"
            disabled={isStreaming || !question.trim()}
            className="bg-blue-600 text-white px-4 py-2 rounded-lg disabled:opacity-50"
          >
            {isStreaming ? "Thinking..." : "Ask"}
          </button>
        </div>
      </form>
    </div>
  );
}

Firestore security rules

Security rules protect user data in multi-tenant deployments. Each user should only be able to read and write their own documents and conversations.

// firestore.rules
rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {

    function isSignedIn() {
      return request.auth != null;
    }

    function isOwner(userId) {
      return isSignedIn() && request.auth.uid == userId;
    }

    match /documents/{documentId} {
      allow read, delete: if isOwner(resource.data.userId);
      allow create: if isSignedIn() && request.resource.data.userId == request.auth.uid;
      allow update: if isOwner(resource.data.userId);

      match /chunks/{chunkId} {
        // Chunks inherit ownership from parent document
        allow read: if isSignedIn() &&
          get(/databases/$(database)/documents/documents/$(documentId)).data.userId == request.auth.uid;
        // Chunks are written only by Cloud Functions (service account)
        allow write: if false;
      }
    }

    match /conversations/{conversationId} {
      allow read, create, update, delete: if isOwner(resource.data.userId);

      match /messages/{messageId} {
        allow read: if isSignedIn() &&
          get(/databases/$(database)/documents/conversations/$(conversationId)).data.userId == request.auth.uid;
        // User messages written client-side; assistant messages written by server
        allow create: if isSignedIn() &&
          get(/databases/$(database)/documents/conversations/$(conversationId)).data.userId == request.auth.uid &&
          request.resource.data.role == "user";
      }
    }
  }
}

Handling large documents and cost management

Two practical constraints govern how far this architecture scales without changes: Firestore read costs and Claude's context window.

Firestore read costs scale with the number of chunks fetched per query. If a user uploads a 500-page PDF chunked into 2000 chunks, every retrieval query fetches and scores 2000 documents. At 60,000 chunks across a user's document library, this becomes expensive. Implement document-level limits (e.g., cap at 200 pages per document) or migrate to a vector database for users whose chunk count exceeds a threshold.

Claude's context window limits how many retrieved chunks you can include. At 5 chunks of 512 tokens each, you use ~2500 tokens of context—well within any current Claude model. But if you are building a system that retrieves from many documents simultaneously, the context fills up quickly. Implement a budget: count tokens for each chunk before adding it to the context, and stop when you approach the limit.

function buildContextWithBudget(
  chunks: RetrievedChunk[],
  maxContextTokens: number = 4000,
): string {
  const selectedChunks: RetrievedChunk[] = [];
  let totalTokens = 0;

  for (const chunk of chunks) {
    // Approximate token count
    const chunkTokens = Math.ceil(chunk.text.length / 4);
    if (totalTokens + chunkTokens > maxContextTokens) break;
    selectedChunks.push(chunk);
    totalTokens += chunkTokens;
  }

  return selectedChunks
    .map((c, i) => `[Source ${i + 1}]\n${c.text}`)
    .join("\n\n---\n\n");
}

Indexing strategies for improving retrieval quality

Beyond cosine similarity, there are several retrieval strategies that materially improve answer quality without requiring a dedicated vector database.

Hybrid retrieval. Combine semantic search with BM25 keyword search and merge the results using Reciprocal Rank Fusion (RRF). Semantic search captures meaning; BM25 captures exact terminology. A user asking about a specific product code ("ZR-4401") will be poorly served by semantic search alone—BM25 will surface that exact string.

Metadata pre-filtering. Before scoring all chunks by embedding similarity, filter by structured metadata. If your documents have a section or chapter field on each chunk, a query like "summarise the pricing section" can first filter to chunks where section === "pricing", drastically reducing the candidate set and improving both latency and precision.

Re-ranking. After fetching the top-20 chunks by cosine similarity, pass them to a re-ranker model (Cohere's rerank-english-v3.0 or a cross-encoder) that scores each chunk in the context of the exact query. Re-ranking consistently outperforms raw cosine similarity because cross-encoders can consider the query and document together, not just their independent embeddings.

// Re-rank retrieved chunks using Cohere
import { CohereClient } from "cohere-ai";

const cohere = new CohereClient({ token: process.env.COHERE_API_KEY });

async function rerankChunks(
  query: string,
  chunks: RetrievedChunk[],
  topN: number = 5,
): Promise<RetrievedChunk[]> {
  const result = await cohere.rerank({
    query,
    documents: chunks.map((c) => c.text),
    model: "rerank-english-v3.0",
    topN,
  });

  return result.results.map((r) => ({
    ...chunks[r.index],
    score: r.relevanceScore,
  }));
}

Adding re-ranking as a final stage after cosine retrieval is one of the highest-ROI improvements available to this architecture—it requires no schema changes and typically increases answer accuracy by 15–25% on domain-specific corpora.

This architecture provides a solid foundation for a document Q&A product. The Firestore + Cloud Functions combination handles ingestion reliably, the streaming Route Handler gives users fast perceived response times, and the security rules enforce proper data isolation from day one.