Building a fact-checking pipeline with LLM and web search

Why fact-checking deserves its own architecture

LLMs are confidently wrong at a measurable rate. They hallucinate statistics, misattribute quotes, and repeat outdated information from training data that may be months or years stale. When you build a product that processes user-submitted claims, news articles, or research summaries, shipping raw LLM output without verification is a liability.

A fact-checking pipeline treats verification as a first-class concern. Instead of asking a model whether something is true (and trusting its parametric memory), you decompose the claim into checkable sub-claims, retrieve fresh evidence from the web, then ask the model to reason over that evidence to produce a verdict. The model's role shifts from oracle to analyst.

This article walks through building that pipeline end to end: claim decomposition, evidence retrieval, cross-source validation, verdict generation, and confidence scoring. The implementation uses TypeScript with the Anthropic SDK and the Brave Search API, but the architecture applies to any LLM and search provider.

Claim decomposition: breaking complex statements into checkable units

Most real-world claims are compound. "Company X reported a 40% revenue increase in Q3 and plans to expand to three new markets by end of year" contains at least two independently verifiable assertions. Checking them as a single unit produces unreliable results because the model may confirm one part while glossing over the other.

The first stage of the pipeline decomposes the input into atomic, checkable claims.

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

interface Claim {
  id: string;
  text: string;
  searchQueries: string[];
}

async function decomposeClaim(input: string): Promise<Claim[]> {
  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 1024,
    messages: [
      {
        role: "user",
        content: `Decompose the following statement into atomic, independently verifiable claims. For each claim, also provide 2-3 search queries that would help verify it.

Statement: ${input}

Respond with a JSON array of objects with this shape:
{
  "id": "c1",
  "text": "the atomic claim",
  "searchQueries": ["query 1", "query 2"]
}`,
      },
    ],
  });

  const text =
    response.content[0].type === "text" ? response.content[0].text : "";
  const jsonMatch = text.match(/\[[\s\S]*\]/);
  if (!jsonMatch) throw new Error("Failed to extract claims from model output");

  return JSON.parse(jsonMatch[0]);
}

The search queries generated alongside each claim are important. You want queries that would surface contradicting evidence, not just confirming evidence. You can prompt the model explicitly to generate both confirming and disconfirming queries:

async function decomposeclaimWithContrarian(input: string): Promise<Claim[]> {
  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 1500,
    system:
      "You are a rigorous fact-checker. For each claim, generate search queries that might DISPROVE it as well as queries that might confirm it. Intellectual honesty requires actively looking for contradicting evidence.",
    messages: [
      {
        role: "user",
        content: `Decompose and generate search queries for: ${input}`,
      },
    ],
  });

  // parse response...
  return [];
}

Evidence retrieval: fetching live web content

With atomic claims and search queries in hand, you call a search API for each query. The Brave Search API returns structured results with snippets, URLs, and publication dates.

interface SearchResult {
  title: string;
  url: string;
  snippet: string;
  publishedDate?: string;
}

async function searchWeb(query: string): Promise<SearchResult[]> {
  const params = new URLSearchParams({
    q: query,
    count: "5",
    result_filter: "web",
  });

  const response = await fetch(
    `https://api.search.brave.com/res/v1/web/search?${params}`,
    {
      headers: {
        Accept: "application/json",
        "Accept-Encoding": "gzip",
        "X-Subscription-Token": process.env.BRAVE_API_KEY!,
      },
    }
  );

  if (!response.ok) {
    throw new Error(`Search API error: ${response.status}`);
  }

  const data = await response.json();
  return (data.web?.results ?? []).map((r: any) => ({
    title: r.title,
    url: r.url,
    snippet: r.description,
    publishedDate: r.age,
  }));
}

async function retrieveEvidence(
  claims: Claim[]
): Promise<Map<string, SearchResult[]>> {
  const evidenceMap = new Map<string, SearchResult[]>();

  for (const claim of claims) {
    const allResults: SearchResult[] = [];

    // Run queries in parallel for each claim
    const results = await Promise.all(
      claim.searchQueries.map((q) => searchWeb(q))
    );

    for (const batch of results) {
      allResults.push(...batch);
    }

    // Deduplicate by URL
    const seen = new Set<string>();
    const deduplicated = allResults.filter((r) => {
      if (seen.has(r.url)) return false;
      seen.add(r.url);
      return true;
    });

    evidenceMap.set(claim.id, deduplicated);
  }

  return evidenceMap;
}

Running search queries in parallel for each claim significantly reduces total latency. If you have 4 claims with 3 queries each, sequential execution would make 12 network calls serially; parallel execution collapses that to roughly 3 round-trips (the 3 queries per claim running concurrently).

One practical concern: search snippets are often 150-200 characters, which may not contain enough context to reason about. You can optionally fetch and parse the full page content for the top result per query, though this increases latency and storage requirements substantially. For most use cases, high-quality snippets are sufficient.

Cross-source validation: measuring evidence agreement

Raw search results are noisy. Different sources may contradict each other, and some sources are more authoritative than others. Before sending evidence to the verdict stage, it helps to run a cross-source consistency check.

interface EvidenceAssessment {
  claimId: string;
  supporting: SearchResult[];
  contradicting: SearchResult[];
  ambiguous: SearchResult[];
  sourceQualityScore: number;
}

async function assessEvidence(
  claim: Claim,
  results: SearchResult[]
): Promise<EvidenceAssessment> {
  const resultsText = results
    .map(
      (r, i) =>
        `[${i + 1}] ${r.title} (${r.url})\n${r.snippet}\n${r.publishedDate ? `Published: ${r.publishedDate}` : ""}`
    )
    .join("\n\n");

  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 2048,
    messages: [
      {
        role: "user",
        content: `Claim to verify: "${claim.text}"

Search results:
${resultsText}

Classify each search result as SUPPORTING, CONTRADICTING, or AMBIGUOUS with respect to the claim.
Also assess overall source quality (0-1 score) based on domain authority, recency, and relevance.

Return JSON:
{
  "supporting": [result indices],
  "contradicting": [result indices],  
  "ambiguous": [result indices],
  "sourceQualityScore": 0.8,
  "reasoning": "brief explanation"
}`,
      },
    ],
  });

  const text =
    response.content[0].type === "text" ? response.content[0].text : "";
  const jsonMatch = text.match(/\{[\s\S]*\}/);
  if (!jsonMatch) throw new Error("Failed to parse evidence assessment");

  const parsed = JSON.parse(jsonMatch[0]);

  return {
    claimId: claim.id,
    supporting: parsed.supporting.map((i: number) => results[i - 1]),
    contradicting: parsed.contradicting.map((i: number) => results[i - 1]),
    ambiguous: parsed.ambiguous.map((i: number) => results[i - 1]),
    sourceQualityScore: parsed.sourceQualityScore,
  };
}

The source quality score feeds into the final confidence calculation. A claim supported only by low-authority blogs scores differently than one confirmed by Reuters and the company's own press release.

Verdict generation: structured reasoning over evidence

With assessed evidence in hand, the verdict stage asks the model to reason explicitly before reaching a conclusion. Chain-of-thought reasoning here is non-negotiable — you want the model to work through what the evidence shows before committing to a verdict.

type Verdict = "TRUE" | "FALSE" | "PARTIALLY_TRUE" | "UNVERIFIABLE" | "MISLEADING";

interface ClaimVerdict {
  claimId: string;
  claimText: string;
  verdict: Verdict;
  confidence: number; // 0-1
  explanation: string;
  supportingUrls: string[];
  contradictingUrls: string[];
}

async function generateVerdict(
  claim: Claim,
  assessment: EvidenceAssessment
): Promise<ClaimVerdict> {
  const formatResults = (results: SearchResult[]) =>
    results.map(r => `- ${r.title}: ${r.snippet} (${r.url})`).join("\n");

  const prompt = `You are verifying a specific claim. Reason step by step over the evidence before reaching a verdict.

CLAIM: "${claim.text}"

SUPPORTING EVIDENCE:
${formatResults(assessment.supporting) || "None found"}

CONTRADICTING EVIDENCE:
${formatResults(assessment.contradicting) || "None found"}

AMBIGUOUS EVIDENCE:
${formatResults(assessment.ambiguous) || "None found"}

SOURCE QUALITY SCORE: ${assessment.sourceQualityScore}

Step 1: Summarize what the supporting evidence shows.
Step 2: Summarize what the contradicting evidence shows.
Step 3: Identify any gaps — what evidence would be needed to fully verify this claim but is absent?
Step 4: Reach a verdict.

Verdicts: TRUE, FALSE, PARTIALLY_TRUE, UNVERIFIABLE, MISLEADING

Return your analysis and then this JSON:
{
  "verdict": "TRUE",
  "confidence": 0.85,
  "explanation": "...",
  "supportingUrls": ["url1"],
  "contradictingUrls": []
}`;

  const response = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 2048,
    messages: [{ role: "user", content: prompt }],
  });

  const text = response.content[0].type === "text" ? response.content[0].text : "";
  const jsonMatch = text.match(/\{[\s\S]*"contradictingUrls"[\s\S]*?\}/);
  if (!jsonMatch) throw new Error("Failed to parse verdict");

  const parsed = JSON.parse(jsonMatch[0]);

  return {
    claimId: claim.id,
    claimText: claim.text,
    verdict: parsed.verdict,
    confidence: parsed.confidence,
    explanation: parsed.explanation,
    supportingUrls: parsed.supportingUrls,
    contradictingUrls: parsed.contradictingUrls,
  };
}

Confidence scoring: aggregating across claims

Individual claim verdicts need to be aggregated into a final pipeline output. The aggregation logic should reflect the asymmetry between truth and falsity: a single definitive FALSE should carry more weight than multiple PARTIALLY_TRUE verdicts, because a claim that contains one false sub-assertion is, at minimum, misleading.

interface PipelineResult {
  originalInput: string;
  overallVerdict: Verdict;
  overallConfidence: number;
  claimVerdicts: ClaimVerdict[];
  summary: string;
}

function aggregateVerdicts(verdicts: ClaimVerdict[]): {
  verdict: Verdict;
  confidence: number;
} {
  if (verdicts.length === 0) {
    return { verdict: "UNVERIFIABLE", confidence: 0 };
  }

  const counts = verdicts.reduce(
    (acc, v) => {
      acc[v.verdict] = (acc[v.verdict] || 0) + 1;
      return acc;
    },
    {} as Record<Verdict, number>
  );

  // Any FALSE verdict makes the whole claim at least PARTIALLY_TRUE or FALSE
  if (counts.FALSE > 0) {
    if (counts.TRUE > 0 || counts.PARTIALLY_TRUE > 0) {
      const avgConfidence =
        verdicts.reduce((sum, v) => sum + v.confidence, 0) / verdicts.length;
      return { verdict: "PARTIALLY_TRUE", confidence: avgConfidence * 0.8 };
    }
    const falseConfidence = verdicts
      .filter((v) => v.verdict === "FALSE")
      .reduce((sum, v) => sum + v.confidence, 0) / counts.FALSE;
    return { verdict: "FALSE", confidence: falseConfidence };
  }

  if (counts.MISLEADING > 0) {
    return {
      verdict: "MISLEADING",
      confidence:
        verdicts.reduce((sum, v) => sum + v.confidence, 0) / verdicts.length,
    };
  }

  if (counts.UNVERIFIABLE === verdicts.length) {
    return { verdict: "UNVERIFIABLE", confidence: 0.3 };
  }

  if (counts.TRUE === verdicts.length) {
    const avgConf =
      verdicts.reduce((sum, v) => sum + v.confidence, 0) / verdicts.length;
    return { verdict: "TRUE", confidence: avgConf };
  }

  const avgConf =
    verdicts.reduce((sum, v) => sum + v.confidence, 0) / verdicts.length;
  return { verdict: "PARTIALLY_TRUE", confidence: avgConf };
}

async function runPipeline(input: string): Promise<PipelineResult> {
  console.log("Decomposing claim...");
  const claims = await decomposeClaim(input);

  console.log(`Found ${claims.length} atomic claims. Retrieving evidence...`);
  const evidenceMap = await retrieveEvidence(claims);

  console.log("Assessing evidence...");
  const assessments = await Promise.all(
    claims.map((claim) =>
      assessEvidence(claim, evidenceMap.get(claim.id) ?? [])
    )
  );

  console.log("Generating verdicts...");
  const verdicts = await Promise.all(
    claims.map((claim, i) => generateVerdict(claim, assessments[i]))
  );

  const { verdict, confidence } = aggregateVerdicts(verdicts);

  // Generate a human-readable summary
  const summaryResponse = await client.messages.create({
    model: "claude-opus-4-5",
    max_tokens: 512,
    messages: [
      {
        role: "user",
        content: `Summarize this fact-check result in 2-3 sentences for a general audience:

Original claim: ${input}
Overall verdict: ${verdict} (confidence: ${(confidence * 100).toFixed(0)}%)
Sub-claim verdicts: ${verdicts.map((v) => `${v.claimText}: ${v.verdict}`).join("; ")}`,
      },
    ],
  });

  const summary =
    summaryResponse.content[0].type === "text"
      ? summaryResponse.content[0].text
      : "";

  return {
    originalInput: input,
    overallVerdict: verdict,
    overallConfidence: confidence,
    claimVerdicts: verdicts,
    summary,
  };
}

Production considerations: rate limiting, caching, and cost control

A fact-checking pipeline makes multiple LLM calls and multiple search API calls per request. In production you need to account for this at the infrastructure level.

Caching search results. The same or similar claims appear repeatedly. Cache search results keyed by query string with a TTL of 24 hours. Redis works well for this:

import { Redis } from "@upstash/redis";

const redis = new Redis({
  url: process.env.UPSTASH_REDIS_URL!,
  token: process.env.UPSTASH_REDIS_TOKEN!,
});

async function cachedSearch(query: string): Promise<SearchResult[]> {
  const cacheKey = `search:${Buffer.from(query).toString("base64")}`;
  const cached = await redis.get<SearchResult[]>(cacheKey);
  if (cached) return cached;

  const results = await searchWeb(query);
  await redis.setex(cacheKey, 86400, JSON.stringify(results));
  return results;
}

Rate limiting. Search APIs enforce per-second and per-day quotas. Wrap your search calls in a token bucket rate limiter. If you're processing high volumes, push claims into a queue (BullMQ or Cloudflare Queues) and process them with worker concurrency limits.

Cost estimation. A typical run on a moderately complex 3-claim input might look like:

• Decomposition: ~600 input tokens, ~300 output tokens
• 3 evidence assessments × ~2000 input tokens = ~6000 input tokens
• 3 verdict generations × ~2500 input tokens = ~7500 input tokens
• Summary: ~400 input tokens

Total: roughly 14,500 input tokens and ~2,000 output tokens per pipeline run. At Claude Opus pricing, budget approximately $0.20-$0.40 per run depending on claim complexity. If cost is a constraint, use Haiku for decomposition and assessment, reserving Opus for verdict generation only.

Timeout handling. Web searches occasionally time out. Build a retry wrapper with exponential backoff and a fallback that marks the claim as UNVERIFIABLE rather than failing the entire pipeline:

async function searchWithRetry(
  query: string,
  maxRetries = 3
): Promise<SearchResult[]> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      return await cachedSearch(query);
    } catch (err) {
      if (attempt === maxRetries - 1) {
        console.warn(`Search failed after ${maxRetries} attempts: ${query}`);
        return [];
      }
      await new Promise((r) => setTimeout(r, 2 ** attempt * 500));
    }
  }
  return [];
}

Extending the pipeline: source authority scoring and feedback loops

The architecture described here produces reliable verdicts, but a production system benefits from two additional layers.

Source authority scoring. Not all URLs are equal. Maintain a domain authority table (seeded from a dataset like Majestic Million or Moz's API) and apply a weight multiplier to evidence from high-authority domains. A supporting snippet from reuters.com should count for more than one from an unknown blog:

const DOMAIN_WEIGHTS: Record<string, number> = {
  "reuters.com": 1.5,
  "apnews.com": 1.5,
  "bbc.com": 1.4,
  "nytimes.com": 1.3,
  // ... etc
};

function getDomainWeight(url: string): number {
  try {
    const hostname = new URL(url).hostname.replace(/^www\./, "");
    return DOMAIN_WEIGHTS[hostname] ?? 1.0;
  } catch {
    return 1.0;
  }
}

Human feedback loops. Store pipeline results alongside verdicts and expose a correction interface. When a human reviewer marks a verdict as wrong, log the input, the search results, and the incorrect verdict as a training example. You can use these examples to refine your prompts or, eventually, to fine-tune a smaller model for the classification stage.

The combination of decomposition, live retrieval, multi-source cross-validation, and structured reasoning over evidence produces a pipeline that is meaningfully more reliable than naive LLM fact-checking. Each stage is independently testable and replaceable, which matters as search APIs, models, and accuracy requirements evolve.