Implementing LLM fallback chains when primary models fail

By Zheng Hui · 17 July 2026105 views
Implementing LLM fallback chains when primary models fail
#llm#reliability#fallback#architecture#error-handling

Why LLM applications fail silently without fallback chains

An API call to an LLM provider can fail for many reasons: the provider is down, you've hit a rate limit, the specific model you requested is overloaded, your request exceeds the context window, or the response quality drops below a threshold you care about. Without a deliberate fallback strategy, any of these failure modes surfaces as an error to your user — or worse, as a subtly degraded experience that's hard to diagnose.

The difference between a toy LLM integration and a production system often comes down to exactly this: what happens in the unhappy path. Production LLM applications need the same reliability engineering discipline that other distributed systems demand — circuit breakers, timeouts, retries, fallbacks, and observability.

This article builds a complete fallback chain implementation from scratch, covering the decision points that actually matter in production: when to retry versus fall back, how to detect quality degradation (not just hard failures), how to route across multiple providers without coupling to any single one, and how to make all of this observable so you can actually diagnose what's happening when things go wrong.

Anatomy of an LLM failure

Before building a fallback chain, it's worth being precise about the failure modes you're protecting against:

Hard failures — the API returns an error code. These include:

  • 429 Too Many Requests (rate limit)
  • 529 Overloaded (provider capacity)
  • 500 Internal Server Error (provider-side bug)
  • Network timeout (request never completes)
  • Connection refused (service down)

Soft failures — the API returns a 200 but the response is unusable:

  • Model returns empty content
  • Model refuses the request (content policy)
  • Model returns malformed output that fails your schema validation
  • Model returns a response that's logically incoherent for the task
  • Response is technically valid but confidence/quality is below your threshold

Degradation failures — performance drops that don't trigger hard errors:

  • Latency spikes (model is slow but eventually responds)
  • Increased hallucination rates during a model rollout
  • Context window changes that silently truncate inputs

A good fallback chain handles hard failures with automatic retries and provider switching, detects soft failures through output validation, and catches degradation through monitoring and circuit breakers.

Building the base fallback chain

Start with a clean abstraction that separates the concern of "which model to call" from "what to do when it fails."

import Anthropic from "@anthropic-ai/sdk";

export interface LLMRequest {
  system?: string;
  messages: Anthropic.MessageParam[];
  maxTokens?: number;
  temperature?: number;
}

export interface LLMResponse {
  content: string;
  model: string;
  provider: string;
  inputTokens: number;
  outputTokens: number;
  latencyMs: number;
}

export interface FallbackResult extends LLMResponse {
  attemptCount: number;
  usedFallback: boolean;
  failureReasons: string[];
}

export type ModelConfig = {
  provider: "anthropic" | "openai" | "google";
  model: string;
  maxRetries?: number;
  timeoutMs?: number;
};

export class FallbackChain {
  private chain: ModelConfig[];
  private maxRetriesPerModel: number;
  private requestTimeoutMs: number;

  constructor(
    chain: ModelConfig[],
    options: { maxRetriesPerModel?: number; requestTimeoutMs?: number } = {}
  ) {
    if (chain.length === 0) throw new Error("Fallback chain must have at least one model");
    this.chain = chain;
    this.maxRetriesPerModel = options.maxRetriesPerModel ?? 2;
    this.requestTimeoutMs = options.requestTimeoutMs ?? 30_000;
  }

  async call(request: LLMRequest): Promise<FallbackResult> {
    const failureReasons: string[] = [];
    let attemptCount = 0;

    for (const modelConfig of this.chain) {
      const retries = modelConfig.maxRetries ?? this.maxRetriesPerModel;

      for (let attempt = 0; attempt <= retries; attempt++) {
        attemptCount++;
        try {
          const result = await this.callWithTimeout(request, modelConfig);
          
          // Validate response quality before accepting
          const validationError = this.validateResponse(result.content, request);
          if (validationError) {
            failureReasons.push(
              `${modelConfig.model}: quality check failed — ${validationError}`
            );
            break; // Don't retry soft quality failures on the same model
          }

          return {
            ...result,
            attemptCount,
            usedFallback: this.chain.indexOf(modelConfig) > 0,
            failureReasons,
          };
        } catch (err) {
          const reason = this.classifyError(err, modelConfig.model, attempt);
          failureReasons.push(reason.message);

          if (!reason.shouldRetry) break;
          if (attempt < retries) await this.backoff(attempt);
        }
      }
    }

    throw new Error(
      `All models in fallback chain exhausted. Failures: ${failureReasons.join("; ")}`
    );
  }

  private async callWithTimeout(
    request: LLMRequest,
    config: ModelConfig
  ): Promise<LLMResponse> {
    const timeoutMs = config.timeoutMs ?? this.requestTimeoutMs;
    const timeoutPromise = new Promise<never>((_, reject) =>
      setTimeout(() => reject(new Error(`Timeout after ${timeoutMs}ms`)), timeoutMs)
    );
    return Promise.race([this.callProvider(request, config), timeoutPromise]);
  }

  private async backoff(attempt: number): Promise<void> {
    const delayMs = Math.min(1000 * Math.pow(2, attempt), 8000);
    const jitter = Math.random() * delayMs * 0.1;
    await new Promise((r) => setTimeout(r, delayMs + jitter));
  }
}

Implementing per-provider adapters

Each provider has a different SDK and different error shapes. Wrapping them in a common interface means your fallback chain doesn't need to know which provider it's talking to.

import Anthropic from "@anthropic-ai/sdk";
import OpenAI from "openai";

async function callAnthropic(
  request: LLMRequest,
  model: string
): Promise<LLMResponse> {
  const client = new Anthropic();
  const start = Date.now();

  const response = await client.messages.create({
    model,
    max_tokens: request.maxTokens ?? 1024,
    system: request.system,
    messages: request.messages,
    temperature: request.temperature,
  });

  const content = response.content
    .filter((b) => b.type === "text")
    .map((b) => (b as Anthropic.TextBlock).text)
    .join("");

  return {
    content,
    model,
    provider: "anthropic",
    inputTokens: response.usage.input_tokens,
    outputTokens: response.usage.output_tokens,
    latencyMs: Date.now() - start,
  };
}

async function callOpenAI(
  request: LLMRequest,
  model: string
): Promise<LLMResponse> {
  const client = new OpenAI();
  const start = Date.now();

  const messages: OpenAI.ChatCompletionMessageParam[] = [];
  if (request.system) {
    messages.push({ role: "system", content: request.system });
  }
  for (const msg of request.messages) {
    messages.push({
      role: msg.role as "user" | "assistant",
      content: typeof msg.content === "string" ? msg.content : JSON.stringify(msg.content),
    });
  }

  const response = await client.chat.completions.create({
    model,
    max_tokens: request.maxTokens ?? 1024,
    messages,
    temperature: request.temperature,
  });

  return {
    content: response.choices[0].message.content ?? "",
    model,
    provider: "openai",
    inputTokens: response.usage?.prompt_tokens ?? 0,
    outputTokens: response.usage?.completion_tokens ?? 0,
    latencyMs: Date.now() - start,
  };
}

// In FallbackChain:
private async callProvider(
  request: LLMRequest,
  config: ModelConfig
): Promise<LLMResponse> {
  switch (config.provider) {
    case "anthropic":
      return callAnthropic(request, config.model);
    case "openai":
      return callOpenAI(request, config.model);
    default:
      throw new Error(`Unknown provider: ${config.provider}`);
  }
}

Error classification: retry vs fall back vs give up

Not all errors should be retried. Classifying errors correctly prevents unnecessary delay and avoids hammering a provider that's clearly having issues.

interface ErrorClassification {
  message: string;
  shouldRetry: boolean;
  shouldFallback: boolean;
  isRateLimit: boolean;
}

private classifyError(
  err: unknown,
  modelName: string,
  attempt: number
): ErrorClassification {
  const message = err instanceof Error ? err.message : String(err);

  // Anthropic-specific error shapes
  if (err instanceof Anthropic.APIStatusError) {
    if (err.status === 429) {
      return {
        message: `${modelName}: rate limited (attempt ${attempt + 1})`,
        shouldRetry: attempt < 2,   // retry rate limits with backoff, but not forever
        shouldFallback: attempt >= 2,
        isRateLimit: true,
      };
    }
    if (err.status === 529 || err.status === 503) {
      return {
        message: `${modelName}: overloaded (${err.status})`,
        shouldRetry: attempt < 1,
        shouldFallback: true,
        isRateLimit: false,
      };
    }
    if (err.status === 400) {
      // Bad request — retrying won't help
      return {
        message: `${modelName}: bad request — ${message}`,
        shouldRetry: false,
        shouldFallback: true,
        isRateLimit: false,
      };
    }
    if (err.status >= 500) {
      return {
        message: `${modelName}: server error (${err.status})`,
        shouldRetry: attempt < 2,
        shouldFallback: attempt >= 1,
        isRateLimit: false,
      };
    }
  }

  if (message.includes("Timeout")) {
    return {
      message: `${modelName}: timeout`,
      shouldRetry: attempt < 1,
      shouldFallback: true,
      isRateLimit: false,
    };
  }

  // Unknown errors — fall back immediately
  return {
    message: `${modelName}: unknown error — ${message}`,
    shouldRetry: false,
    shouldFallback: true,
    isRateLimit: false,
  };
}

Soft failure detection: quality-based fallback

Hard errors are easy. The harder problem is detecting when the model returned a valid 200 response but the content is unusable. You need task-specific quality checks.

// Base quality validator — override per use case
protected validateResponse(
  content: string,
  request: LLMRequest
): string | null {
  if (!content || content.trim().length === 0) {
    return "empty response";
  }
  if (content.length < 10) {
    return "suspiciously short response";
  }
  // Check for refusal patterns
  const refusalPatterns = [
    /i (cannot|can't|am unable to)/i,
    /i (won't|will not) (help|assist|provide)/i,
    /this (request|content) (violates|goes against)/i,
  ];
  for (const pattern of refusalPatterns) {
    if (pattern.test(content)) {
      return `model refused: matched pattern ${pattern}`;
    }
  }
  return null; // null means valid
}

For structured extraction tasks, extend this with schema validation:

class JsonExtractionChain extends FallbackChain {
  private schema: z.ZodType;

  constructor(chain: ModelConfig[], schema: z.ZodType) {
    super(chain);
    this.schema = schema;
  }

  protected validateResponse(content: string): string | null {
    const baseError = super.validateResponse(content, {} as LLMRequest);
    if (baseError) return baseError;

    try {
      const cleaned = content.replace(/^```(?:json)?\s*/m, "").replace(/\s*```$/m, "").trim();
      const parsed = JSON.parse(cleaned);
      this.schema.parse(parsed);
      return null;
    } catch (err) {
      return `JSON validation failed: ${err instanceof Error ? err.message : String(err)}`;
    }
  }
}

Circuit breakers: avoiding thundering-herd retries

When a provider has an extended outage, retrying every request independently causes a thundering herd. A circuit breaker tracks failure rate across requests and opens (stops sending) when failures exceed a threshold.

type CircuitState = "closed" | "open" | "half-open";

class CircuitBreaker {
  private state: CircuitState = "closed";
  private failureCount = 0;
  private lastFailureTime = 0;
  private successCount = 0;

  constructor(
    private readonly failureThreshold = 5,
    private readonly resetTimeMs = 60_000,
    private readonly halfOpenSuccessThreshold = 2
  ) {}

  canRequest(): boolean {
    if (this.state === "closed") return true;
    if (this.state === "open") {
      if (Date.now() - this.lastFailureTime > this.resetTimeMs) {
        this.state = "half-open";
        this.successCount = 0;
        return true;
      }
      return false;
    }
    return true; // half-open: allow test request
  }

  recordSuccess(): void {
    this.failureCount = 0;
    if (this.state === "half-open") {
      this.successCount++;
      if (this.successCount >= this.halfOpenSuccessThreshold) {
        this.state = "closed";
      }
    }
  }

  recordFailure(): void {
    this.failureCount++;
    this.lastFailureTime = Date.now();
    if (this.failureCount >= this.failureThreshold) {
      this.state = "open";
    }
    if (this.state === "half-open") {
      this.state = "open";
    }
  }

  get currentState(): CircuitState {
    return this.state;
  }
}

// Integrate circuit breakers per model in the chain
class ResilientFallbackChain extends FallbackChain {
  private breakers: Map<string, CircuitBreaker>;

  constructor(chain: ModelConfig[]) {
    super(chain);
    this.breakers = new Map(
      chain.map((c) => [c.model, new CircuitBreaker()])
    );
  }

  private getBreakerKey(config: ModelConfig): string {
    return `${config.provider}:${config.model}`;
  }

  protected async callWithBreaker(
    request: LLMRequest,
    config: ModelConfig
  ): Promise<LLMResponse> {
    const key = this.getBreakerKey(config);
    const breaker = this.breakers.get(config.model)!;

    if (!breaker.canRequest()) {
      throw new Error(`Circuit breaker open for ${config.model}`);
    }

    try {
      const result = await this.callProvider(request, config);
      breaker.recordSuccess();
      return result;
    } catch (err) {
      breaker.recordFailure();
      throw err;
    }
  }
}

Observability: making fallbacks visible

Fallbacks that are invisible are dangerous. You need to know when they're firing, why, and how often — otherwise a broken primary model goes unnoticed while your fallback quietly serves degraded results.

interface FallbackEvent {
  timestamp: string;
  requestId: string;
  primaryModel: string;
  usedModel: string;
  usedFallback: boolean;
  attemptCount: number;
  failureReasons: string[];
  latencyMs: number;
  inputTokens: number;
  outputTokens: number;
}

class ObservableFallbackChain extends ResilientFallbackChain {
  private logger: (event: FallbackEvent) => void;
  private metrics: MetricsClient;

  constructor(
    chain: ModelConfig[],
    logger: (event: FallbackEvent) => void,
    metrics: MetricsClient
  ) {
    super(chain);
    this.logger = logger;
    this.metrics = metrics;
  }

  async call(request: LLMRequest): Promise<FallbackResult> {
    const requestId = crypto.randomUUID();
    const start = Date.now();

    try {
      const result = await super.call(request);

      const event: FallbackEvent = {
        timestamp: new Date().toISOString(),
        requestId,
        primaryModel: this.chain[0].model,
        usedModel: result.model,
        usedFallback: result.usedFallback,
        attemptCount: result.attemptCount,
        failureReasons: result.failureReasons,
        latencyMs: Date.now() - start,
        inputTokens: result.inputTokens,
        outputTokens: result.outputTokens,
      };

      this.logger(event);

      this.metrics.increment("llm.request.success", {
        model: result.model,
        provider: result.provider,
        used_fallback: String(result.usedFallback),
      });

      if (result.usedFallback) {
        this.metrics.increment("llm.fallback.triggered", {
          primary_model: this.chain[0].model,
          fallback_model: result.model,
        });
      }

      return result;
    } catch (err) {
      this.metrics.increment("llm.request.failed", {
        primary_model: this.chain[0].model,
      });
      throw err;
    }
  }
}

Set up alerts on these metrics:

  • llm.fallback.triggered rate > 5% of requests over a 5-minute window
  • llm.request.failed rate > 1% of requests over a 5-minute window
  • P99 latency > 10 seconds (indicates fallbacks are adding unacceptable delay)

Putting it together: realistic chain configurations

// High-reliability chain: same provider, smaller model as fallback
const reliableChain = new ObservableFallbackChain(
  [
    { provider: "anthropic", model: "claude-opus-4-5", maxRetries: 1, timeoutMs: 20_000 },
    { provider: "anthropic", model: "claude-sonnet-4-5", maxRetries: 2, timeoutMs: 15_000 },
    { provider: "anthropic", model: "claude-haiku-4-5", maxRetries: 2, timeoutMs: 10_000 },
  ],
  (event) => structuredLog.info("llm_fallback_event", event),
  metricsClient
);

// Cross-provider chain: switch providers on failure
const crossProviderChain = new ObservableFallbackChain(
  [
    { provider: "anthropic", model: "claude-sonnet-4-5", maxRetries: 1 },
    { provider: "openai", model: "gpt-4o", maxRetries: 1 },
    { provider: "openai", model: "gpt-4o-mini", maxRetries: 2 },
  ],
  logger,
  metricsClient
);

// Usage
const result = await reliableChain.call({
  system: "You are a helpful assistant.",
  messages: [{ role: "user", content: "Summarise this document: ..." }],
  maxTokens: 512,
});

console.log(`Responded with ${result.model}, ${result.attemptCount} attempts`);
if (result.usedFallback) {
  console.warn("Fallback was used:", result.failureReasons);
}

The patterns in this article — retry with backoff, per-error-class routing, quality validation, circuit breakers, and structured observability — form the reliability layer that separates brittle LLM demos from applications that work consistently in production. Start with the simplest version of each layer and add complexity only when your metrics show you need it.

Comments

No comments yet. Be the first!

Sign in to leave a comment.

Implementing LLM fallback chains when primary models fail — ANN Tech