Zero-downtime deployments: the checklist that covers the cases most teams miss

By Tomás Ferreira · 21 July 20260 views
#devops#deployment#performance#databases#backend

An engineering team deploys a new version of their application using blue-green deployment. The new version starts receiving traffic. Error reports come in from users mid-session. The error: a required JSON field that the new frontend sends is rejected by the old backend instances still serving some requests.

The blue-green deployment executed correctly. The problem was not with the deployment strategy — it was with a backwards-incompatible API change between versions that had not been accounted for in the deployment plan.

Zero-downtime deployment is not a deployment strategy. It is a set of constraints on what can change across a deployment boundary, combined with appropriate deployment mechanics. A team can use the most sophisticated canary release pipeline in the industry and still cause downtime if a database column is dropped before the code that reads it is fully retired.

The scenarios that cause downtime in otherwise correctly-executed deployments are predictable. They are also preventable, but only if the team identifies them before deployment — not during.

The root cause: deployment as a point in time

The mental model that produces downtime is treating deployment as an instantaneous event. In this model, version N is running; at T+0, version N+1 is running. The transition is atomic. In reality, deployment is a window. During a rolling deployment, blue-green switch, or canary release, both versions are simultaneously serving traffic. This window can last seconds for a fast rolling deployment or minutes for a canary that ramps traffic gradually.

Everything that changes between versions must be safe to run concurrently. An API change that is incompatible between versions causes errors during the overlap window. A database migration that removes a column causes errors in the code of the version that still reads it. A CDN cache that serves the old HTML pointing to new bundle hashes causes 404s.

The backwards compatibility window is not an implementation detail. It is the central constraint that zero-downtime deployment depends on.

The backwards compatibility window

During a rolling deployment, new and old versions run simultaneously. Any request might be handled by either version. API contracts must be maintained across both versions for the duration of the rollout.

Additive changes are safe: adding a new optional JSON field, adding a new endpoint, adding a new enum value to a non-exhaustive list.

Breaking changes require a deprecation window: removing a field, changing a field's type, making an optional field required, renaming an endpoint.

The pattern for breaking API changes:

// Version N: field is optional
interface CheckoutRequest {
  userId: string;
  items: CartItem[];
  promoCode?: string;  // Optional
}

// Version N+1: make promoCode required
// WRONG: Breaking change — old clients don't send promoCode
interface CheckoutRequest {
  userId: string;
  items: CartItem[];
  promoCode: string;  // Now required — breaks old clients during rollout
}

// CORRECT: Three-phase migration
// Phase 1 (Version N): Mark as optional, handle absence on server
// Phase 2 (Version N+1): Continue accepting both forms, add logic for new field
// Phase 3 (Version N+2): After all clients are on N+1, make field required

// Server handles both cases during the transition:
async function handleCheckout(request: CheckoutRequest) {
  const promoCode = request.promoCode ?? '';  // Graceful default for old clients
  // ...
}

The three-phase migration requires patience. It is tempting to collapse phases 1 and 2 or skip phase 3 cleanup. Skipping cleanup means the compatibility shim remains indefinitely, which accumulates technical debt. The right discipline is committing to all three phases in the same planning cycle.

What teams try instead

The most common shortcut is coordinating the deployment to minimize the overlap window. "Deploy backend first, wait for it to be fully rolled out, then deploy frontend." This works when the deployment completes in seconds. It fails when a deployment takes longer than expected, when rollback is needed mid-deployment, or when the "fully rolled out" state is hard to verify.

A second common shortcut is maintenance windows. Take the service offline for 10 minutes during the deployment. This guarantees no concurrent version overlap, but it is explicitly not zero-downtime. Maintenance windows are an admission that the deployment is not safe to run while users are active.

The correct approach does not require shortcuts. It requires treating the overlap window as a constraint that shapes the design of every change — not an implementation detail to be managed with coordination.

Database migration sequencing

Database migrations and code deployments must be sequenced to maintain compatibility:

The safe sequence:

  1. Deploy database migration that is backwards compatible (additive only)
  2. Deploy new code that can handle both old and new schema
  3. (Optional) Deploy cleanup migration to remove old columns/tables

Column addition (safe):

-- Migration 1: Add column with default (no code change required yet)
ALTER TABLE orders ADD COLUMN discount_amount DECIMAL(10,2) NOT NULL DEFAULT 0;

-- Now both old code (ignores discount_amount) and new code (uses it) work correctly

Column removal (requires care):

-- Phase 1: Stop writing to the column in code (deploy first)
-- Phase 2: Remove the column after the code is deployed everywhere
ALTER TABLE orders DROP COLUMN legacy_promo_code;
-- Safe only after all code that reads legacy_promo_code is gone

Column rename (two-phase):

-- Phase 1: Add new column alongside old
ALTER TABLE orders ADD COLUMN customer_email VARCHAR(254);
UPDATE orders SET customer_email = user_email;  -- Backfill

-- Phase 2: Deploy code that writes to both columns
-- (new code writes customer_email, still reads user_email for old records)

-- Phase 3: Deploy code that reads from customer_email only

-- Phase 4: Remove old column
ALTER TABLE orders DROP COLUMN user_email;

The column rename sequence takes four deployments and at least four deployment cycles. This is not a failure of process — this is what zero-downtime column renames require. Teams that are surprised by this have never needed to do it with real traffic.

Large table migrations

Migrations on large tables present an additional constraint: locking. An ALTER TABLE that adds a NOT NULL column on a 50-million-row table may hold a table lock for minutes. During that lock, writes fail.

-- For large tables: add as nullable first, then fill in a background job
-- PostgreSQL example

-- Step 1: Add column as nullable (no lock needed for nullable columns in Postgres 11+)
ALTER TABLE orders ADD COLUMN discount_tier VARCHAR(20);

-- Step 2: Backfill in batches (background job, no table lock)
UPDATE orders
SET discount_tier = 'standard'
WHERE discount_tier IS NULL
  AND id BETWEEN 1 AND 100000;

-- Repeat in small batches until all rows are filled

-- Step 3: Add NOT NULL constraint (Postgres validates without full table lock in Postgres 12+)
ALTER TABLE orders ALTER COLUMN discount_tier SET NOT NULL;
-- Requires all rows to have been filled in Step 2

The batch backfill takes longer but keeps the table available for writes throughout. The constraint addition in Postgres 12+ uses a partial lock that allows reads and writes during validation.

Static asset CDN timing

A frontend deployment serves new HTML that references JavaScript bundles by content-hash. If a CDN edge node has cached the old HTML but not yet propagated the new bundle, users receive old HTML pointing to a new bundle that does not exist yet — or new HTML pointing to old bundles that have expired.

The fix: deploy assets to the CDN before deploying the HTML that references them, and keep old assets available for the duration of the rollout.

# Deploy order for Next.js/static sites:

# 1. Upload new static assets (JS, CSS, images) to CDN
#    These are content-addressed — old assets still exist
aws s3 sync .next/static s3://my-app-assets/_next/static

# 2. (Wait for CDN propagation — 30-120 seconds depending on CDN)

# 3. Deploy the HTML that references the new assets
firebase hosting:deploy
# At this point:
# - New assets are on the CDN
# - Old HTML still references old assets (still available)
# - New HTML references new assets (now available)
# - Both old and new visitors are served correctly

For Cloudflare Workers or similar edge deployments, the sequencing is similar but the propagation mechanism differs:

# Cloudflare Pages deployment sequence
# 1. Upload _next/static assets to R2 (persistent object storage)
wrangler r2 object put my-app-assets/_next/static --recursive --directory=.next/static

# 2. Deploy the Pages project (triggers edge propagation of HTML)
wrangler pages deploy .next --project-name=my-app

# R2 objects are available immediately across all edge nodes
# Pages deployment propagates over 30-60 seconds
# Old assets remain in R2 until explicitly deleted

A common mistake is configuring aggressive cache-control on HTML files. The HTML should not be cached at the CDN level (or cached with a very short TTL like 60 seconds) while the static assets — which are content-addressed and never change — should be cached indefinitely.

// next.config.js — configure cache headers correctly
const nextConfig = {
  async headers() {
    return [
      {
        // HTML pages: short cache, always revalidate
        source: '/((?!_next/static).*)',
        headers: [
          { key: 'Cache-Control', value: 'public, max-age=0, must-revalidate' },
        ],
      },
      {
        // Static assets: immutable (content-addressed)
        source: '/_next/static/:path*',
        headers: [
          { key: 'Cache-Control', value: 'public, max-age=31536000, immutable' },
        ],
      },
    ];
  },
};

Session continuity

When a user has an active session and the session store format changes between versions, their session may be invalid on the new version.

The safe migration for session format changes:

// Version N session format
interface SessionV1 {
  userId: string;
  role: string;
  createdAt: number;
}

// Version N+1 session format — added tenantId
interface SessionV2 {
  userId: string;
  role: string;
  createdAt: number;
  tenantId: string;  // New field
}

// Session reading code during migration window
function readSession(session: SessionV1 | SessionV2): SessionV2 {
  if ('tenantId' in session) {
    return session;  // Already V2
  }

  // V1 session — derive tenantId from userId
  return {
    ...session,
    tenantId: getUserTenantId(session.userId)
  };
}
// After all sessions have naturally expired (or after forcing re-login),
// the V1 compatibility code can be removed

The session compatibility code has a natural lifecycle: it is needed during the deployment window and for the duration of any active V1 sessions. If V1 sessions expire after 30 days, the compatibility code can be removed 30 days after the deployment.

For applications where forcing re-login is acceptable, the simplest approach is incrementing the session version on deployment and invalidating all sessions with an older version:

const CURRENT_SESSION_VERSION = 2;  // Increment on breaking session format changes

function validateSession(session: any): boolean {
  if (session.version !== CURRENT_SESSION_VERSION) {
    // Force re-login — session format changed
    return false;
  }
  return true;
}

This approach trades session continuity for simplicity. Whether it is acceptable depends on the application: a consumer app with "remember me" will frustrate users; a banking application may prefer to force re-login on any session format change.

Message queue format changes

Rolling deployments with queued work introduce an additional compatibility constraint: messages already in the queue may have been produced by version N and will be consumed by version N+1.

// Message format migration — producer/consumer compatibility

// Version N message
interface OrderEventV1 {
  orderId: string;
  userId: string;
  amount: number;
}

// Version N+1 message — added currency field
interface OrderEventV2 {
  orderId: string;
  userId: string;
  amount: number;
  currency: string;  // New field
}

// Version N+1 consumer — must handle both V1 and V2 messages
async function handleOrderEvent(message: OrderEventV1 | OrderEventV2) {
  const currency = 'currency' in message ? message.currency : 'USD';  // Default for V1
  // Process the event
}

// Version N+1 producer — always produces V2 messages
function createOrderEvent(order: Order): OrderEventV2 {
  return {
    orderId: order.id,
    userId: order.userId,
    amount: order.total,
    currency: order.currency,
  };
}

The consumer must be deployed and capable of handling both formats before the producer starts emitting the new format. Deploy the consumer first; the producer second. Only retire V1 compatibility in the consumer after all V1 messages in the queue have been consumed.

The zero-downtime deployment checklist

Before each deployment:

  • Is this API change additive? If not, is there a compatibility layer for the rollout window?
  • Does this deploy include a database migration? If yes, is the migration backwards compatible with the current code version?
  • Does the migration lock large tables? If yes, is there a batched alternative?
  • Do new static assets need to be pre-deployed to the CDN before the HTML that references them?
  • Does the session format change? If yes, does the new code handle both old and new session formats?
  • Does this deploy change any message queue message format? If yes, are both old and new formats supported for the duration of the rollout?
  • Is there an automated smoke test that runs against production after deployment? Is it configured to trigger a rollback on failure?
  • Is there a documented rollback plan with a tested execution time?

Common mistakes in otherwise-correct deployments

Mistake 1: Assuming the rollout is fast. Blue-green deployments switch traffic almost instantly, but the old version stays alive for in-flight requests. Rolling deployments can take 10-30 minutes on large clusters. Plan for the worst-case overlap duration.

Mistake 2: Rolling back only the code, not the database migration. If a deployment is rolled back, the database migration has already run. The rolled-back code must be compatible with the migrated schema. This means database migrations must be backwards compatible even in rollback scenarios — which is another reason additive-only migrations are safer.

Mistake 3: Not verifying rollback. The rollback procedure should be tested in staging before every production deployment. A rollback that has never been exercised may fail when it is needed most.

The blue-green deployment failure was the result of skipping the first question on the checklist. The API change was not additive — the new frontend required a field that the old backend rejected. A compatibility shim (accepting the field but ignoring it in the old backend) would have allowed the rollout to complete before any old-backend instances were shut down. The shim cost one hour of development. The outage cost two hours of recovery and 20 minutes of user-facing errors. The math is straightforward; the discipline to apply it consistently is what separates teams that achieve zero downtime from teams that schedule maintenance windows.

Comments

No comments yet. Be the first!

Sign in to leave a comment.

Zero-downtime deployments: the checklist that covers the cases most teams miss — ANN Tech