Designing for rollback: the deployment discipline that prevents incidents

By Tomás Ferreira · 22 July 2026111 views
Designing for rollback: the deployment discipline that prevents incidents
#devops#deployment#architecture#reliability

A team pushes a critical bug fix to production at 11 PM. CI passes. The deploy succeeds. Two minutes later, monitoring shows 503 errors climbing. The fix introduced a regression. The on-call engineer hits the rollback button. The deployment system reverts to the previous version. The errors continue. The previous version now reads from a database schema that was migrated forward by the buggy deployment. The schema change is not backward compatible. The previous version of the application cannot read the new column format.

The rollback fails. The team is now in a worse position than before the rollback attempt: the current version is the buggy one that cannot be rolled forward, and the previous version cannot run against the current schema. The only option is an emergency forward fix — writing a patch at midnight to address both the original bug and the schema incompatibility.

This scenario is preventable. The team that designed backward-compatible migrations, tested rollback procedures before incidents, and verified that the previous version works against the current schema before deploying would have completed the rollback in two minutes and gone back to sleep.

Why rollback fails

Rollback fails at one of three points.

Schema incompatibility. The new version migrated the database schema in a way that the previous version cannot understand. A column was renamed without maintaining a read alias. A NOT NULL constraint was added without a default that the previous version supplies. A column was dropped that the previous version reads.

Application state incompatibility. The new version wrote data in a format that the previous version cannot parse. A JSON field was restructured. An enum value was added that the previous version's code does not handle. A feature flag was enabled that the previous version does not support and fails on if the flag data is present.

Infrastructure state. Message queue consumers processed events in the new version's format. Cache entries were populated with the new version's data structure. The previous version's readers expect the old format.

Each of these can be addressed at design time, before the deployment. None of them can be addressed quickly at incident time.

The backward-compatible migration discipline

Every database migration that must support rollback must leave the previous version of the application able to run without errors.

The tests for this: after running the migration but before deploying the new code, does the previous version start successfully and handle requests correctly?

# Rollback validation procedure — run before every deployment
# Step 1: Apply migrations to staging
flyway migrate -url=jdbc:postgresql://staging-db/app -target=HEAD

# Step 2: Start the PREVIOUS application version against migrated schema
docker run --env DATABASE_URL=postgresql://staging-db/app app:previous-version

# Step 3: Run smoke tests against the previous version on the migrated schema
./run_smoke_tests.sh staging

# If smoke tests pass: the migration is backward-compatible.
# If smoke tests fail: the migration must be redesigned before going to production.

This validation catches incompatibilities before they reach production. A migration that fails this test is redesigned before it ships.

The specific migration patterns that break rollback and their safe alternatives:

-- BREAKS ROLLBACK: Renaming a column
-- Previous version reads 'name', new version reads 'full_name'
-- After migration, previous version fails with "column name not found"
ALTER TABLE users RENAME COLUMN name TO full_name;

-- SAFE: Add new column, keep old column until previous version is not running
ALTER TABLE users ADD COLUMN full_name TEXT;
UPDATE users SET full_name = name WHERE full_name IS NULL;
-- Deploy new version (reads full_name, writes both)
-- After rollback window expires (48 hours): drop old column
ALTER TABLE users DROP COLUMN name;

-- BREAKS ROLLBACK: Adding NOT NULL without default
-- Previous version writes rows without the new column
-- After migration, previous version's writes fail the NOT NULL constraint
ALTER TABLE orders ADD COLUMN fulfillment_priority INT NOT NULL;

-- SAFE: Add with default, remove default later if needed
ALTER TABLE orders ADD COLUMN fulfillment_priority INT NOT NULL DEFAULT 1;
-- Previous version's writes receive the default
-- After rollback window: the default can be removed if it is no longer needed

-- BREAKS ROLLBACK: Dropping a column the previous version reads
-- After migration, previous version fails with "column status not found"
ALTER TABLE orders DROP COLUMN status;

-- SAFE: Deprecate first, drop after rollback window
-- In the current deployment: stop reading 'status' in new version
-- In the next deployment: drop the column (no version in production still reads it)
ALTER TABLE orders DROP COLUMN status;  -- Only in the NEXT deployment

Designing rollback into deployment configuration

Rollback is most reliable when it is automated and fast. The configuration that enables this:

Blue-green deployment. Two identical production environments — blue and green — alternate as the active environment. Deployment goes to the inactive environment. After validation, traffic is switched. Rollback is a traffic switch back to the previous environment, taking seconds.

# AWS Application Load Balancer — switch traffic between blue and green
# Blue is currently live, green is the new deployment

# After deploying to green and validating:
aws elbv2 modify-rule \
  --rule-arn arn:aws:elasticloadbalancing:us-east-1:123456789:listener-rule/... \
  --actions Type=forward,TargetGroupArn=arn:...:targetgroup/green-tg

# If green has a problem, rollback takes seconds:
aws elbv2 modify-rule \
  --rule-arn arn:aws:elasticloadbalancing:us-east-1:123456789:listener-rule/... \
  --actions Type=forward,TargetGroupArn=arn:...:targetgroup/blue-tg

The key requirement for blue-green to support rollback: both environments must be able to run against the same database schema. This is exactly the backward-compatible migration discipline applied at the infrastructure level.

Canary deployments. New version receives a small percentage of traffic. Monitoring determines whether to proceed with full rollout or revert. Rollback affects only the canary percentage.

# Canary routing — Kubernetes deployment with weighted traffic split
# Values are weight percentages; routing happens at the load balancer level
CANARY_CONFIG = {
    "stable": {
        "weight": 90,  # 90% of traffic goes to stable version
        "image": "app:v2.3.1"
    },
    "canary": {
        "weight": 10,  # 10% of traffic goes to canary
        "image": "app:v2.3.2"
    }
}

# Rollback: reduce canary weight to 0
ROLLBACK_CONFIG = {
    "stable": {"weight": 100, "image": "app:v2.3.1"},
    "canary": {"weight": 0, "image": "app:v2.3.2"}
}

Rollback window. Every deployment has an explicit rollback window — the period during which rollback is a valid option. After the window closes, the previous version's compatibility shims can be removed and migrations can proceed with operations that would have broken the previous version. The window duration depends on the deployment's risk profile: high-risk changes get 72 hours; low-risk changes get 24 hours.

Stateful rollback: the harder problem

Application rollback is straightforward: revert to the previous image, restart. Database rollback is harder. Migrations applied forward must be reversed.

For migrations that added columns or tables, reversal is straightforward:

-- Forward migration (in the rollback window)
ALTER TABLE users ADD COLUMN email_preferences JSONB DEFAULT '{}';

-- Rollback migration
ALTER TABLE users DROP COLUMN email_preferences;

For migrations that modified existing data, reversal requires preserving the previous data:

-- Forward migration
-- Splits full_name into first_name and last_name
ALTER TABLE users ADD COLUMN first_name TEXT;
ALTER TABLE users ADD COLUMN last_name TEXT;
UPDATE users SET
    first_name = SPLIT_PART(full_name, ' ', 1),
    last_name = SPLIT_PART(full_name, ' ', 2);

-- Rollback migration — only works if full_name is still present
-- (was NOT dropped in the forward migration)
ALTER TABLE users DROP COLUMN first_name;
ALTER TABLE users DROP COLUMN last_name;
-- full_name is still there — backward-compatible migration design

The migration that splits full_name into first_name and last_name but retains full_name is backward-compatible. The previous version still reads full_name. Rollback succeeds. The team can iterate on the split logic and deploy a new forward migration when they have confidence in it.

The migration that drops full_name during the split is not backward-compatible. Rollback requires a reverse migration that reconstructs full_name from first_name and last_name — which may lose data if the concatenation is lossy. That reverse migration is the 11 PM emergency script.

The operational habit that changes the calculus

Teams that never practice rollback learn that rollback fails exactly when it is most needed: under pressure, during incidents, at 11 PM. Teams that practice rollback regularly — by including it in deployment runbooks, by validating backward compatibility in staging before every deployment, and by explicitly designing the rollback procedure when designing the forward migration — learn that rollback is reliable when it is needed.

Common mistakes that prevent successful rollback

Coupling the migration to the deployment. When migrations run in the same pipeline step as the application deployment, they cannot be independently controlled. A deployment failure leaves the schema in an unknown state — the migration may have run before the application failed to start, or the deployment may have partially succeeded. Decoupling migrations from deployments — running migrations as a separate step with explicit success verification before starting the application deployment — gives the team control over each phase independently.

Not testing rollback in staging. Teams that plan to roll back but never test rollback learn that rollback fails at the worst possible time. The rollback procedure should be tested in staging as part of the deployment runbook for any significant change. The test: apply the migration, deploy the new version, verify it works, then roll back to the previous version and verify the previous version works against the migrated schema. If this test fails in staging, the deployment design must be changed before going to production.

Assuming rollback is equivalent to reverting the deployment. Application rollback (reverting to the previous image) is fast and reliable. Database rollback (reverting migrations) is not equivalent. Database rollback requires a reverse migration that undoes the forward migration. If the forward migration deleted data, the reverse migration cannot recover it. If the forward migration changed data formats, the reverse migration must convert back — and the conversion may be lossy. The rollback design must include a reverse migration plan that accounts for these cases.

Letting the rollback window expire without explicit closure. If the rollback window is 48 hours but nobody explicitly tracks when it closes, cleanup steps (removing backward-compatibility code, dropping old columns) may be delayed indefinitely. The previous version's compatibility shims accumulate in the codebase. The correct practice is to create an explicit task at deployment time for the cleanup steps, scheduled at the window's close, with a specific engineer assigned.

Deploying with feature flags disabled as a rollback mechanism. Feature flags provide a fast way to disable a new code path without reverting the deployment. This is a valid operational tool, but it is not equivalent to rollback. Disabling a feature flag does not revert data changes, schema changes, or infrastructure changes made by the new code. A feature flag rollback leaves the new code deployed, with its side effects present. Database rollback undoes the side effects. For changes that have significant schema or data effects, a flag disable is a stabilization measure, not a rollback.

What rollback readiness looks like as a team practice

A team with mature rollback readiness has the following in place before any significant deployment:

Documented rollback procedure. The deployment runbook includes specific steps for rolling back: which image to deploy, which reverse migrations to run (in what order), how to verify the rollback succeeded. This is written before the deployment, not during an incident.

Backward-compatibility verified in staging. The previous application version is confirmed to start and pass smoke tests against the migrated database schema. This verification is a gate in the deployment pipeline, not an optional check.

Revert metrics defined. The team has agreed on which metrics, at which thresholds, trigger an automatic rollback recommendation. Error rate above 1%? Latency p99 above 2 seconds? 503 rate above 0.1%? The thresholds are set before the deployment. During the deployment window, the on-call engineer watches these metrics — not raw logs — and makes the rollback decision based on predefined criteria, not intuition under pressure.

Rollback window explicitly tracked. The rollback window has a specific end time. When it closes, cleanup tasks are scheduled and assigned. The compatibility shims are removed. The old code paths are deleted. The next deployment does not inherit the complexity of the current deployment's transition state.

The engineering discipline is to treat rollback not as a fallback that might work but as a first-class requirement that must be designed, implemented, and tested before the deployment ships. A deployment that cannot be rolled back is a deployment that must be forward-fixed at incident time. At 11 PM on a Friday, the choice between "rollback in two minutes" and "forward fix under pressure" has an obvious correct answer.

Encoding rollback requirements in the deployment process

Rollback readiness should be a gate in the deployment pipeline, not a checklist item that is easy to skip. A deployment pipeline that enforces rollback requirements before advancing to production:

# Example deployment pipeline with rollback gates
stages:
  - name: migration
    steps:
      - run: flyway migrate -url=$STAGING_DB_URL
      - run: |
          # Verify backward compatibility: previous version must start against migrated schema
          docker run --rm \
            --env DATABASE_URL=$STAGING_DB_URL \
            app:$PREVIOUS_VERSION \
            ./bin/health_check.sh
        on_failure: fail_pipeline  # If previous version fails, block deployment

  - name: rollback_plan
    steps:
      - run: |
          # Require rollback procedure to be documented in the PR description
          if ! grep -q "## Rollback" "$PR_DESCRIPTION_FILE"; then
            echo "ERROR: Deployment PR must include a Rollback section"
            exit 1
          fi

  - name: deploy_canary
    steps:
      - run: deploy_canary.sh --percentage=10
      - run: |
          sleep 300  # Wait 5 minutes
          ./bin/check_canary_metrics.sh --error-threshold=0.5 --latency-p99=2000
        on_failure: rollback_canary.sh

  - name: deploy_full
    steps:
      - run: deploy_full.sh
      - run: create_rollback_window_task.sh --hours=48 --assignee="$DEPLOYING_ENGINEER"

The pipeline enforces what is otherwise a practice: backward compatibility is checked before deployment, the rollback procedure is documented before deployment, and the rollback window task is created automatically. Engineers who skip the documentation step are blocked by the pipeline, not by social norms.

Practicing rollback before incidents require it

A team that has never practiced rollback in a non-incident context will execute it poorly in an incident. The rollback procedure for each type of deployment should be practiced at least quarterly — in staging, with the actual previous version and the actual migration:

  1. Apply the migration to staging
  2. Deploy the new version to staging
  3. Verify the new version works
  4. Execute the rollback procedure against staging
  5. Verify the previous version works against the migrated schema

The practice run identifies gaps in the procedure before they matter. A reverse migration that takes 45 minutes in staging is a 45-minute outage window in production — which may be acceptable for some systems and unacceptable for others. That calculation is better made at practice time than at 11 PM under pressure.

Teams that maintain a deployment runbook for every significant release, practice rollback quarterly, and track the rollback window explicitly consistently achieve faster incident recovery times than teams that rely on tribal knowledge and hope. The difference is not capability — it is preparation.

Comments

No comments yet. Be the first!

Sign in to leave a comment.