Infrastructure as code drift: how it accumulates and how to contain it

By Tomás Ferreira · 21 July 20260 views
#devops#cloud#terraform#iac#reliability

A DevOps engineer runs terraform plan on infrastructure that has been running in production for 14 months. The plan shows 43 changes. Nothing in the Terraform code was modified. The changes come from 14 months of manual fixes applied directly to the console, emergency modifications made during incidents, and environment-specific overrides that were never codified.

Applying the plan would revert those 43 changes. Some of them are critical fixes. None of them are documented. The Terraform code cannot be safely applied to production, which means the infrastructure is effectively undocumented and unreproducible. If the team needs to create a new environment — a staging environment, a disaster recovery copy, or a migration to a new region — they cannot do it reliably. The infrastructure exists in one place, maintained by a combination of code and institutional memory.

This is where drift leads. It begins with a single manual change during an incident that nobody has time to codify. It compounds across months until the gap between code and reality is too wide to close without risk.

How drift accumulates

Console modifications during incidents. An on-call engineer adjusts a Cloud Function's memory or timeout to resolve a production issue. The fix works. The incident is closed. The Terraform code is never updated. The next terraform apply reverts the fix.

Feature deployments that bypass IaC. A product team enables a new Firebase feature directly in the Firebase console. The Terraform resource for Firebase does not include the feature. The Terraform state diverges.

Environment-specific exceptions. Production has a manually-added firewall rule that staging does not. The exception was applied by hand and never documented. Reproducing production in a new region would not include the rule.

Abandoned terraform state. Resources are created with terraform apply, then deleted manually from the console. Terraform still has the resource in state, producing errors when it tries to manage a resource that no longer exists.

Gradual configuration drift. A Cloud Run service's environment variables are updated via gcloud CLI during a troubleshooting session. The variables are important but were never added to the Terraform resource. Three months later, a new team member runs terraform apply and removes the variables.

Each individual drift event seems minor. The cumulative effect is a system whose documented state cannot be trusted and whose production state cannot be reproduced.

Detecting drift

Terraform's terraform plan is the primary drift detector — it shows differences between the current state and the desired configuration. Running it regularly (not just before deployments) surfaces drift before it compounds:

# Run terraform plan in CI on a schedule — detect drift without applying changes
terraform plan -detailed-exitcode
# Exit code 0: no changes (no drift)
# Exit code 1: error
# Exit code 2: changes required (drift detected)

# In GitHub Actions:
- name: Check for infrastructure drift
  run: |
    terraform init
    terraform plan -detailed-exitcode || echo "DRIFT_DETECTED=true" >> $GITHUB_ENV

- name: Notify on drift
  if: env.DRIFT_DETECTED == 'true'
  uses: slackapi/slack-github-action@v1
  with:
    payload: '{"text":"⚠️ Infrastructure drift detected. Run `terraform plan` to review."}'
  env:
    SLACK_WEBHOOK_URL: ${{ secrets.SLACK_WEBHOOK }}

A weekly drift check workflow provides early warning before drift compounds:

# .github/workflows/drift-check.yml
name: Infrastructure Drift Check

on:
  schedule:
    - cron: '0 9 * * MON'  # Every Monday at 9am UTC
  workflow_dispatch:  # Manual trigger

jobs:
  drift-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: '1.7.0'

      - name: Authenticate to GCP
        uses: google-github-actions/auth@v2
        with:
          credentials_json: ${{ secrets.GCP_CREDENTIALS }}

      - name: Terraform Init
        run: terraform init
        working-directory: infrastructure/

      - name: Check for drift
        id: plan
        run: |
          terraform plan -detailed-exitcode -out=tfplan 2>&1 | tee plan-output.txt
          echo "exit_code=$?" >> $GITHUB_OUTPUT
        working-directory: infrastructure/
        continue-on-error: true

      - name: Create issue on drift
        if: steps.plan.outputs.exit_code == '2'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const planOutput = fs.readFileSync('infrastructure/plan-output.txt', 'utf8');
            
            await github.rest.issues.create({
              owner: context.repo.owner,
              repo: context.repo.repo,
              title: `Infrastructure drift detected — ${new Date().toISOString().split('T')[0]}`,
              body: `## Terraform plan detected changes\n\n\`\`\`\n${planOutput.slice(0, 4000)}\n\`\`\`\n\nReview and codify or apply the changes.`,
              labels: ['infrastructure', 'drift'],
            });

For Google Cloud with Firebase:

# GCP-specific drift detection using gcloud
# Compare Firestore security rules in code vs. deployed
diff <(cat firestore.rules) <(firebase firestore:rules:get 2>/dev/null)

# Check if any Firebase Functions have settings not in code
firebase functions:list | grep -v "$(cat functions-list-expected.txt)"

Preventing drift: the emergency change protocol

The most common drift source is emergency changes. The protocol that prevents this without slowing down incident response:

  1. Apply the emergency change (manually, in the console, to resolve the incident)
  2. File a ticket immediately titled "Codify emergency change: [brief description]"
  3. Codify the change in Terraform within 24 hours (or the next business day for non-critical changes)
  4. Run terraform plan to confirm the plan is empty after codification
# Example: Emergency memory increase for a Cloud Function
# Applied in console during incident → codified in Terraform after incident

# Before (in code — pre-incident):
resource "google_cloudfunctions2_function" "process_orders" {
  name    = "process-orders"
  location = "us-central1"

  build_config {
    runtime     = "nodejs20"
    entry_point = "processOrders"
    source {
      storage_source {
        bucket = google_storage_bucket.functions.name
        object = google_storage_bucket_object.function_source.name
      }
    }
  }

  service_config {
    available_memory = "256M"    # Pre-incident value
    timeout_seconds  = 60
  }
}

# After (codified post-incident):
resource "google_cloudfunctions2_function" "process_orders" {
  # ...
  service_config {
    available_memory = "512M"    # Increased during incident — OOM at 256M under load
    timeout_seconds  = 120       # Increased — database queries were timing out at 60s
    # Added 2026-07-15 during incident PROD-2847 (order processing failures)
  }
}

The 24-hour window is important. Incidents generate adrenaline and urgency; the immediate aftermath is the right time to codify the change while the context is fresh. Waiting a week means the engineer who made the change may not remember the exact values or the reason.

Import: catching up on existing manual changes

When drift already exists, terraform import brings manually-created resources under Terraform management without destroying and recreating them:

# Import a Firebase Firestore database that was created manually
terraform import google_firestore_database.default \
  "projects/my-project/databases/(default)"

# After import, terraform plan should show no changes
# If it shows changes, update the Terraform resource to match the actual configuration

# Import Firebase Hosting site
terraform import google_firebase_hosting_site.default \
  "projects/my-project/sites/my-project"

# Import a Cloud Function
terraform import google_cloudfunctions2_function.my_function \
  "projects/my-project/locations/us-central1/functions/my-function"

# Import a Cloud Run service
terraform import google_cloud_run_v2_service.api \
  "projects/my-project/locations/us-central1/services/api"

For the 43-change scenario, importing each manually-created resource and then updating the Terraform code to match the actual configuration brings the code back into sync with reality.

After importing, terraform plan should show zero changes. If it shows changes, those changes represent the difference between what the Terraform code says the resource should look like and what the resource actually looks like. Update the code (not the resource) to match the actual configuration.

Terraform state management

The Terraform state file records what Terraform believes about the current state of infrastructure. Mismanaged state is itself a form of drift:

# backend.tf — store state remotely, not locally
terraform {
  backend "gcs" {
    bucket  = "my-project-terraform-state"
    prefix  = "terraform/state"
  }
}

Remote state storage ensures:

  • Multiple engineers share the same state view
  • State is not lost if a local machine is unavailable
  • State is locked during applies, preventing concurrent modifications

State locking prevents two engineers from running terraform apply simultaneously and corrupting the state:

# GCS backend provides automatic state locking
# If another apply is in progress:
# Error: Error locking state: Error acquiring the state lock

# To force-unlock if a lock is stuck (use carefully):
terraform force-unlock LOCK_ID

The policy: no console changes in production

The most effective drift prevention is a policy that production infrastructure changes must go through Terraform, with a narrow exception for emergencies that requires codification within 24 hours. Enforcing this policy technically — using IAM roles that prevent console modifications to managed resources — eliminates the most common drift source:

# GCP IAM — restrict direct resource modification in production
# Only the CI/CD service account (which runs Terraform) can modify most resources
gcloud projects add-iam-policy-binding my-project-prod \
  --member="serviceAccount:[email protected]" \
  --role="roles/firebase.admin"

# Human users in production have read-only access by default
gcloud projects add-iam-policy-binding my-project-prod \
  --member="group:[email protected]" \
  --role="roles/firebase.viewer"  # Cannot modify, only view

This creates a break-glass procedure for emergencies: the engineer escalates to get temporary elevated access, applies the emergency change, files the ticket, and the access is revoked. The temporary access is logged and auditable.

For teams not yet ready for full IAM enforcement, a softer approach is a pre-apply checklist in the PR template for infrastructure changes:

## Infrastructure Change Checklist

- [ ] `terraform plan` output reviewed and attached
- [ ] Change tested in staging environment first
- [ ] Rollback procedure documented
- [ ] Any manual changes from the past week codified before this PR
- [ ] `terraform plan` shows no drift after this change

Measuring drift over time

Track the number of resources in drift as a metric. A drift count that trends upward over time indicates the protocol is not being followed. A drift count near zero confirms it is working:

# Count resources with drift
terraform plan -detailed-exitcode 2>/dev/null
# Parse the output for "to change" and "to destroy" counts
terraform plan 2>/dev/null | grep -E "^\s+[~+-]" | wc -l

The DevOps engineer with 43 changes in plan resolved the drift over two weeks: importing manually-created resources, codifying emergency changes, and running weekly drift checks to catch new divergences early. The effort to codify 14 months of drift was significant — approximately 40 hours of engineer time. The benefit — infrastructure that can be reproduced, reviewed, and confidently applied — would have been free if the policy had been in place from the start.

The cost of drift is not just the codification effort. It is the risk of every terraform apply reverting a critical fix, the inability to reproduce production in a new environment, and the knowledge gap that exists when the engineers who made the manual changes leave the team. Infrastructure that exists only in the console is infrastructure that only one person understands.

Handling drift in multi-environment setups

Projects typically have multiple environments: development, staging, and production. Drift is hardest to manage when the environments diverge from each other, because staging no longer reflects production conditions.

The canonical approach is to apply Terraform changes to staging first, verify them, then apply the same configuration to production. If production has manual changes that staging does not, staging cannot be used to validate Terraform plans:

# Workflow: test in staging before production
# 1. Apply to staging
cd infrastructure/staging
terraform plan -out=staging.tfplan
terraform apply staging.tfplan

# 2. Verify the application in staging
./scripts/smoke-test.sh https://staging.myapp.com

# 3. Apply the same configuration to production
cd ../production
terraform plan -out=prod.tfplan
terraform apply prod.tfplan

Use Terraform workspaces or separate directories per environment with shared modules:

# modules/cloud-function/main.tf — shared module
variable "memory" { default = "256M" }
variable "timeout" { default = 60 }
variable "environment" {}

resource "google_cloudfunctions2_function" "function" {
  service_config {
    available_memory = var.memory
    timeout_seconds  = var.timeout
    environment_variables = {
      ENVIRONMENT = var.environment
    }
  }
}

# infrastructure/staging/main.tf
module "process_orders" {
  source      = "../../modules/cloud-function"
  memory      = "256M"
  environment = "staging"
}

# infrastructure/production/main.tf
module "process_orders" {
  source      = "../../modules/cloud-function"
  memory      = "512M"   # Production is sized differently
  environment = "production"
}

Environment-specific differences should be explicit in code, not hidden in manual changes. The Terraform code becomes the documentation for how environments differ — and why.

Common mistakes when addressing drift

Running terraform apply without reviewing the plan. A plan with 43 changes should not be applied without understanding each change. Some of those changes may revert critical fixes. The review step is not optional.

Using terraform taint to force recreation. terraform taint marks a resource for destruction and recreation on the next apply. If the resource has been modified manually and the Terraform code does not reflect the actual configuration, recreation will not produce the correct state. Fix the code first, then apply.

Ignoring terraform state mv when resources are renamed. Renaming a resource in Terraform (changing the resource identifier) causes Terraform to destroy the old resource and create a new one. For stateful resources (databases, storage buckets), this destroys production data. Use terraform state mv to rename the state entry without destroying the resource:

# Rename resource in state without destroying it
terraform state mv \
  google_cloudfunctions2_function.old_name \
  google_cloudfunctions2_function.new_name

Forgetting to update state after manual deletions. When a resource is deleted from the console without going through Terraform, Terraform still has it in state. The next terraform plan errors when it tries to read the resource. Remove it from state:

# Remove a deleted resource from Terraform state
terraform state rm google_cloudfunctions2_function.deleted_function

Drift prevention is easier than drift remediation. The effort to codify 14 months of drift was significant. The effort to prevent it from accumulating is a five-minute checklist embedded in the incident response protocol. The choice between them is made at the moment of every emergency change.

Treating drift detection as a team practice

The weekly drift check workflow is only as useful as the team's response to it. A drift detection that creates an issue which nobody acts on provides no benefit. Assign drift remediation as a rotating responsibility — the same way teams assign on-call rotations — so that every drift notification is owned by a specific engineer for that week.

Make the remediation visible in sprint planning: "codify emergency change from incident PROD-2847" is a real task with real scope, not background work that happens when someone has time. Teams that treat drift codification as invisible overhead accumulate drift; teams that make it visible in their planning cadence eliminate it.

The goal is not zero drift forever — that is unrealistic in any system that has real incidents. The goal is drift that is detected quickly, assigned to an owner, and resolved within a sprint. A system with that property is reliably reproducible, safely deployable, and fully documented in code. That is the value of infrastructure as code: not that humans never touch the console, but that every console touch is reflected in code within a predictable window.

Comments

No comments yet. Be the first!

Sign in to leave a comment.