Secrets management that survives key rotation without causing outages

By Tomás Ferreira · 21 July 20260 views
#devops#security#secrets#backend#infrastructure

A startup's security team mandates quarterly API key rotation for all production credentials. The engineering lead looks at the system: API keys are injected as environment variables into three services, stored in GitHub Actions secrets under eight workflow names, hardcoded in the Terraform state as outputs, and mentioned in two developers' local .env files. Rotating one key requires eight places to update, coordination across three teams, and a deployment window where the new key must propagate before the old one is revoked.

They schedule a two-hour rotation window. It takes four hours. One service misses the new key, causes a 20-minute outage, and requires an emergency deployment.

This scenario plays out on engineering teams across every industry. The problem is not that rotation is hard — rotation is hard because the secrets architecture was never designed with rotation in mind. Most systems are built to work, not built to be changed cleanly. By the time a rotation is required, the secret has leaked into six different systems and the team is discovering this for the first time.

Secrets management that handles rotation cleanly is designed for rotation from the start — not retrofitted when rotation becomes required.

The root cause: secrets without ownership

The engineering lead's rotation problem started before the first secret was ever committed. It started when the team made an implicit assumption: that secrets are configuration, and configuration can live wherever is convenient.

This assumption produces secret sprawl. A Stripe key starts as a single entry in GitHub Secrets. Then someone needs it for local development and adds it to their .env. A new Terraform module needs to output it for another service. CI needs it in four different workflow files. An emergency fix puts it directly in a Kubernetes manifest. Six months later, no one knows where all the copies are.

The root cause of painful rotation is not a process failure. It is an architectural failure: the secret has no single owner, no canonical location, and no mechanism to enumerate where it lives.

The core principle: secrets from a single source of truth

When the same secret exists in multiple places (GitHub Secrets, environment variables, Terraform state, developer .env files), rotation requires updating all of them — with no automated way to verify completeness. A single missed location causes an outage.

The architecture that handles rotation cleanly:

  • Secrets stored in one secrets manager (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault)
  • Applications fetch secrets at startup, not hardcoded in configuration
  • Rotation updates the secrets manager — nothing else
// WRONG: Secret as environment variable — requires coordination to rotate
const apiKey = process.env.STRIPE_SECRET_KEY;

// BETTER: Secret fetched from secrets manager at startup
import { SecretManagerServiceClient } from '@google-cloud/secret-manager';

const client = new SecretManagerServiceClient();

async function getSecret(secretName: string): Promise<string> {
  const [version] = await client.accessSecretVersion({
    name: `projects/my-project/secrets/${secretName}/versions/latest`
  });
  return version.payload!.data!.toString();
}

// At startup:
const stripeKey = await getSecret('stripe-secret-key');

Cloud Functions in Firebase can access Google Cloud Secret Manager:

// Cloud Function — access secrets at startup (not per-invocation)
import { defineSecret } from 'firebase-functions/params';
import { onRequest } from 'firebase-functions/v2/https';

// Declare the secret — Firebase manages the Secret Manager reference
const stripeKey = defineSecret('STRIPE_SECRET_KEY');

export const handlePayment = onRequest(
  { secrets: [stripeKey] },  // Function has access to this secret
  async (req, res) => {
    const stripe = new Stripe(stripeKey.value());  // Retrieved from Secret Manager
    // ...
  }
);

When the Stripe key is rotated in Secret Manager, redeploying the function (or waiting for the next cold start) picks up the new value. No other changes are needed.

What teams try first (and why it fails)

The first response to painful rotation is usually a spreadsheet. Document every place each secret lives, assign an owner, create a checklist for rotation day. This works once, maybe twice. Then someone creates a new workflow without updating the spreadsheet. A service gets deployed to a new environment and the secret is added without telling anyone. The spreadsheet drifts from reality.

The second response is a wrapper script. Write a script that reads from the secrets manager and injects values into all the other places — GitHub Secrets, Kubernetes ConfigMaps, environment variable files. This is better, but it still requires running the script, still requires knowing all the locations, and still fails if a location is added after the script was written.

Both approaches treat the symptom. The fix is eliminating the other locations entirely. Every system that holds a copy of the secret is a liability. The goal is reducing the number of canonical locations to one.

Rotation without downtime: versioned secrets

Rotating a secret causes a window where the old key is revoked but some instances are still using it. Avoiding this requires a brief period where both old and new keys are valid.

# Google Cloud Secret Manager — rotation with version management

# 1. Add new secret version (old version still active)
echo -n "sk_new_key_123" | gcloud secrets versions add stripe-secret-key --data-file=-

# 2. Update applications to use 'latest' version
# (If applications use 'latest', they automatically pick up the new version)
# Wait for all instances to restart/refresh

# 3. Test that the new key works
curl -X GET "https://api.stripe.com/v1/customers" \
  -H "Authorization: Bearer sk_new_key_123"

# 4. Disable the old version (not delete — useful for audit trail)
gcloud secrets versions disable stripe-secret-key --version=1
# Old version is now inactive — any remaining instances using it will fail
# (This should not happen if step 2 was given enough time)

For Cloud Functions using defineSecret, the function uses the version pinned at deploy time unless configured to use latest. Redeploy after adding a new version:

firebase functions:secrets:set STRIPE_SECRET_KEY
# Enter the new value when prompted
# Firebase stores it in Secret Manager and updates the function reference on next deploy

firebase deploy --only functions:handlePayment

The key insight with versioned secrets is that rotation becomes a non-event. The new version is added while the old is still active. Applications pick up the new version on their next restart or redeployment. Only after confirming the new version is serving traffic does the old version get disabled. There is no coordination window, no simultaneous cutover, no "rotation day."

Some external services support concurrent key validity natively. Stripe lets multiple API keys be active simultaneously. The rotation sequence for services with this capability:

# Rotation for services that support concurrent validity

# 1. Create the new key in the external service's dashboard
# 2. Add new key to Secret Manager as a new version
gcloud secrets versions add stripe-secret-key --data-file=<(echo -n "sk_new_key_new")

# 3. Deploy applications — they pick up the new key
firebase deploy --only functions

# 4. Verify new key is working (monitor logs for auth errors)
gcloud logging read "resource.type=cloud_function AND severity>=ERROR" --limit=50

# 5. Revoke the old key in the external service
# (Nothing in infrastructure needs to change — Secret Manager already has new version)

Detecting secret sprawl before rotation

Before rotating a secret, find all the places where the old value might exist:

# Search the codebase for hardcoded secret values (before committing)
# Using git-secrets or detect-secrets
pip install detect-secrets
detect-secrets scan > .secrets.baseline
detect-secrets audit .secrets.baseline

# GitHub Actions: scan for secrets in workflow files
grep -r "STRIPE_SECRET_KEY" .github/workflows/  # Look for direct references

# Check if secret is in environment variables in CI/CD
gh secret list  # Lists GitHub Actions secrets — rotation requires updating each

The audit should happen before the rotation plan, not during it. Finding a new location during an active rotation event is when mistakes happen. Build the audit into the rotation preparation phase: two weeks before the rotation, scan for sprawl and eliminate it.

A more thorough scan checks git history, not just the current state:

# Scan git history for committed secrets (even if removed in a later commit)
git log --all --full-history -- '*.env' | head -20
git log -p --all | grep -E 'STRIPE_SECRET|sk_live_' | head -20

# If a secret was ever committed, consider it compromised
# Rotation is required regardless of whether it was "fixed" in a later commit

GitHub Actions secrets cannot be fetched from a centralized secrets manager without a custom action. For organizations that rotate secrets frequently, consider replacing GitHub Secrets with OIDC authentication:

# .github/workflows/deploy.yml
# Use OIDC to authenticate to Google Cloud — no secrets to rotate
permissions:
  id-token: write
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Authenticate to Google Cloud
        uses: google-github-actions/auth@v2
        with:
          workload_identity_provider: 'projects/123/locations/global/workloadIdentityPools/github/providers/github'
          service_account: '[email protected]'

      - name: Deploy
        run: gcloud run deploy my-service --image gcr.io/my-project/my-image

OIDC tokens are issued per-workflow-run and never need rotation. The service account's permissions are managed in Google Cloud IAM — no secret key is ever stored in GitHub.

Developer local environments

Local .env files are outside the team's secret management, which means they are not rotated when production secrets are rotated. This creates a divergence that causes "it works on my machine" issues after rotation.

# Approach 1: Local development uses emulators (no real secrets)
firebase emulators:start  # All Firebase services emulated locally

# Approach 2: Local development uses a separate set of non-production keys
# Rotate production keys independently of development keys
# Development keys have no access to production data

# Approach 3: Developers fetch secrets from the secrets manager using their personal credentials
gcloud secrets versions access latest --secret=stripe-test-key
# Personal access is audited and can be revoked per-user

Approach 1 is the most secure: developers never need real credentials. Approach 2 reduces the blast radius of a compromised developer machine — leaked development keys can't access production. Approach 3 is the most ergonomic when developers need access to real external services.

The anti-pattern is Approach 0: everyone shares the same production API key in a shared .env file stored in a private Slack channel. This key is never rotated because no one knows who has a copy. When a developer leaves the team, the key should be rotated — but no one does because the rotation process is too painful.

Terraform state and infrastructure secrets

Terraform is an often-overlooked source of secret sprawl. Terraform state files record every resource attribute, including sensitive outputs marked with sensitive = true. State files stored in S3 or GCS may have broad read access within the organization.

# WRONG: Secrets in Terraform outputs
output "stripe_api_key" {
  value     = var.stripe_api_key
  sensitive = true  # Prevents display in CLI, but still in state file
}

# BETTER: Secrets stored in Secret Manager, referenced by ID
resource "google_secret_manager_secret" "stripe_key" {
  secret_id = "stripe-secret-key"
  replication {
    auto {}
  }
}

# Applications reference the secret ID, not the value
output "stripe_key_secret_id" {
  value = google_secret_manager_secret.stripe_key.secret_id
}
# Applications fetch the actual value from Secret Manager at runtime

When secrets are only referenced by their Secret Manager ID in Terraform, the state file contains only the resource metadata — not the secret value. Rotation in Secret Manager requires no Terraform changes.

Common mistakes

Mistake 1: Using latest version in production for all secrets. Using latest means a new secret version is picked up as soon as the application restarts. This is convenient for rotation but dangerous for other reasons: a misconfigured secret version immediately affects production. Consider pinning to a specific version in production and having an automated pipeline promote a new version after validation.

Mistake 2: Deleting old secret versions immediately. Keep disabled versions for at least 30 days. If the new secret is invalid (wrong value, wrong format), being able to re-enable the old version prevents a service outage while the correct value is retrieved.

Mistake 3: Not auditing secret access. Secret Manager logs every access to a secret version. Enabling Cloud Audit Logs for Secret Manager creates a record of which service account accessed which secret version and when. This is essential for post-incident analysis.

# Enable audit logging for Secret Manager
gcloud projects get-iam-policy my-project > policy.yaml
# Add Secret Manager audit log config to policy.yaml
# ...
gcloud projects set-iam-policy my-project policy.yaml

# Query audit logs after a rotation
gcloud logging read \
  'protoPayload.serviceName="secretmanager.googleapis.com"
   AND protoPayload.methodName="google.cloud.secretmanager.v1.SecretManagerService.AccessSecretVersion"' \
  --limit=50

Mistake 4: Skipping rotation because "the key was never compromised." Rotation is not just a response to compromise — it is a defense against undetected compromise. A key that has never been rotated may have been accessed by a threat actor who is patiently exfiltrating data. Regular rotation limits the window any compromised key is useful.

The forward-looking picture: automated rotation

Mature secrets management does not require a "rotation day." It relies on automated rotation that runs on a schedule without human involvement.

AWS Secrets Manager supports native rotation using Lambda functions. GCP Secret Manager's rotation notifications trigger Pub/Sub messages that can invoke Cloud Functions to perform the actual rotation:

// Cloud Function triggered by Secret Manager rotation notification
import { onMessagePublished } from 'firebase-functions/v2/pubsub';
import { SecretManagerServiceClient } from '@google-cloud/secret-manager';

const client = new SecretManagerServiceClient();

export const rotateStripeKey = onMessagePublished(
  'secret-manager-rotation',
  async (event) => {
    const secretName = event.data.message.attributes['secretName'];
    const version = event.data.message.attributes['secretVersion'];

    if (!secretName.includes('stripe')) return;  // Only handle Stripe keys

    // 1. Generate a new Stripe API key via the Stripe API
    const newKey = await createNewStripeKey();

    // 2. Add the new key as a new version
    await client.addSecretVersion({
      parent: secretName,
      payload: { data: Buffer.from(newKey) },
    });

    // 3. Disable the old version
    await client.disableSecretVersion({ name: `${secretName}/versions/${version}` });

    // 4. Notify monitoring that rotation completed
    console.log(`Rotated ${secretName} — new version active`);
  }
);

With automated rotation, the quarterly rotation window disappears. Secrets rotate continuously on a schedule. Compromise of a key is self-limiting: within the rotation period, the old key becomes invalid regardless.

The engineering lead's rotation difficulty was a symptom of a secrets architecture that was not designed for rotation. Consolidating secrets into Secret Manager, using defineSecret for Cloud Functions, and adopting OIDC for CI/CD reduced the next quarterly rotation from a four-hour coordinated event to a 20-minute task: update the secret in Secret Manager, redeploy the affected services, verify they start cleanly. Automated rotation is the next step — making even that 20-minute task unnecessary.

Comments

No comments yet. Be the first!

Sign in to leave a comment.

Secrets management that survives key rotation without causing outages — ANN Tech