p24-infra Docs

encryption-key-rotation

3 min read

Playbook: Encryption Key Rotation (Last Resort — Confirmed Exposure Only)

Covers: SUPABASE_JWT_SECRET · IAM_ENC_KEY · N8N_ENCRYPTION_KEY · WEBHOOK_SECRET

THESE ARE LAST-RESORT ROTATIONS.

Do NOT rotate on schedule. These keys are designated static per credential-rotation-policy.md. Rotate only on confirmed exposure (value seen in logs, committed to git, or exposed in chat).

Impact of rotation:

  • SUPABASE_JWT_SECRET / IAM_ENC_KEY — all user sessions invalidated immediately
  • N8N_ENCRYPTION_KEY — all n8n stored credentials become unreadable; all workflows halt
  • WEBHOOK_SECRET — webhook HMAC validation fails until both producer and consumer are updated

Overview

KeySOPS fileImpact
SUPABASE_JWT_SECRETwhatsupAll signed JWTs for p24-wa-a invalidated; users logged out
IAM_ENC_KEYwhatsupAll IAM-encrypted tokens become undecryptable
N8N_ENCRYPTION_KEYn8n-bms4All n8n stored credentials (OAuth tokens, API keys) unreadable
WEBHOOK_SECRETwhatsupIn-flight webhook calls fail HMAC validation

Tier: ❌ Tier 3 Manual — keys are generated locally but require a maintenance window and coordinated rollout.

Pre-rotation requirements (all keys)

  • Open a GH issue: sec: rotate KEY_NAME — confirmed exposure YYYY-MM-DD (label: critical)
  • Maintenance window scheduled and stakeholders notified
  • Rollback plan confirmed before making any change
  • For N8N_ENCRYPTION_KEY: n8n credential vault exported (see Step 1 below)

SUPABASE_JWT_SECRET / IAM_ENC_KEY

Effect: All active p24-wa-a user sessions are immediately terminated. Users must re-authenticate.

Steps

  1. Schedule maintenance window — notify users: “All sessions will terminate at <time>.”

  2. Generate new keys:

# 32-byte random values — never print these variables
$env:NEW_JWT = [Convert]::ToBase64String([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))
$env:NEW_IAM = [Convert]::ToBase64String([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))
# Store to .env.local for handoff; clear $env:NEW_* after SOPS update

  1. Distribute: Update secrets/whatsup.env.sops — keys: SUPABASE_JWT_SECRET, IAM_ENC_KEY. Follow docs/playbooks/sops-edit-operations.md.

  2. Deploy: Commit → merge → secrets-sync.yml → restart all JWT-signing consumers:

docker compose restart p24-wa-a  # or equivalent service name

  1. Verify: New user login succeeds. Old session tokens return 401.

  2. Rollback if broken: git revert the SOPS commit → re-merge → secrets-sync.yml → restart.

N8N_ENCRYPTION_KEY

Effect: All stored n8n credentials become unreadable. Every workflow using a stored credential fails immediately. This is a major operation requiring full credential re-entry.

Steps

  1. Export credential vault BEFORE changing the key:
ssh root@100.86.202.85
docker exec -it n8n n8n export:credentials --all \
  --output=/home/node/creds-backup-$(date +%Y%m%d).json
docker cp n8n:/home/node/creds-backup-*.json /opt/backups/

Keep this backup — it is encrypted with the OLD key.

  1. Generate new encryption key:
NEW_KEY=$(openssl rand -hex 16)  # 32-char hex — never echo

  1. Update secrets/n8n-bms4.env.sops — key: N8N_ENCRYPTION_KEY. Follow docs/playbooks/sops-edit-operations.md.

  2. Deploy and restart n8n:

ssh root@100.86.202.85 "cd /opt/n8n && docker compose stop n8n && docker compose up -d n8n"

  1. Re-enter all credentials in n8n UI. All existing credentials will show as broken. Use the exported JSON (decryptable with the old key) as a reference for the original values. Re-enter each credential manually via n8n UI → Credentials.

  2. Verify: Trigger each critical workflow manually; confirm credentials load without errors.

  3. Rollback if n8n fails to start: Revert SOPS commit → redeploy → credentials from backup.

WEBHOOK_SECRET

Effect: All in-flight webhook calls fail HMAC validation until producer and consumer are both updated.

Steps

  1. Identify all producer and consumer endpoints for this secret before proceeding.

  2. Generate new secret:

$env:NEW_WH = ([BitConverter]::ToString([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))).Replace("-","").ToLower()

  1. Coordinate atomic rollout — during low-traffic window:

    • Update secrets/whatsup.env.sops — key: WEBHOOK_SECRET
    • Restart consumer first, then producer to minimize validation failures

  2. Verify: Send a test webhook and confirm successful HMAC validation.

Log rotation

| YYYY-MM-DD HH:MM UTC | #ISSUE | N8N_ENCRYPTION_KEY | confirmed exposure | radieu | n8n-bms4.env.sops — full credential migration completed |

References

  • secret-rotation-access-matrix.md — Tier 3 danger notes
  • docs/playbooks/sops-edit-operations.md — SOPS write pattern
  • docs/playbooks/static-api-key-incident-rotation.md — exposure incident response
  • docs/playbooks/credential-rotation-policy.md — why these keys are static

Graph View