p24-infra Docs

encryption-key-rotation

8 min read

Playbook: Encryption Key Rotation (Last Resort — Confirmed Exposure Only)

Covers: SUPABASE_JWT_SECRET · IAM_ENC_KEY · N8N_ENCRYPTION_KEY · WEBHOOK_SECRET

THESE ARE LAST-RESORT ROTATIONS.

Do NOT rotate on schedule. These keys are designated static per credential-rotation-policy.md. Rotate only on confirmed exposure (value seen in logs, committed to git, or exposed in chat).

Impact of rotation:

  • SUPABASE_JWT_SECRET — all user sessions invalidated immediately (signing key; no data at rest)
  • IAM_ENC_KEY — encrypts data at rest; a bare key-swap orphans ALL existing ciphertext. Requires a dual-key read code change + a re-encryption pass FIRST — see its dedicated section below (#6561/#6615)
  • N8N_ENCRYPTION_KEY — all n8n stored credentials become unreadable; all workflows halt
  • WEBHOOK_SECRET — webhook HMAC validation fails until both producer and consumer are updated

Overview

KeySOPS fileImpact
SUPABASE_JWT_SECRETwhatsupAll signed JWTs for p24-wa-a invalidated; users logged out
IAM_ENC_KEYwhatsuppgcrypto passphrase for data at rest — bare swap orphans all p24_wap_user_storage ciphertext; needs dual-key read + re-encryption pass first
N8N_ENCRYPTION_KEYn8n-bms4All n8n stored credentials (OAuth tokens, API keys) unreadable
WEBHOOK_SECRETwhatsupIn-flight webhook calls fail HMAC validation

Tier: ❌ Tier 3 Manual — keys are generated locally but require a maintenance window and coordinated rollout.

Pre-rotation requirements (all keys)

  • Open a GH issue: sec: rotate KEY_NAME — confirmed exposure YYYY-MM-DD (label: critical)
  • Maintenance window scheduled and stakeholders notified
  • Rollback plan confirmed before making any change
  • For N8N_ENCRYPTION_KEY: n8n credential vault exported (see Step 1 below)

SUPABASE_JWT_SECRET

Effect: All active p24-wa-a user sessions are immediately terminated. Users must re-authenticate. This key only signs JWTs — it does not encrypt any data at rest, so a straight update → redeploy is safe. (Contrast IAM_ENC_KEY below, which encrypts data at rest and must not be rotated this way.)

Steps

  1. Schedule maintenance window — notify users: “All sessions will terminate at <time>.”

  2. Generate new key:

# 32-byte random value — never print this variable
$env:NEW_JWT = [Convert]::ToBase64String([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))
# Store to .env.local for handoff; clear $env:NEW_JWT after SOPS update

  1. Distribute: Update secrets/whatsup.env.sops — key: SUPABASE_JWT_SECRET. Follow docs/playbooks/sops-edit-operations.md.

  2. Deploy: Commit → merge → secrets-sync.yml (or manual wrangler secret put / Vercel env for the whatsup consumers — whatsup.env.sops is excluded from secrets-sync.yml’s automated push) → restart all JWT-signing consumers:

docker compose restart p24-wa-a  # or equivalent service name

  1. Verify: New user login succeeds. Old session tokens return 401.

  2. Rollback if broken: git revert the SOPS commit → re-deploy → restart.

IAM_ENC_KEY

⛔ DO NOT “just update SOPS and restart”. IAM_ENC_KEY is not a signing key — it is the pgcrypto passphrase that encrypts data at rest. Every provisioned user’s Wasabi IAM credentials are stored as pgcrypto ciphertext (pgp_sym_encrypt) in p24_wap_user_storage.iam_access_key / .iam_secret_key. Swapping the key without a re-encryption pass first makes pgp_sym_decrypt(..., NEW_KEY) throw “Wrong key or corrupt data” on every existing row — permanently orphaning all ciphertext (the OLD key is gone once SOPS is overwritten). This is irreversible user-data loss, not a session reset. The previous version of this playbook lumped IAM_ENC_KEY in with SUPABASE_JWT_SECRET under a bare “update SOPS → restart” recipe; applied literally that recipe is the outage, not the fix (#6561, #6615).

Consumers

  • Live read/write: p24-wa-app-api Cloudflare Worker (repo radieu/whatsup-android-chat-puller), deployed via wrangler secret put IAM_ENC_KEY. It decrypts via the Postgres accessor p24_wap_get_user_iam_keys (pgcrypto pgp_sym_decrypt).
  • Data at rest: p24_wap_user_storage rows in shared Supabase project mwkqmgadqnkkihjdeqsi.
  • whatsup.env.sops is excluded from secrets-sync.yml’s automated push — deployment is manual.

Prerequisite: a code change must land FIRST (dual-key read)

Do not attempt this rotation until p24-wa-app-api supports a dual-key read: try the current IAM_ENC_KEY first, fall back to IAM_ENC_KEY_OLD on a decrypt failure. Without it, the cutover has an unavoidable window where either the old or the new ciphertext cannot be read, forcing a hard maintenance window and making rollback impossible once rows are re-encrypted.

  • Tracking (whatsup): radieu/whatsup-android-chat-puller — “Add dual-key IAM_ENC_KEY read (NEW then _OLD fallback) to wa-app-api” (opened as the #6561 spinoff).
  • Why it matters for automation: with the dual-key read shipped, the rotation drops from IRREDUCIBLY-HUMAN (hard maintenance window, no clean rollback) to NEEDS-CODE-CHANGE-FIRST → AUTONOMOUS (zero-downtime, rollback-safe): during the window both the OLD and NEW keys are valid decrypt paths, so re-encryption can proceed row-by-row with no user-facing read failures and can be aborted at any point.

Method — pgcrypto re-encryption inside a single Postgres transaction

The re-encryption never lets plaintext leave Postgres. Both passphrases are passed transiently as SQL arguments (bound parameters / SET LOCAL, never logged, never written to a column). The whole pass is one transaction that only commits if 100 % of rows verify under the NEW key.

⛔ apply_migration / supabase db push are BANNED against shared project mwkqmgadqnkkihjdeqsi (#5653/#5645) — they corrupt the shared schema_migrations ledger. The re-encryption RPC/migration therefore must ship through the whatsup per-repo migration ledger in radieu/whatsup-android-chat-puller (its own supabase/migrations/ + CI runner), not as a p24-infra secret-manager job and not via any p24-infra migration path. See supabase-per-repo-migration-ledger.md.

Phase 0 — ciphertext backup (zero mutation, Gate A). Dump the current iam_access_key / iam_secret_key ciphertext columns (still OLD-key-encrypted) for every row in p24_wap_user_storage to an off-DB backup. This is the restore point — keep the OLD key retained until the window closes and Phase 3 has verified.

Phase 1 — dry-run decrypt under OLD key (zero mutation, Gate A). For all N rows, run pgp_sym_decrypt(col, OLD_KEY) inside a read-only transaction and count successes. Gate A: proceed only if all N rows decrypt cleanly under the OLD key. Any failure here means the OLD key or a row is already corrupt — STOP and investigate before mutating anything.

Phase 2 — re-encrypt OLD → NEW in one transaction (the only mutating step).

BEGIN;
-- OLD_KEY / NEW_KEY supplied transiently as bound args — never inlined into logged SQL, never printed.
UPDATE p24_wap_user_storage
SET iam_access_key = pgp_sym_encrypt(pgp_sym_decrypt(iam_access_key, :OLD_KEY), :NEW_KEY),
    iam_secret_key = pgp_sym_encrypt(pgp_sym_decrypt(iam_secret_key, :OLD_KEY), :NEW_KEY)
WHERE iam_access_key IS NOT NULL;
 
-- Phase 3 verify (same transaction): every row must decrypt under NEW_KEY.
-- Count rows where NEW-key decrypt fails; must be 0.
SELECT count(*) AS failures
FROM p24_wap_user_storage
WHERE iam_access_key IS NOT NULL
  AND (pgp_sym_decrypt(iam_access_key, :NEW_KEY) IS NULL
    OR pgp_sym_decrypt(iam_secret_key, :NEW_KEY) IS NULL);
 
-- COMMIT only if failures = 0; otherwise ROLLBACK — the whole pass is atomic.
COMMIT;   -- or: ROLLBACK;

Because it is one transaction, a partial failure rolls back to the exact pre-rotation ciphertext — no half-migrated table.

Phase 3 — verify (inside the same transaction, before COMMIT). The failures count above must be exactly 0 across all N rows. COMMIT only then; any non-zero count → ROLLBACK and abort. (With the dual-key read live, a rollback is invisible to users — the Worker keeps reading via the still-valid OLD key.)

Full ordered runbook

  1. Land + verify the dual-key read (IAM_ENC_KEY → IAM_ENC_KEY_OLD fallback) in p24-wa-app-api. Gate — do not start the rotation until this is deployed.
  2. Generate NEW_IAM (32-byte random, never printed):
$env:NEW_IAM = [Convert]::ToBase64String([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))
  1. Set IAM_ENC_KEY_OLD = the current live value and IAM_ENC_KEY = NEW_IAM in secrets/whatsup.env.sops (secret-manager role; follow sops-edit-operations.md), then wrangler secret put both to p24-wa-app-api. The Worker now reads NEW, falls back to OLD → all existing (OLD-encrypted) rows still decrypt. Zero downtime.
  2. Phase 0 + Phase 1 (backup + Gate A dry-run) via the whatsup per-repo migration ledger.
  3. Phase 2 + Phase 3 — atomic re-encrypt + verify; COMMIT only on 100 %.
  4. Retire the OLD key: once Phase 3 has committed and the Worker is confirmed reading via NEW, remove IAM_ENC_KEY_OLD from whatsup.env.sops and wrangler secret delete IAM_ENC_KEY_OLD.
  5. Rollback (only before step 6 completes): the dual-key read means the Worker still serves reads via IAM_ENC_KEY_OLD; ROLLBACK/revert the re-encryption, restore ciphertext from the Phase 0 backup if needed, keep OLD live. After step 6 there is no clean rollback — that is why OLD is retained through the whole window.

Blockers checklist (all must be true before rotating)

  • Dual-key read shipped + verified in p24-wa-app-api (whatsup repo)
  • Re-encryption migration authored in the whatsup per-repo ledger (NOT apply_migration on mwkqmgadqnkkihjdeqsi)
  • Phase 0 ciphertext backup taken; OLD key retained for the whole window
  • Someone who can reach/verify the p24-wa-app-api CF Worker deploy is coordinating the cutover

N8N_ENCRYPTION_KEY

Effect: All stored n8n credentials become unreadable. Every workflow using a stored credential fails immediately. This is a major operation requiring full credential re-entry.

Steps

  1. Export credential vault BEFORE changing the key:
ssh root@100.86.202.85
docker exec -it n8n n8n export:credentials --all \
  --output=/home/node/creds-backup-$(date +%Y%m%d).json
docker cp n8n:/home/node/creds-backup-*.json /opt/backups/

Keep this backup — it is encrypted with the OLD key.

  1. Generate new encryption key:
NEW_KEY=$(openssl rand -hex 16)  # 32-char hex — never echo

  1. Update secrets/n8n-bms4.env.sops — key: N8N_ENCRYPTION_KEY. Follow docs/playbooks/sops-edit-operations.md.

  2. Deploy and restart n8n:

ssh root@100.86.202.85 "cd /opt/n8n && docker compose stop n8n && docker compose up -d n8n"

  1. Re-enter all credentials in n8n UI. All existing credentials will show as broken. Use the exported JSON (decryptable with the old key) as a reference for the original values. Re-enter each credential manually via n8n UI → Credentials.

  2. Verify: Trigger each critical workflow manually; confirm credentials load without errors.

  3. Rollback if n8n fails to start: Revert SOPS commit → redeploy → credentials from backup.

WEBHOOK_SECRET

Effect: All in-flight webhook calls fail HMAC validation until producer and consumer are both updated.

Steps

  1. Identify all producer and consumer endpoints for this secret before proceeding.

  2. Generate new secret:

$env:NEW_WH = ([BitConverter]::ToString([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))).Replace("-","").ToLower()

  1. Coordinate atomic rollout — during low-traffic window:

    • Update secrets/whatsup.env.sops — key: WEBHOOK_SECRET
    • Restart consumer first, then producer to minimize validation failures

  2. Verify: Send a test webhook and confirm successful HMAC validation.

Log rotation

| YYYY-MM-DD HH:MM UTC | #ISSUE | N8N_ENCRYPTION_KEY | confirmed exposure | radieu | n8n-bms4.env.sops — full credential migration completed |

References

  • secret-rotation-access-matrix.md — Tier 3 danger notes
  • docs/playbooks/sops-edit-operations.md — SOPS write pattern
  • docs/playbooks/static-api-key-incident-rotation.md — exposure incident response
  • docs/playbooks/credential-rotation-policy.md — why these keys are static

Graph View

Backlinks