p24-infra Docs

pinbox24-credential-rotation

4 min read

Playbook: Pinbox24 W3/W4 Credential Rotation

Last updated: 2026-07-06 Applies to: bms-1 — W3 (v32-prod, s3-v32-prod) and W4 (v42-prod, s3-v42-prod) Trigger: Credential exposure, 90-day rotation schedule, or security incident

1. Overview

After this CI/CD PR merges, credential rotation is fully automated:

Edit SOPS file → PR → merge to main
→ secrets-sync.yml fires automatically
→ deploys pinbox24-w{3,4}.env to bms-1
→ updates build-dir env files via Python
→ force-recreates containers
→ verifies no auth errors (W3 only)

Manual steps are only needed if CI/CD is unavailable or partially fails.

2. SOPS key mapping

W3 — secrets/pinbox24-w3.env.sops

SOPS keyEnv fileEnv varContainer
V32_MONGODB_URLbackend-environment.envMONGODB_URLv32-prod
V32_PMONGODB_URLbackend-environment.envPMONGODB_URLv32-prod
V32_DB_URIs3-environment.envDB_URIs3-v32-prod
V32_s3Bucket_api_accessKeyIdboth env filess3Bucket_api_accessKeyIdboth
V32_s3Bucket_api_secretAccessKeyboth env filess3Bucket_api_secretAccessKeyboth

W4 — secrets/pinbox24-w4.env.sops

All V42_ prefixed keys map to the same key name without the prefix (V42_NEW_MONGODB_URI → NEW_MONGODB_URI) in backend-environment.env. The Python deploy script strips the V42_ prefix and updates all matching keys.

Key credential keys:

SOPS keyEnv varNotes
V42_NEW_MONGODB_URINEW_MONGODB_URIW4 main MongoDB
V42_v3MongoUrlv3MongoUrlW3 DB backward-compat reads
V42_RABBIT_MQ_URLRABBIT_MQ_URLRabbitMQ AMQP
V42_REDIS_PASSWORDREDIS_PASSWORDredis-v42 (local)
V42_JWT_TOKEN_SECRETJWT_TOKEN_SECRETJWT signing
V42_CONTAINER_NAMECONTAINER_NAMEUsed by docker-compose variable substitution

3. Automated rotation (standard flow)

Step 1 — Edit SOPS file

# Windows dev
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
 
# W3
sops secrets\pinbox24-w3.env.sops
 
# W4
sops secrets\pinbox24-w4.env.sops

Edit the values directly in the SOPS interactive editor.

Step 2 — Canary verify

sops --decrypt --input-type dotenv --output-type dotenv secrets\pinbox24-w3.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }

Step 3 — Commit and merge

git add secrets/pinbox24-w3.env.sops
git commit -m "chore: rotate W3 MongoDB credentials — w3_app"
# PR → merge to main → secrets-sync.yml fires automatically

Step 4 — Monitor CI

  • Watch sync-pinbox24-w3 / sync-pinbox24-w4 jobs in GitHub Actions
  • “Deploy W3 containers” step verifies zero auth errors before completing
  • Check infra_operations Supabase table for deploy log entry

4. Manual rotation (CI unavailable)

Use this if secrets-sync.yml is broken or unavailable.

W3 manual procedure

# 1. Decrypt W3 SOPS silently
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:V32_MONGODB_URL = (sops --decrypt --input-type dotenv --output-type dotenv secrets\pinbox24-w3.env.sops | Select-String "^V32_MONGODB_URL=").ToString().Split("=",2)[1]
$env:V32_PMONGODB_URL = (sops --decrypt --input-type dotenv --output-type dotenv secrets\pinbox24-w3.env.sops | Select-String "^V32_PMONGODB_URL=").ToString().Split("=",2)[1]
$env:V32_DB_URI = (sops --decrypt --input-type dotenv --output-type dotenv secrets\pinbox24-w3.env.sops | Select-String "^V32_DB_URI=").ToString().Split("=",2)[1]
 
# 2. Write update script to temp file (no secrets in script content)
# 3. SSH to bms-1 and run update script (see pinbox24-w3-operations.md §Correct procedure)
# 4. Force-recreate containers on bms-1

Step 4 must RECREATE the containers, never docker restart them. docker restart reuses the environment baked into each container at creation time and does not re-read the on-disk env file — so a restarted container keeps the OLD credential even though backend-environment.env / mailgun-environment.env on disk was just updated. Only docker-compose up -d --force-recreate (after the old container is stopped/removed) picks up the new value. Recreate every container that consumes the rotated MongoDB credential in the same operation:

  • W3 (w3_app): v32-prod
  • W4 (w4_app): v42-prod, mailgun-v42-prod, and s3-v2-v42-prod

Skipping this recreate is exactly what caused the 2026-07-10 silent mailgun outage (mailgun-mongodb-stale-credential-hang.md): the w4_app password was corrected on disk but mailgun-v42-prod was never recreated, so every inbound webhook hung on a dead MongoDB connection and ~450 emails produced zero records with no error. Confirm the recreate took with the sha256 env-hash comparison from that playbook (never prints the secret). The per-consumer steps are automated in w4-mongodb-credential-rotation.md / w3-mongodb-credential-rotation.md.

Reference: docs/playbooks/pinbox24-w3-operations.md §Correct procedure: update credentials

5. Security rules for credentials

MongoDB URIs must use app users, not admin

StackMongoDB userauthSource
W3 (v32-prod, s3-v32-prod)w3_appw3_db
W4 (v42-prod, s3-v42-prod)w4_appw4_db

Never use admin credentials in container env files.

Redis password (W4 only)

redis-v42 is a local bms-1 container. Password rotation requires:

  1. Update V42_REDIS_PASSWORD in secrets/pinbox24-w4.env.sops
  2. Also update the redis-v42 container’s Redis password (via redis-cli CONFIG SET requirepass <new> while live, or recreate with --requirepass)
  3. Both must change atomically — v42-prod and redis-v42 must use the same password

MongoDB URI URL-encoding

MongoDB passwords with special characters must be URL-encoded in the connection URI:

from urllib.parse import quote
encoded_password = quote(raw_password, safe='')
uri = f"mongodb://w3_app:{encoded_password}@host/db?authSource=w3_db"

The SOPS file stores the full pre-encoded URI — not the raw password.

6. Rotation schedule

CredentialRotation intervalOwner
MongoDB w3_app90 daysp24-infra sys-security
MongoDB w4_app90 daysp24-infra sys-security
Redis password (redis-v42)90 daysp24-infra sys-security
JWT_TOKEN_SECRET180 days or on exposurep24-infra sys-security
Wasabi IAM keysautomated via wasabi-iam-rotatorwasabi-iam-rotator (Vercel)
PayU, Przelewy24, Twilioon exposure onlymanual (external providers)

7. Incident response

If a credential is suspected exposed:

  1. Create GH issue immediately: [Security] W3/W4 <key-name> — suspected exposure
  2. Rotate the credential in the external system first (MongoDB admin console / provider dashboard)
  3. Edit SOPS file with new value
  4. Create PR — merge immediately (skip normal review queue if P1)
  5. Confirm CI deploys successfully (check “Deploy W3/W4 containers” step)
  6. Verify no service errors after deploy
  7. Update GH issue with resolution

PM2 log exposure: if /var/log/v32-prod/pm2/ or /var/log/v42-prod/pm2/ logs showed a MongoDB URI with a password, rotate that credential immediately following steps above. The log files will contain the old password in plain text — treat them as sensitive until log rotation clears them.

Reference: docs/playbooks/static-api-key-incident-rotation.md

Graph View