p24-infra Docs

w3-mongodb-credential-rotation

16 min read

Playbook: w3.pinbox24.com — MongoDB w3_app Credential Rotation

Created: 2026-07-02 Incident origin: 2026-07-02 — after rotating w3_app password on 2026-07-01 15:31, password contained special characters requiring URL-encoding. mongojs tolerated the raw password; mongoose rejected it — all operations on collections forms, langs, widgets timed out (10 s each). Issue reference: see docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md Fix C.

Scope

ItemValue
Domainw3.pinbox24.com / api.w3.pinbox24.com
Backend containerv32-prod on bms-1 (94.23.26.113)
Compose working dir on bms-1/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/ — confirm live before use, see warning below
Env file on bms-1<compose-dir>/backend-environment.env (i.e. /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env)
MongoDB clusterrs0 on bms-2 (145.239.133.104) and bms-3 (51.68.155.224)
Application userw3_app with readWrite on w3_db
Admin credentialssecrets/bms-servers.env.sops — keys MONGODB_RS0_ADMIN_USER / MONGODB_RS0_ADMIN_PASSWORD

⚠️ Confirm the live compose working dir before editing any file (#4985). The authoritative path is the one the sync-pinbox24-w3 job in .github/workflows/secrets-sync.yml deploys to — currently /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/ (matched by the GitLab CI config, docs/bms-1/gitlab-ci-w3.yml → BUILDER_DIR). The GitLab-CI runner owns this directory and its build-hash segment can change if the runner is re-registered.

A separate root-owned /root/builds/pn3C9eHo/0/pinbox24/p24-v-3.2/ copy still exists on bms-1 and is NOT the prod serving copy — editing it has zero effect on v32-prod (#4985, discovered during #4925, which wasted a rotation on it). Do not edit it for prod changes, and do not delete it either: an on-server audit (2026-08-01) found it is an active W3-staging mount source — see docs/w3-w4-stack-operations.md §3.

Always ground-truth the real directory first, then use it in place of the path above:

ssh root@94.23.26.113 "docker inspect v32-prod --format '{{index .Config.Labels \"com.docker.compose.project.working_dir\"}}'"

Two-connection architecture (CRITICAL)

v32-prod maintains two independent MongoDB connections — both must be updated simultaneously. Updating only one still causes 422/504 errors on all endpoints:

Env varLibraryParser strictness
MONGODB_URLmongojsLiberal — tolerates un-encoded special characters
PMONGODB_URLMongooseStrict URL parser — rejects un-encoded @, !, =, #, ?, /, +

Lesson from 2026-07-02 incident: if PMONGODB_URL is not updated (or password not URL-encoded), mongoose disconnects immediately on every PM2 restart. Collections forms, langs, and widgets time out after 10 s even though mongojs appears connected.

Consumer map — recreate EVERY w3_app consumer in the same rotation (#5153)

Audited read-only on bms-1 2026-08-02 (#5153, docker inspect env — passwords never printed). The w3_app credential is consumed by more containers than the two originally documented. A rotation must recreate every container that authenticates as w3_app@w3_db, or the missed ones keep the dead password baked into their process env and fail silently (the mailgun-mongodb-stale-credential-hang.md failure class).

In the authoritative GitLab-CI compose — automated by the rotation script

/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/docker-compose.yml (confirm the build-hash dir live, #4985) defines five services; four consume w3_app:

ServiceContainerenv_fileKeysServes
backendv32-prodbackend-environment.envMONGODB_URL, PMONGODB_URL, DB_URIapi.w3.pinbox24.com
s3s3-v32-prods3-environment.envDB_URIinternal (test-net)
resov32-prod-resobackend-environment.env (shared with backend)MONGODB_URL, PMONGODB_URL, DB_URIw3.reso-integration-addrecords.pinbox24.com
socketv32-prod-socketbackend-environment.env (shared with backend)MONGODB_URL, PMONGODB_URL, DB_URIsocket.w3.pinbox24.com

Because reso and socket share backend-environment.env with backend, the Step 3 file write already updates their password on disk — but Step 4 must force-recreate all four together (backend s3 reso socket). scripts/rotate/w3_app_mongodb_password.sh does this automatically as of #5153; before that it recreated only backend s3, silently stranding reso/socket.

Out-of-band consumers — MANUAL handling required (not automated)

ContainerWhy not automatedRotation action
s3-v32-stageSeparate compose /opt/p24-infra/bms-1/docker-compose-w3-stage.yml (service s3); shares the prod w3_app credential (DB_URI, authSource=w3_db)After rotation, recreate manually: cd /opt/p24-infra/bms-1 && docker-compose -f docker-compose-w3-stage.yml up -d --force-recreate s3. Architecture flag: staging sharing the prod credential is drift — track a separate staging credential (#5153 Ask #3).
s3-v32-prod-renamedOrphan docker run container (no compose project, RestartPolicy=no), created 2026-08-01 during incident response; consumes w3_app via DB_URICannot be force-recreated by a compose command. Before rotating, decide with a human: remove it (docker rm -f s3-v32-prod-renamed if confirmed unneeded) or bring it under compose. Do not leave it running on a stale credential.

NOT w3_app consumers — rotation-safe, leave alone (audited #5153)

  • cron-v32-prod, cron-v32-prod-reso, cron-v32-prod-socket (/home/p24-server-scripts/v3/**): no MongoDB env vars at all — they do not connect to w3_db directly, so a w3_app rotation cannot break them. The pre-audit assumption that these consume w3_app was incorrect.
  • s3-v32-prod-reso, s3-v32-prod-socket (/home/p24-server-scripts/v3/v3reso|v3socket): DB_URI uses authSource=admin with a non-w3_app user — unaffected by a w3_app rotation. ⚠️ sys-security follow-up: an app-tier s3 microservice authenticating against admin violates security-policy §1 (“no service connects as MongoDB admin; w3 services must use authSource=w3_db”). File a security-labelled issue for sys-security review (#5153 Ask #3).

Deploy-path drift (#5153 Ask #3 — architect decision). /home/p24-server-scripts/v3/** is a parallel, undocumented compose path distinct from the GitLab-CI build dir. None of its containers consume w3_app, so it does not block a w3_app rotation — but it is accumulated drift worth a consolidation review. Flag to the human/architect; do not auto-decommission.

Trigger — when to rotate

  • Scheduled rotation (~every 90 days)
  • Suspected credential exposure (team member offboarding, accidental log leak, git history)
  • Security audit finding
  • The MongoDB rs0 admin password was rotated and w3_app URIs must be rebuilt

Prerequisites

Before starting, ensure you have:

  1. Age key accessible: C:\Users\konar\.age\p24-infra-keys.txt
  2. SSH access to bms-1 (root@94.23.26.113) and bms-2/bms-3 via key C:\Users\konar\.ssh\id_ed25519
  3. mongosh installed on bms-2 or bms-3 (already present)

Step 0 — Find the current rs0 PRIMARY

Always run password changes on the PRIMARY — they replicate automatically to secondaries.

ssh root@145.239.133.104 "mongosh --quiet --eval 'rs.isMaster().ismaster'"
# true  => bms-2 (145.239.133.104) is PRIMARY
# false => bms-3 (51.68.155.224) is PRIMARY

Note the PRIMARY IP as $PRIMARY_HOST for subsequent steps.

Step 1 — Generate a safe new password

Generate a password that contains no URI-unsafe characters (@, #, /, ?, +, =, !, %). Alphanumeric plus - and _ are always safe and require no URL-encoding.

# On local Windows dev machine — store in env var, never print
$rawBytes = [System.Security.Cryptography.RandomNumberGenerator]::GetBytes(24)
$env:W3_APP_NEW_PASSWORD = [System.Convert]::ToBase64String($rawBytes) -replace '[^A-Za-z0-9_-]', ''
Write-Host "Password length: $($env:W3_APP_NEW_PASSWORD.Length)"
# Minimum 20 chars — regenerate if shorter due to character stripping

If the password contains special characters (e.g., assigned by a password manager), URL-encode it before embedding in any URI (see Step 3). Mongoose will reject un-encoded chars.

Step 1b — Acquire the rotation lock FIRST — MANDATORY (ADR 004)

Step 2b’s db.changeUserPassword('w3_app', …) on rs0 is a non-idempotent live server-side write, so per ADR 004 this rotation MUST hold the per-secret advisory lock before that write. Two sessions rotating w3_app at once each set a different password and silently diverge (live rs0 vs SOPS/bms-1 env) — the #5925 race. SOPS-only edits don’t need this; this live password change does.

Run it from the repo root so the repo-relative script path resolves on the Windows dev machine (the absolute /opt/p24-infra/... path does not exist there):

$env:LOCK_ID = python scripts/rotation-log-entry.py acquire --secret W3_APP_MONGODB_PASSWORD --repo p24-infra --reason "#<issue>"
if ($LASTEXITCODE -ne 0) { throw "acquire denied/unverified — STOP, reconcile, do NOT rotate w3_app" }
  • Exit 3 = another session holds it → stop and reconcile (do not blind-overwrite). Exit 4 = DB unreachable → fail-closed, do not proceed. $LOCK_ID is the pending row — do not also open one. On a Linux worker run LOCK_ID=$(python3 scripts/rotation-log-entry.py acquire … ) with || exit 1 instead.

Then immediately before Step 2b’s changeUserPassword re-check the lock is still held:

python scripts/rotation-log-entry.py check $env:LOCK_ID
if ($LASTEXITCODE -ne 0) { throw "lock lost — abort the live write and reconcile" }

Release after distribution (Step 4 / verification): rotation-log-entry.py close "$LOCK_ID" --sops-commit "$(git rev-parse HEAD)" (or fail on failure). CLI reference: secret-manager-rotation-log.md.

Step 2 — Rotate w3_app password in MongoDB rs0

2a. Read admin credentials silently from SOPS

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
 
$decrypted = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\bms-servers.env.sops
 
$env:MG_ADMIN_PASS = ($decrypted | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").ToString().Split("=",2)[1]
$env:MG_ADMIN_USER = ($decrypted | Select-String "^MONGODB_RS0_ADMIN_USER=").ToString().Split("=",2)[1]
$decrypted = $null

2b. Change the w3_app password in MongoDB

Connect to PRIMARY via SSH. Pass the mongosh script via a temp file to avoid exposing values in process arguments:

$PRIMARY_HOST = "145.239.133.104"  # or 51.68.155.224 — from Step 0
 
# Write mongosh script to temp file (UTF-8 no BOM, LF endings for Linux)
$mongoScript = "use admin`ndb.auth('$($env:MG_ADMIN_USER)', '$($env:MG_ADMIN_PASS)')`nuse w3_db`ndb.changeUserPassword('w3_app', '$($env:W3_APP_NEW_PASSWORD)')`nprint('password_changed_ok')"
$tmpScript = "C:\Users\konar\AppData\Local\Temp\w3_rotate.js"
[System.IO.File]::WriteAllText($tmpScript, $mongoScript.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
 
# Copy and run on PRIMARY
scp $tmpScript "root@${PRIMARY_HOST}:/tmp/w3_rotate.js"
$result = ssh root@$PRIMARY_HOST "mongosh --quiet /tmp/w3_rotate.js; rm /tmp/w3_rotate.js"
 
if ($result -notmatch "password_changed_ok") {
    throw "Password change FAILED — review output: $result"
}
Write-Host "w3_app password changed on MongoDB rs0 PRIMARY"
 
# Clear admin creds immediately after use
$env:MG_ADMIN_PASS = ""; $env:MG_ADMIN_USER = ""
Remove-Item $tmpScript -Force -ErrorAction SilentlyContinue

2c. Verify new credentials authenticate

$verifyScript = "use w3_db`ndb.auth('w3_app', '$($env:W3_APP_NEW_PASSWORD)')`nprint('auth_ok')"
$tmpVerify = "C:\Users\konar\AppData\Local\Temp\w3_verify.js"
[System.IO.File]::WriteAllText($tmpVerify, $verifyScript.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
 
scp $tmpVerify "root@${PRIMARY_HOST}:/tmp/w3_verify.js"
$verifyResult = ssh root@$PRIMARY_HOST "mongosh --quiet /tmp/w3_verify.js; rm /tmp/w3_verify.js"
 
if ($verifyResult -notmatch "auth_ok") {
    throw "Verification FAILED — w3_app auth rejected with new password."
}
Write-Host "w3_app auth verified on PRIMARY"
Remove-Item $tmpVerify -Force -ErrorAction SilentlyContinue

Step 3 — Update env file on bms-1

3a. URL-encode the new password

Always encode — even if the password looks safe. Mongoose uses a strict URL parser.

$encoded = [System.Uri]::EscapeDataString($env:W3_APP_NEW_PASSWORD)

3b. Check current env file (key names only — do not print values)

ssh root@94.23.26.113 "grep -E '^(MONGODB_URL|PMONGODB_URL)=' /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env | cut -d= -f1"
# Expected output (both keys must exist):
#   MONGODB_URL
#   PMONGODB_URL

Also note any extra query parameters in the current URIs so you can preserve them:

# Safe — extract only the query string portion (after the database name), not the password
ssh root@94.23.26.113 "grep '^MONGODB_URL=' /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env | sed 's|.*w3_db||'"
# e.g., outputs: ?replicaSet=rs0&authSource=w3_db

3c. Back up the env file before editing

ssh root@94.23.26.113 "cp /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env.bak.$(date +%Y%m%d_%H%M%S)"

3d. Update both MONGODB_URL and PMONGODB_URL via Python on bms-1

Use Python to do the replacement server-side — the new password travels over the SSH session (encrypted) rather than appearing in shell command arguments.

# Build the new URI values (adjust query string if needed from Step 3b)
$qs = "?replicaSet=rs0&authSource=w3_db"
$newMongoVal  = "mongodb://w3_app:${encoded}@145.239.133.104,51.68.155.224/w3_db${qs}"
$newPMongoVal = "mongodb://w3_app:${encoded}@145.239.133.104,51.68.155.224/w3_db${qs}"
 
# Write a Python update script locally
$pyScript = @"
import sys
env_path = '/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env'
new_mongo_url  = sys.argv[1]
new_pmongo_url = sys.argv[2]
with open(env_path, 'r') as f:
    lines = f.readlines()
updated = []
for line in lines:
    if line.startswith('MONGODB_URL='):
        updated.append('MONGODB_URL=' + new_mongo_url + '\n')
    elif line.startswith('PMONGODB_URL='):
        updated.append('PMONGODB_URL=' + new_pmongo_url + '\n')
    else:
        updated.append(line)
with open(env_path, 'w') as f:
    f.writelines(updated)
print('env_updated_ok')
"@
 
$tmpPy = "C:\Users\konar\AppData\Local\Temp\update_w3_env.py"
[System.IO.File]::WriteAllText($tmpPy, $pyScript.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
scp $tmpPy root@94.23.26.113:/tmp/update_w3_env.py
 
$result = ssh root@94.23.26.113 "python3 /tmp/update_w3_env.py '$newMongoVal' '$newPMongoVal'; rm /tmp/update_w3_env.py"
if ($result -notmatch "env_updated_ok") {
    throw "Env file update FAILED"
}
Write-Host "backend-environment.env updated on bms-1"
 
# Clear sensitive vars
$encoded = ""; $newMongoVal = ""; $newPMongoVal = ""
Remove-Item $tmpPy -Force -ErrorAction SilentlyContinue

Manual alternative (if scripting fails): SSH to bms-1, then:

NEW_ENCODED=$(python3 -c "import urllib.parse; print(urllib.parse.quote('RAW_PASSWORD', safe=''))")
sed -i "s|^MONGODB_URL=.*|MONGODB_URL=mongodb://w3_app:${NEW_ENCODED}@145.239.133.104,51.68.155.224/w3_db?replicaSet=rs0\&authSource=w3_db|" \
  /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env
sed -i "s|^PMONGODB_URL=.*|PMONGODB_URL=mongodb://w3_app:${NEW_ENCODED}@145.239.133.104,51.68.155.224/w3_db?replicaSet=rs0\&authSource=w3_db|" \
  /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env

Step 4 — Recreate (not restart) ALL FOUR w3_app containers — MANDATORY

docker restart does NOT re-read the host env file. A running container has the MONGODB_URL / PMONGODB_URL values baked into its process environment at creation time; docker restart reuses that baked-in environment, so it keeps the OLD credential even after you edited backend-environment.env on disk. Only fully recreating the container (docker-compose up -d after the old one is stopped/removed, adding --force-recreate if the image tag is unchanged) picks up the new env file. Skipping this recreate leaves the container silently authenticating with a dead credential — the exact failure class documented in mailgun-mongodb-stale-credential-hang.md (a w4_app rotation where mailgun-v42-prod was never recreated, so every webhook hung and ~450 emails produced zero MongoDB records with no error). Recreate every container that consumes the w3_app credential in the same rotation operation — see the Consumer map above. In the GitLab-CI compose that is all four of backend s3 reso socket (not just backend s3; #5153). reso and socket share backend-environment.env with backend, so the Step 3 file write already updated their password on disk — they just need the recreate to load it.

ssh root@94.23.26.113 "cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2 && CONTAINER_NAME=v32-prod IMAGE_NAME=563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod docker-compose up -d --no-build --force-recreate backend s3 reso socket"

Out-of-band consumers (s3-v32-stage, s3-v32-prod-renamed) are NOT recreated by the command above — they live outside this compose project. Handle them per the Consumer map → Out-of-band consumers table before declaring the rotation complete (#5153).

Wait ~15 seconds for PM2 to start all workers before verifying.

Confirm the fresh container actually loaded the new credential (never prints the secret):

# Hashes must MATCH after recreation. Differing hashes = still stale = recreate did not take.
grep "^MONGODB_URL=" /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env | sha256sum
docker exec v32-prod sh -c "cat /proc/1/environ | tr '\0' '\n' | grep '^MONGODB_URL=' | sha256sum"

Do NOT use docker exec v32-prod pm2 restart --update-env as the permanent fix. Env vars injected this way are lost on the next docker restart. Always edit the env file and recreate the container as shown above.

Step 5 — Verify the application is healthy

5a. HTTP endpoint check (primary signal)

Start-Sleep -Seconds 15
 
$r = Invoke-WebRequest "https://api.w3.pinbox24.com/api/i18n/langs" -TimeoutSec 15 -UseBasicParsing
Write-Host "Status: $($r.StatusCode)"
# Expected: 200
 
$body = $r.Content | ConvertFrom-Json
Write-Host "Language entries: $($body.Count)"
# Expected: > 0 (actual language data, not empty array)
Status codeMeaning
200Both MONGODB_URL and PMONGODB_URL are working
422mongojs (MONGODB_URL) is failing — check encoding or auth
504Mongoose (PMONGODB_URL) is timing out — check PMONGODB_URL encoding or auth
502Container is not reachable — check if v32-prod is running

5b. PM2 logs inside the container

ssh root@94.23.26.113 "docker exec v32-prod tail -40 /var/log/v32-prod/pm2/pm2_v32-prod_production_out.log"
# Good signs: "Mongoose connected"
# Bad signs:  "Mongoose disconnected" — Mongoose auth still failing
# Docker-level startup / crash errors
ssh root@94.23.26.113 "docker logs v32-prod --tail 30 2>&1"

Do NOT run docker exec v32-prod pm2 env 0 without filtering — it prints all env vars including MONGODB_URL and PMONGODB_URL in plaintext. Safe alternative: docker exec v32-prod pm2 env 0 | grep -i mongo | sed 's/value: .*/value: REDACTED/'

NEVER add grep context flags (-A / -B / -C) to a pm2 env grep. Every line of pm2 env output is a distinct KEY: value secret, so a context window pulls in neighbouring variables’ full values, not related lines. Use only an exact anchored match on the key name(s) (grep -i mongo above), never -A1/-B1 “just to inspect the format” — that is exactly how RABBIT_MQ_URL

  • CONVERT_API leaked on the parallel W4 stack (#4966: pm2 env 0 | grep -i -A1 -B1 'NODE_ENV' captured adjacent vars’ values). To see the format, redact first with the sed above.

Step 6 — Update SOPS with the new w3_app password

After confirming the application is healthy, record the new password in the canonical secret store.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$secretsDir = "C:\code_2026\p24-infra\secrets"
 
$plain = sops --decrypt --input-type dotenv --output-type dotenv "$secretsDir\bms-servers.env.sops"
$tempPath = "$secretsDir\bms-servers-edit.env.sops"
 
# Replace existing key or append if it does not exist yet
$keyExists = $plain | Where-Object { $_ -match '^W3_APP_MONGODB_PASSWORD=' }
$updated = $plain | ForEach-Object {
    if ($_ -match '^W3_APP_MONGODB_PASSWORD=') {
        "W3_APP_MONGODB_PASSWORD=$($env:W3_APP_NEW_PASSWORD)"
    } else { $_ }
}
if (-not $keyExists) {
    $updated += "W3_APP_MONGODB_PASSWORD=$($env:W3_APP_NEW_PASSWORD)"
}
 
[System.IO.File]::WriteAllText($tempPath, ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
$enc = sops --encrypt --input-type dotenv --output-type dotenv $tempPath
[System.IO.File]::WriteAllText("$secretsDir\bms-servers.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
# Canary — MANDATORY before git add
sops --decrypt --input-type dotenv --output-type dotenv "$secretsDir\bms-servers.env.sops" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS canary FAILED — do NOT commit" }
Write-Host "SOPS canary OK"
 
Remove-Item $tempPath -Force -ErrorAction SilentlyContinue
$env:W3_APP_NEW_PASSWORD = ""

Commit via PR (do not push directly to main):

cd C:\code_2026\p24-infra
git add secrets\bms-servers.env.sops
git commit -m "chore: rotate w3_app MongoDB password (w3.pinbox24.com)"
# Then: gh pr create --base main --title "chore: rotate w3_app MongoDB password"

Step 7 — Log the rotation

Append to docs/secrets-rotation-log.md:

| YYYY-MM-DD HH:MM UTC | w3.pinbox24.com | W3_APP_MONGODB_PASSWORD | <reason> | <your-name> | yes |

Rollback — if Step 5 verification fails

Option A — Temporary injection (fastest, survives PM2 restart only)

# Restore from backup env file created in Step 3c
ssh root@94.23.26.113 "ls /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env.bak.*"
# Pick the latest backup timestamp, e.g. backend-environment.env.bak.20260702_153100
 
ssh root@94.23.26.113 "cp /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env.bak.20260702_153100 /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env"

Then restart the container (Step 4) and verify (Step 5).

Option B — Full rollback (revert MongoDB password + env file)

  1. Connect to MongoDB PRIMARY (Step 0)
  2. Change w3_app password back to the old value (Step 2, using the old password from SOPS or backup)
  3. Restore env file from backup (Step 3c backup) or rebuild URIs with old password (Step 3d)
  4. Restart container (Step 4)
  5. Verify (Step 5)

Escalation

SymptomAction
mongosh auth fails for w3_app with new passwordCheck replication lag — connect to the SECONDARY and confirm the password change replicated. If lag > 60 s, wait and retry.
Mongoose still disconnecting after env update + restartConfirm BOTH keys were updated: ssh root@94.23.26.113 "grep -c 'MONGODB_URL' /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env" — expect 2.
422 (not 504) on /api/i18n/langsmongojs (MONGODB_URL) is failing — check encoding of that key specifically.
504 but /api/auth POST returns 401/422Mongoose (PMONGODB_URL) is timing out — check encoding of PMONGODB_URL.
SOPS canary fails after writeDo NOT commit. See docs/playbooks/sops-windows-crlf.md for recovery.
rs0 PRIMARY unknown (both hosts return false to rs.isMaster())Election in progress — wait 30 s and retry. If persists > 2 min, check network between bms-2, bms-3, bms-4. See docs/playbooks/mongodb-credential-rotation.md §Step 0.
w3_app user missing from MongoDB after rotation attemptThe user must be recreated: use w3_db; db.createUser({user:'w3_app',pwd:'NEWPWD',roles:[{role:'readWrite',db:'w3_db'}]}) on PRIMARY with admin auth.

Prevention

  1. Generate passwords without URI-unsafe characters. Use the method in Step 1 or openssl rand -base64 18 | tr -d '+/=' on Linux. Eliminates the URL-encoding requirement entirely.
  2. Always update both MONGODB_URL and PMONGODB_URL simultaneously. They are independent library connections — updating only one causes hard-to-diagnose partial failures.
  3. Back up backend-environment.env before every edit (Step 3c). A timestamped backup makes rollback a one-liner.
  4. Verify with the HTTP endpoint (/api/i18n/langs), not just PM2 logs. PM2 may show “Mongoose connected” while mongojs is still failing, causing 422 on all data endpoints.
  5. Never rely on pm2 restart --update-env as the permanent fix. Env vars injected via docker exec are lost on docker restart. Always edit the env file and recreate the container.
  6. Add a calendar reminder for the next rotation (~90 days) immediately after completing this one.
  • docs/playbooks/mongodb-credential-rotation.md — rs0 admin and prometheus user rotation
  • docs/playbooks/mailgun-mongodb-stale-credential-hang.md — the silent-hang incident that makes Step 4 (recreate, not restart) mandatory: a rotation where the container was never recreated to pick up the new credential
  • docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md — full outage diagnosis (incl. Fix C: two-connection problem)
  • docs/playbooks/mongodb-exporter-uri-special-chars.md — URI special character encoding for exporters
  • docs/playbooks/sops-windows-crlf.md — SOPS write corruption on Windows (BOM/CRLF)

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="W3_APP_MONGODB_PASSWORD",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — w3_app MongoDB password rotated on rs0 and v32-prod env updated",
    env="bms-2",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'W3_APP_MONGODB_PASSWORD', 'success', 'Scheduled rotation — w3_app MongoDB password rotated on rs0 and v32-prod env updated', 'bms-2')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks