Playbook: w4.pinbox24.com — MongoDB Credential Rotation

Created: 2026-07-02 Origin: Derived from w3 incident 2026-07-02 — mongoose strict URL parser rejects un-encoded special characters in MongoDB URIs. v42-prod uses the same two-connection architecture as v32-prod. Issue reference: see docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md Fix C.

⚠️ Stale-user correction (2026-08-01, 3754): the v3MongoUrl / mongojs connection for v42-prod now authenticates as the dedicated read-only user w3_read_v42 on w3_db — not pinbox_production (which is retained below only as historical context). This user’s credential is not a standalone password key; it is the full connection URI stored in SOPS key V42_v3MongoUrl in secrets/pinbox24-w4.env.sops. To rotate it: changeUserPassword('w3_read_v42', …) on the rs0 PRIMARY (w3_db), rebuild the URI (host/query preserved, URI-safe hex password → no encoding), update V42_v3MongoUrl, PR to main → secrets-sync sync-pinbox24-w4 rewrites v3MongoUrl in backend-environment.env and force-recreates v42-prod. Note w3_read_v42 is read-only, so the readWrite role stated for the mongojs user below no longer applies to v42-prod.

Differences vs w3

Dimension	w3 (v32-prod)	w4 (v42-prod)
Domain	`w3.pinbox24.com`	`w4.pinbox24.com`
Container	`v32-prod`	`v42-prod`
Build path	`/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/` (authoritative; `/root/builds/pn3C9eHo/...` is stale — #4985; confirm live via `docker inspect`)	`/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/`
App language	JavaScript	TypeScript — compiled to `/app/dist/`
Mongoose env key	`PMONGODB_URL` + `MONGODB_URL` (same URL, two keys)	`NEW_MONGODB_URI`
mongojs env key	`MONGODB_URL`	`v3MongoUrl`
Mongoose user	`w3_app` on `w3_db`	`w4_app` on `w4_db`
mongojs user	`w3_app` on `w3_db`	`w3_read_v42` (read-only) on `w3_db` — historically `pinbox_production`; see correction note above (#4910)
Secrets wrapping	Direct env vars	`secrets.ts` maps `NEW_MONGODB_URI` → `MONGODB_URI` at runtime when `NODE_ENV=production`
Env file encoding	UTF-8 with BOM	ASCII — BOM risk lower but URL-encoding risk identical

Key insight: w4 has two different MongoDB users — w4_app (mongoose) and w3_read_v42 (mongojs, read-only, formerly pinbox_production). Their credentials are rotated independently (w4_app via a standalone password key; w3_read_v42 via the full URI in V42_v3MongoUrl). Rotating only w4_app does NOT rotate w3_read_v42, and vice versa. Choose which rotation to perform based on the trigger.

Scope

Item	Value
Domain	`w4.pinbox24.com` / `api.w4.pinbox24.com`
Backend container	`v42-prod` on bms-1 (`94.23.26.113`)
Second `w4_app` consumer	`mailgun-v42-prod` (bms-1) — see “mailgun-v42-prod is a second w4_app consumer” below. Must be recreated in the same operation as `v42-prod`.
Third `w4_app` consumer (read-only, indirect)	`mailgun-pipeline-exporter` (bms-1) — see “mailgun-pipeline-exporter is a third, self-healing w4_app consumer” below. No manual rotation step today — this one heals itself on the rotation’s own SOPS push.
Fourth consumer — different user, NOT `w4_app`	`et-operational-platform` (Vercel) — reads `w4_db.regRecords` via a dedicated `et_oper` MongoDB user, authSource `admin`. See “et-operational-platform (`et_oper`) is a fourth, independently-credentialed consumer” below. Not covered by any step in this playbook — rotating `w4_app` does NOT rotate or affect `et_oper`, and vice versa.
Env file on bms-1 (`v42-prod`)	`/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env`
Env file on bms-1 (`mailgun-v42-prod`)	`/root/mailgun-prod/mailgun-environment.env` (key: `MONGODB_URL`)
Env file on bms-1 (`mailgun-pipeline-exporter`)	shared `/opt/mezmo-agent/.env` (key: `MAILGUN_PIPELINE_MONGODB_URI`) — written by `secrets-sync.yml`’s `sync-bms-1` job only, never by hand
MongoDB cluster	rs0 on bms-2 (`145.239.133.104`) and bms-3 (`51.68.155.224`)
Mongoose user	`w4_app` with `readWrite` on `w4_db` — shared by `v42-prod` (`NEW_MONGODB_URI`) and `mailgun-v42-prod` (`MONGODB_URL`)
mongojs user	`w3_read_v42` with `read` (read-only) on `w3_db` (legacy v3 data, `v42-prod` only; credential = full URI in SOPS `V42_v3MongoUrl`). Formerly `pinbox_production` (#4910/#3738/#3754)
Admin credentials	`secrets/bms-servers.env.sops` — keys `MONGODB_RS0_ADMIN_USER` / `MONGODB_RS0_ADMIN_PASSWORD` — not used by `mailgun-v42-prod` since 2026-07-10 (previously was, see note below)

mailgun-v42-prod is a second w4_app consumer (added 2026-07-10)

mailgun-v42-prod (the Mailgun inbound-webhook handler) also connects directly to w4_db for its own dedup collection (mailConfirmation) — independent of the HTTP calls it makes to v42-prod’s own REST API for register-record creation. Before 2026-07-10 it used the rs0 admin (root) credential for this, via a separate secrets-sync.yml step — a needless privilege exposure (a webhook-processing container holding a root-equivalent MongoDB credential) that also meant it was rotated on a different schedule than v42-prod, and could silently drift onto a stale credential without anyone noticing (incident: docs/playbooks/mailgun-mongodb-stale-credential-hang.md, ~450 emails lost with zero visible errors).

As of 2026-07-10, mailgun-v42-prod uses the same w4_app credential and the same multi-host replicaSet URI shape as v42-prod’s own NEW_MONGODB_URI (see Step 4 below — both containers are now recreated together in a single rotation operation, so they can never drift apart again). secrets-sync.yml’s “Sync mailgun-environment.env keys” step rebuilds MONGODB_URL fresh from mongodb_w4_app_password on every CI run and its “Restart mailgun-prod” step now uses --force-recreate (previously missing — the same anti-pattern that caused the 2026-07-05 v42-prod outage, just not yet applied to this later-added mailgun step).

mailgun-pipeline-exporter is a third, self-healing w4_app consumer (added 2026-08-01, 4657)

mailgun-pipeline-exporter (bms-1, monitoring — counts w4_db.regRecords writes to detect the mailgun-v42-prod stale-credential hang, see docs/playbooks/mailgun-mongodb-stale-credential-hang.md) also connects to w4_db, read-only, using the same w4_app credential. Unlike the two consumers above, it does not need a dedicated rotation step today:

secrets-sync.yml’s sync-bms-1 job resolves MAILGUN_PIPELINE_MONGODB_URI fresh from SOPS on every push — it prefers MAILGUN_MONGODB_URL in secrets/pinbox24-backends.env.sops (not yet populated, blocked on a secret-manager session per #3850) and falls back to this playbook’s own NEW_MONGODB_URI value, V42_NEW_MONGODB_URI in secrets/pinbox24-w4.env.sops.
Because Step 3d below rewrites V42_NEW_MONGODB_URI (via NEW_MONGODB_URI/newMongoUri) and the rotation’s own commit pushes pinbox24-w4.env.sops, the very next sync-bms-1 run picks up the new password automatically, fingerprints the change, and force-recreates mailgun-pipeline-exporter — no extra manual step for this consumer during a w4_app rotation.
Once MAILGUN_MONGODB_URL is added to secrets/pinbox24-backends.env.sops (the intended canonical, least-privilege source — same w4_app user, sourced from the same mongodb_w4_app_password this playbook uses), it moves out of the self-healing fallback and must be rebuilt in the same operation as Step 3d/3e, or it will go stale the moment w4_app’s password next rotates. Add that as an explicit Step 3f when that SOPS key exists — flagged here so it isn’t missed, not implemented yet since it requires a live secret write.
Post-rotation check (either path): mailgun_pipeline_exporter_api_errors_total for the mongodb source stays at 0 after the sync completes; a spike means the exporter recreated on a URI the rotation hadn’t finished propagating.

et-operational-platform (`et_oper`) is a fourth, independently-credentialed consumer (added 2026-08-04, #1363)

et-operational-platform’s /zarzad/pinbox feature (Pinbox24 document browser, src/services/pinboxRecordsService.ts

src/services/pinbox/pinboxMongoClient.ts) reads w4_db.regRecords directly from Vercel, via a dedicated MongoDB user distinct from w4_app:

Item	Value
MongoDB user	`et_oper`
authSource	`admin` (not `w4_db` like `w4_app`)
Roles	`readWrite` on `w4_db` — corrected 2026-08-09 (#5925/#5954); this row previously said “read-only”, which was wrong. `docs/policies/credential-isolation-policy.md` (#2728) and this repo’s own `CLAUDE.md` §Secrets both document `et_oper` as `readWrite`. The verification steps below still use a read-only smoke test (`estimatedDocumentCount()`) — that is a deliberately minimal liveness check, not a statement of the user’s actual permission scope.
Hosts	same rs0 cluster: `145.239.133.104:27017,51.68.155.224:27017`
Credential storage	full URI in SOPS key `PINBOX24_MONGODB_URI`, file `secrets/et-operational-platform.env.sops` (not `pinbox24-w4.env.sops`)
Sync path	Vercel env vars (all targets), auto-synced via `secrets-sync.yml` on push to `main`/`dev`; `.env.local` regenerated via `scripts/decrypt-et-op-env.ps1` in that repo
Rotation coverage	Documented (added #5396) — see the “Rotating `et_oper`” sub-playbook immediately below. Note it is still separate from Step 2/2e/Step 6 (which target `w4_app`/`w3_read_v42` on bms-1), is not in the “Two-connection architecture” table, and its distribution is Vercel-only (no bms-1/bms-4 container).

How this was discovered: /zarzad/pinbox started returning 500 {"error":"Authentication failed."} (a raw MongoDB SCRAM auth error) on both local dev and the staging Vercel deployment. Byte-for-byte (SHA256) comparison confirmed .env.local, SOPS, and the live Vercel env var for PINBOX24_MONGODB_URI all agreed with each other — ruling out an env-sync problem — which means the et_oper credential itself had gone stale against the live rs0 cluster (rotated/removed on the MongoDB side without SOPS being updated, or never fully provisioned to match what’s in SOPS).

Rotating `et_oper` — sub-playbook (per-key, added #5396)

⚠️ STEP 0 — MANDATORY PRE-CHECK: is a rotation actually warranted? Do this before touching MongoDB. et-op#1363 was mis-diagnosed as “SOPS credential invalid against live MongoDB” and requested rotation of a live-working credential — no rotation was warranted. Always test the current SOPS value against the live rs0 cluster first; only rotate if it genuinely fails to authenticate.
# Extract the current URI silently (never echo it) and smoke-test it with a read-only

query against BOTH rs0 members. This is a minimal liveness check, not a statement that

et_oper only has read permissions — it has readWrite (see the table above).

URI= $(sops --decrypt --input-type dotenv --output-type dotenv \ secrets/et-operational-platform.env.sops | grep '^PINBOX24_MONGODB_URI=' | cut -d= -f2-) mongosh "$ URI” —quiet —eval ‘db.getSiblingDB(“w4_db”).regRecords.estimatedDocumentCount() >= 0’
&& echo “AUTH_OK — credential is LIVE, do NOT rotate (this is the et-op#1363 trap)”
|| echo “AUTH_FAILED — rotation genuinely needed, proceed” unset URI
If `AUTH_OK`: the credential works. A `500 {"error":"Authentication failed."}` from `/zarzad/pinbox`
with a passing pre-check points at an **env-sync** issue (Vercel/`.env.local` drift), not a stale
DB credential — chase that instead. Verified in et-op#1363: the SOPS value authenticated + read
`w4_db.regRecords` against both members and was byte-identical (SHA256) in Vercel PROD and STAGING.

Only if Step 0 returns AUTH_FAILED, rotate (same shape as Step 0 “find PRIMARY” + Step 2 “change password”, but targeting et_oper on authSource: admin):

Find the rs0 PRIMARY — Step 0 above (bms-2 145.239.133.104 / bms-3 51.68.155.224).
Source the rs0 admin credential from secrets/administration.env.sops — keys MONGODB_RS0_ADMIN_USER + MONGODB_RS0_ADMIN_PASSWORD. This is the canonical source, not bms-servers.env.sops: Phase A (2026-07-09, secrets-rotation-log.md) confirmed bms-servers.env.sops holds mongodb_rs0_admin_password but no admin-user key, so it is insufficient on its own. Note the bms-4 secret-manager worker cannot decrypt administration.env.sops — a worker-run rotation must obtain the admin creds from an administration.env.sops recipient or escalate to human-action.

Acquire the rotation lock FIRST, then rotate on the PRIMARY with a URI-safe hex password (no percent-encoding needed downstream). updateUser is a non-idempotent live write — #5925 was this exact et_oper/PINBOX24_MONGODB_URI rotation racing itself across two sessions — so per ADR 004 hold the lock before the write:

# BEFORE the updateUser: acquire the per-secret lock (keyed on the requester's repo + secret).
# Repo-relative path — run from the repo root (Linux worker: python3; Windows dev: python).
LOCK_ID=$(python3 scripts/rotation-log-entry.py acquire \
  --secret PINBOX24_MONGODB_URI --repo et-operational-platform --reason "#<issue>") || {
    echo "acquire denied/unverified — STOP, reconcile, do NOT rotate et_oper"; exit 1; }
# Immediately before the live write, re-check the lock is still held:
python3 scripts/rotation-log-entry.py check "$LOCK_ID" \
  || { echo "lock lost — abort and reconcile"; exit 1; }

// via mongosh on the rs0 PRIMARY, authenticated as the rs0 admin
db.getSiblingDB("admin").updateUser("et_oper", { pwd: "<new URI-safe hex>" })

After Step 6 distribution, release the lock: rotation-log-entry.py close "$LOCK_ID" (or fail).

Rebuild the URI (preserve host list + ?authSource=admin query string; only the password changes) and update PINBOX24_MONGODB_URI in secrets/et-operational-platform.env.sops (Step 6 pattern below, but a different file).
Verify the new URI authenticates against both rs0 members using the same read-only smoke test as Step 0 (mongosh … estimatedDocumentCount()) before distributing — a passing read is sufficient to prove the new password works; it doesn’t need a write test.
Distribute — Vercel only. This SOPS file has no bms-4 (and no bms-1) deploy step — its secrets-sync.yml job ships et-operational-platform.env.sops to Vercel only (confirmed in the 2026-07-09 Phase A row). Push the SOPS change to main; secrets-sync.yml updates both et-op Vercel projects — PROD prj_ziLl911… and STAGING prj_lli8aw…. There is no container to --force-recreate for this consumer; Vercel functions pick up the new env var on next invocation (or immediately if Vercel env is patched directly, ahead of the next push).
Log the rotation in docs/secrets-rotation-log.md (Step 7 below).

Proven: this exact procedure was executed end-to-end during #2889 — Phase A (SOPS + Vercel, PR #3567, 2026-07-09) and Phase B (db.updateUser('et_oper', …) on bms-2 PRIMARY with the administration.env.sops admin creds, verified via db.auth()-equivalent connection check).

Open design question (unchanged from #1363, not resolved here): whether et_oper should exist as a fully separate credential outside the w4_app / w3_read_v42 two-user model this playbook otherwise documents, or be folded into one of those. That is a least-privilege scope decision, not a password reset — flagging, not resolving.

Two-connection architecture (CRITICAL)

v42-prod maintains two independent MongoDB connections via dist/config/mongoDB.js:

Env var	Function	Library	Parser	User	Database
`NEW_MONGODB_URI`	`mongoConnection`	Mongoose (`useNewUrlParser: true`)	Strict	`w4_app`	`w4_db`
`v3MongoUrl`	`mongoConnectionV3`	mongojs	Liberal	`w3_read_v42` (read-only; formerly `pinbox_production`)	`w3_db`

Runtime wrapping: secrets.ts re-exports NEW_MONGODB_URI as MONGODB_URI when NODE_ENV=production. The compiled output in /app/dist/ uses this wrapped value.

Failure modes:

NEW_MONGODB_URI with un-encoded special chars → mongoose rejects URL at startup → all primary endpoints time out after 10 s (Operation X buffering timed out) while mongojs still responds.
v3MongoUrl with un-encoded special chars → mongojs rejects auth → legacy v3 endpoints fail (widgets, i18n backed by w3_db), primary endpoints may still respond.

Trigger — when to rotate

Scheduled rotation (~every 90 days)
Suspected credential exposure (team member offboarding, accidental log leak, git history)
Security audit finding
The MongoDB rs0 admin password was rotated and user URIs must be rebuilt
Rotating w3_app (w3 playbook) does NOT automatically rotate w4_app or pinbox_production

Prerequisites

Before starting, ensure you have:

Age key accessible: C:\Users\konar\.age\p24-infra-keys.txt
SSH access to bms-1 (root@94.23.26.113) and bms-2/bms-3 via key C:\Users\konar\.ssh\id_ed25519
mongosh installed on bms-2 or bms-3 (already present)
Decide which user to rotate: w4_app, pinbox_production, or both (independent operations)

Step 0 — Find the current rs0 PRIMARY

Always run password changes on the PRIMARY — they replicate automatically to secondaries.

ssh root@145.239.133.104 "mongosh --quiet --eval 'rs.isMaster().ismaster'"
# true  => bms-2 (145.239.133.104) is PRIMARY
# false => bms-3 (51.68.155.224) is PRIMARY

Note the PRIMARY IP as $PRIMARY_HOST for subsequent steps.

Step 1 — Generate a safe new password

Generate a password that contains no URI-unsafe characters (@, #, /, ?, +, =, !, %). Alphanumeric plus - and _ are always safe and require no URL-encoding.

# Generate for w4_app (repeat for pinbox_production if rotating that user too)
$rawBytes = [System.Security.Cryptography.RandomNumberGenerator]::GetBytes(24)
$env:W4_APP_NEW_PASSWORD = [System.Convert]::ToBase64String($rawBytes) -replace '[^A-Za-z0-9_-]', ''
Write-Host "w4_app password length: $($env:W4_APP_NEW_PASSWORD.Length)"
# Minimum 20 chars — regenerate if shorter due to character stripping
 
# If also rotating pinbox_production:
$rawBytes2 = [System.Security.Cryptography.RandomNumberGenerator]::GetBytes(24)
$env:PINBOX_PROD_NEW_PASSWORD = [System.Convert]::ToBase64String($rawBytes2) -replace '[^A-Za-z0-9_-]', ''
Write-Host "pinbox_production password length: $($env:PINBOX_PROD_NEW_PASSWORD.Length)"

If the password contains special characters (e.g., assigned by a password manager), URL-encode it before embedding in any URI (see Step 3). Mongoose will reject un-encoded chars. mongojs is lenient but encoding is still best practice for consistency.

Step 1b — Acquire the rotation lock FIRST — MANDATORY (ADR 004)

Step 2b/2c’s db.changeUserPassword('w4_app' / 'pinbox_production', …) on rs0 is a non-idempotent live server-side write, so per ADR 004 this rotation MUST hold the per-secret advisory lock before that write. Two sessions rotating the same user at once each set a different password and silently diverge (live rs0 vs SOPS/bms-1 env) — the #5925 race. SOPS-only edits don’t need this; a live password change does. Acquire one lock per user you rotate:

Run it from the repo root so the repo-relative script path resolves on the Windows dev machine (the absolute /opt/p24-infra/... path does not exist there):

$env:LOCK_ID = python scripts/rotation-log-entry.py acquire --secret W4_APP_MONGODB_PASSWORD --repo p24-infra --reason "#<issue>"
if ($LASTEXITCODE -ne 0) { throw "acquire denied/unverified — STOP, reconcile, do NOT rotate w4_app" }
# (If also rotating pinbox_production, acquire a second lock for its secret name too.)

Exit 3 = another session holds it → stop and reconcile (do not blind-overwrite). Exit 4 = DB unreachable → fail-closed, do not proceed. $LOCK_ID is the pending row — do not also open one. On a Linux worker run LOCK_ID=$(python3 scripts/rotation-log-entry.py acquire … ) with || exit 1 instead.

Then immediately before Step 2b/2c’s changeUserPassword re-check the lock is still held:

python scripts/rotation-log-entry.py check $env:LOCK_ID
if ($LASTEXITCODE -ne 0) { throw "lock lost — abort the live write and reconcile" }

Release after distribution: rotation-log-entry.py close "$LOCK_ID" --sops-commit "$(git rev-parse HEAD)" (or fail on failure). CLI reference: secret-manager-rotation-log.md.

Step 2 — Rotate password(s) in MongoDB rs0

2a. Read admin credentials silently from SOPS

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
 
$decrypted = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\bms-servers.env.sops
 
$env:MG_ADMIN_PASS = ($decrypted | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").ToString().Split("=",2)[1]
$env:MG_ADMIN_USER = ($decrypted | Select-String "^MONGODB_RS0_ADMIN_USER=").ToString().Split("=",2)[1]
$decrypted = $null

2b. Change w4_app password in MongoDB (w4_db)

$PRIMARY_HOST = "145.239.133.104"  # or 51.68.155.224 — from Step 0
 
$mongoScript = "use admin`ndb.auth('$($env:MG_ADMIN_USER)', '$($env:MG_ADMIN_PASS)')`nuse w4_db`ndb.changeUserPassword('w4_app', '$($env:W4_APP_NEW_PASSWORD)')`nprint('w4_app_password_changed_ok')"
$tmpScript = "C:\Users\konar\AppData\Local\Temp\w4_rotate.js"
[System.IO.File]::WriteAllText($tmpScript, $mongoScript.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
 
scp $tmpScript "root@${PRIMARY_HOST}:/tmp/w4_rotate.js"
$result = ssh root@$PRIMARY_HOST "mongosh --quiet /tmp/w4_rotate.js; rm /tmp/w4_rotate.js"
 
if ($result -notmatch "w4_app_password_changed_ok") {
    throw "Password change for w4_app FAILED — review output: $result"
}
Write-Host "w4_app password changed on MongoDB rs0 PRIMARY"
Remove-Item $tmpScript -Force -ErrorAction SilentlyContinue

2c. Change pinbox_production password in MongoDB (w3_db) — if rotating that user

# NOTE: pinbox_production lives in w3_db, not w4_db
$mongoScript2 = "use admin`ndb.auth('$($env:MG_ADMIN_USER)', '$($env:MG_ADMIN_PASS)')`nuse w3_db`ndb.changeUserPassword('pinbox_production', '$($env:PINBOX_PROD_NEW_PASSWORD)')`nprint('pinbox_prod_password_changed_ok')"
$tmpScript2 = "C:\Users\konar\AppData\Local\Temp\pinbox_prod_rotate.js"
[System.IO.File]::WriteAllText($tmpScript2, $mongoScript2.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
 
scp $tmpScript2 "root@${PRIMARY_HOST}:/tmp/pinbox_prod_rotate.js"
$result2 = ssh root@$PRIMARY_HOST "mongosh --quiet /tmp/pinbox_prod_rotate.js; rm /tmp/pinbox_prod_rotate.js"
 
if ($result2 -notmatch "pinbox_prod_password_changed_ok") {
    throw "Password change for pinbox_production FAILED — review output: $result2"
}
Write-Host "pinbox_production password changed on MongoDB rs0 PRIMARY (w3_db)"
Remove-Item $tmpScript2 -Force -ErrorAction SilentlyContinue

2d. Clear admin credentials

$env:MG_ADMIN_PASS = ""; $env:MG_ADMIN_USER = ""

2e. Verify new credentials authenticate

# Verify w4_app on w4_db
$verifyScript = "use w4_db`ndb.auth('w4_app', '$($env:W4_APP_NEW_PASSWORD)')`nprint('w4_app_auth_ok')"
$tmpVerify = "C:\Users\konar\AppData\Local\Temp\w4_verify.js"
[System.IO.File]::WriteAllText($tmpVerify, $verifyScript.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
 
scp $tmpVerify "root@${PRIMARY_HOST}:/tmp/w4_verify.js"
$verifyResult = ssh root@$PRIMARY_HOST "mongosh --quiet /tmp/w4_verify.js; rm /tmp/w4_verify.js"
 
if ($verifyResult -notmatch "w4_app_auth_ok") {
    throw "Verification FAILED — w4_app auth rejected with new password."
}
Write-Host "w4_app auth verified on PRIMARY"
Remove-Item $tmpVerify -Force -ErrorAction SilentlyContinue
 
# Verify pinbox_production on w3_db (if rotated)
$verifyScript2 = "use w3_db`ndb.auth('pinbox_production', '$($env:PINBOX_PROD_NEW_PASSWORD)')`nprint('pinbox_prod_auth_ok')"
$tmpVerify2 = "C:\Users\konar\AppData\Local\Temp\pinbox_prod_verify.js"
[System.IO.File]::WriteAllText($tmpVerify2, $verifyScript2.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
 
scp $tmpVerify2 "root@${PRIMARY_HOST}:/tmp/pinbox_prod_verify.js"
$verifyResult2 = ssh root@$PRIMARY_HOST "mongosh --quiet /tmp/pinbox_prod_verify.js; rm /tmp/pinbox_prod_verify.js"
 
if ($verifyResult2 -notmatch "pinbox_prod_auth_ok") {
    throw "Verification FAILED — pinbox_production auth rejected with new password."
}
Write-Host "pinbox_production auth verified on PRIMARY"
Remove-Item $tmpVerify2 -Force -ErrorAction SilentlyContinue

Step 3 — Update env file on bms-1

3a. URL-encode the new password(s)

Always encode — even if the password looks safe. Mongoose uses a strict URL parser.

$encodedW4App = [System.Uri]::EscapeDataString($env:W4_APP_NEW_PASSWORD)
$encodedPinboxProd = [System.Uri]::EscapeDataString($env:PINBOX_PROD_NEW_PASSWORD)
# If only rotating one user, only compute the encoded value for that user

3b. Check current env file (key names only — do not print values)

ssh root@94.23.26.113 "grep -E '^(NEW_MONGODB_URI|v3MongoUrl)=' /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env | cut -d= -f1"
# Expected output (both keys must exist):
#   NEW_MONGODB_URI
#   v3MongoUrl

Also extract the query string portion to preserve any parameters:

# Safe — extracts only the portion after database name, not the password
ssh root@94.23.26.113 "grep '^NEW_MONGODB_URI=' /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env | sed 's|.*w4_db||'"
# e.g., outputs: ?replicaSet=rs0&authSource=w4_db
 
ssh root@94.23.26.113 "grep '^v3MongoUrl=' /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env | sed 's|.*w3_db||'"
# e.g., outputs: ?replicaSet=rs0&authSource=w3_db

3c. Back up the env file before editing

ssh root@94.23.26.113 'cp /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env.bak.$(date +%Y%m%d_%H%M%S)'

3d. Update env keys via Python on bms-1

Use Python for the replacement — the new password travels over the SSH session (encrypted) rather than appearing in shell command arguments. Adjust which keys to update based on which user(s) you rotated.

# Build the new URI values (adjust query strings from Step 3b if needed)
$qsW4  = "?replicaSet=rs0&authSource=w4_db"
$qsV3  = "?replicaSet=rs0&authSource=w3_db"
$newMongoUri    = "mongodb://w4_app:${encodedW4App}@145.239.133.104,51.68.155.224/w4_db${qsW4}"
$newV3MongoUrl  = "mongodb://pinbox_production:${encodedPinboxProd}@145.239.133.104,51.68.155.224/w3_db${qsV3}"
# Pass 'SKIP' for whichever user you are NOT rotating this time
 
$pyScript = "import sys`nenv_path = '/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env'`nnew_mongo_uri = sys.argv[1]`nnew_v3_mongo_url = sys.argv[2]`nwith open(env_path, 'r') as f:`n    lines = f.readlines()`nupdated = []`nfor line in lines:`n    if line.startswith('NEW_MONGODB_URI=') and new_mongo_uri != 'SKIP':`n        updated.append('NEW_MONGODB_URI=' + new_mongo_uri + '\n')`n    elif line.startswith('v3MongoUrl=') and new_v3_mongo_url != 'SKIP':`n        updated.append('v3MongoUrl=' + new_v3_mongo_url + '\n')`n    else:`n        updated.append(line)`nwith open(env_path, 'w') as f:`n    f.writelines(updated)`nprint('env_updated_ok')"
 
$tmpPy = "C:\Users\konar\AppData\Local\Temp\update_w4_env.py"
[System.IO.File]::WriteAllText($tmpPy, $pyScript.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
scp $tmpPy root@94.23.26.113:/tmp/update_w4_env.py
 
$result = ssh root@94.23.26.113 "python3 /tmp/update_w4_env.py '$newMongoUri' '$newV3MongoUrl'; rm /tmp/update_w4_env.py"
if ($result -notmatch "env_updated_ok") {
    throw "Env file update FAILED"
}
Write-Host "backend-environment.env updated on bms-1"
 
# Clear sensitive vars
$encodedW4App = ""; $encodedPinboxProd = ""
$newMongoUri = ""; $newV3MongoUrl = ""
Remove-Item $tmpPy -Force -ErrorAction SilentlyContinue

Selective rotation (only one user): Pass 'SKIP' as the argument for the user you are NOT rotating. The Python script skips lines where the value is exactly SKIP.

Manual alternative (if scripting fails): SSH to bms-1, then:
NEW_ENCODED=$(python3 -c "import urllib.parse; print(urllib.parse.quote('RAW_PASSWORD', safe=''))")
sed -i "s|^NEW_MONGODB_URI=.*|NEW_MONGODB_URI=mongodb://w4_app:${NEW_ENCODED}@145.239.133.104,51.68.155.224/w4_db?replicaSet=rs0\&authSource=w4_db|" \
  /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env

3e. If rotating `w4_app`: update mailgun-v42-prod’s env file too (MANDATORY, added 2026-07-10)

mailgun-v42-prod uses the same w4_app credential as v42-prod (see “mailgun-v42-prod is a second w4_app consumer” above) — its MONGODB_URL must be updated to the exact same value as the NEW_MONGODB_URI you just built in Step 3d. Skip this sub-step only if you are rotating pinbox_production alone (that user is not used by mailgun).

# Reuses $newMongoUri built in Step 3d — same w4_app credential, same URI
$pyScriptMailgun = "import sys`nenv_path = '/root/mailgun-prod/mailgun-environment.env'`nnew_mongo_url = sys.argv[1]`nwith open(env_path, 'r') as f:`n    lines = f.readlines()`nupdated = [l for l in lines if not l.startswith('MONGODB_URL=')]`nupdated.append('MONGODB_URL=' + new_mongo_url + '\n')`nwith open(env_path, 'w') as f:`n    f.writelines(updated)`nprint('mailgun_env_updated_ok')"
 
$tmpPyMailgun = "C:\Users\konar\AppData\Local\Temp\update_mailgun_env.py"
[System.IO.File]::WriteAllText($tmpPyMailgun, $pyScriptMailgun.Replace("`r`n","`n"), [System.Text.UTF8Encoding]::new($false))
scp $tmpPyMailgun root@94.23.26.113:/tmp/update_mailgun_env.py
 
# NOTE: run this BEFORE clearing $newMongoUri in Step 3d's cleanup, or rebuild it here from
# $env:W4_APP_NEW_PASSWORD if that step's cleanup already ran.
$resultMailgun = ssh root@94.23.26.113 "cp /root/mailgun-prod/mailgun-environment.env /root/mailgun-prod/mailgun-environment.env.bak.$(Get-Date -Format yyyyMMdd_HHmmss); python3 /tmp/update_mailgun_env.py '$newMongoUri'; rm /tmp/update_mailgun_env.py"
if ($resultMailgun -notmatch "mailgun_env_updated_ok") {
    throw "mailgun-environment.env update FAILED"
}
Write-Host "mailgun-environment.env updated on bms-1"
Remove-Item $tmpPyMailgun -Force -ErrorAction SilentlyContinue

Step 4 — Recreate v42-prod AND mailgun-v42-prod together (MANDATORY, both containers)

This step is automated when running scripts/rotate/w4_app_mongodb_password.sh. The script calls docker-compose up -d --no-build --force-recreate backend immediately after updating backend-environment.env. Only follow this step for manual rotations.

As of 2026-07-10, recreate mailgun-v42-prod in the SAME rotation operation — it shares the w4_app credential (Step 3e above) and must never be left running on a stale copy. Do not treat it as optional or defer it to “later” — that gap is exactly what caused the incident in docs/playbooks/mailgun-mongodb-stale-credential-hang.md.

--force-recreate is required for both containers — without it Docker reuses the running container’s environment (old password baked in at startup) even though the env file on disk has been updated. This was the root cause of the W4 outage on 2026-07-05 (v42-prod) and the mailgun pipeline outage on 2026-07-10 (mailgun-v42-prod, via the CI restart step that was missing the flag — now fixed in secrets-sync.yml).

# v42-prod
ssh root@94.23.26.113 "cd /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts && CONTAINER_NAME=v42-prod IMAGE_NAME=563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod docker-compose up -d --no-build --force-recreate backend"
 
# mailgun-v42-prod — same operation, same rotation
ssh root@94.23.26.113 "cd /root/mailgun-prod && CONTAINER_NAME=mailgun-v42-prod IMAGE_NAME=mailgun-v42-prod:latest CLUSTER_NETWORK=test-net docker-compose up -d --no-deps --force-recreate mailgun"

Wait ~15 seconds for PM2 to start all workers on both containers before verifying.

Do NOT use docker exec <container> pm2 restart --update-env as the permanent fix on either container. Env vars injected this way are lost on the next docker restart. Always edit the env file and force-recreate the container as shown above.

Step 5 — Verify the application is healthy

5a. HTTP endpoint check (primary signal)

Start-Sleep -Seconds 15
 
$r = Invoke-WebRequest "https://api.w4.pinbox24.com/api/i18n/langs" -TimeoutSec 15 -UseBasicParsing
Write-Host "Status: $($r.StatusCode)"
# Expected: 200 with response time < 200 ms
 
$body = $r.Content | ConvertFrom-Json
Write-Host "Language entries: $($body.Count)"
# Expected: > 0 (actual language data, not empty array)

Status code	Meaning
200	Both `NEW_MONGODB_URI` (mongoose) and `v3MongoUrl` (mongojs) are working
422	mongojs (`v3MongoUrl` / `pinbox_production`) is failing — check encoding or auth
504	Mongoose (`NEW_MONGODB_URI` / `w4_app`) is timing out — check `NEW_MONGODB_URI` encoding or auth
502	Container is not reachable — check if `v42-prod` is running

5b. PM2 logs inside the container

# Discover log path first (v42-prod may differ from v32-prod pattern)
ssh root@94.23.26.113 "docker exec v42-prod find /var/log -name '*out.log' 2>/dev/null | head -5"
ssh root@94.23.26.113 "docker exec v42-prod tail -40 /var/log/v42-prod/pm2/pm2_v42-prod_production_out.log"
# Good signs: "Mongoose connected" + "Mongoose connection opened"
# Bad signs:  "Password contains an illegal unescaped character" — URL-encoding missing
#             "Mongoose disconnected" — mongoose auth still failing
#             "Operation X buffering timed out after 10000ms" — mongoose cannot connect

# Docker-level startup / crash errors
ssh root@94.23.26.113 "docker logs v42-prod --tail 30 2>&1"

Do NOT run docker exec v42-prod pm2 env 0 without filtering — it prints all env vars including NEW_MONGODB_URI and v3MongoUrl in plaintext. Safe alternative:
ssh root@94.23.26.113 "docker exec v42-prod pm2 env 0 | grep -iE 'mongo|v3mongo' | sed 's/value: .*/value: REDACTED/'"
NEVER add grep context flags (-A / -B / -C) to a pm2 env grep. In pm2 env output every line is a distinct KEY: value secret, so a context window pulls in neighbouring variables’ full values, not related lines. Use only an exact anchored match on the key name(s) with no context — grep -iE 'mongo|v3mongo' above already does this; do not “widen” it with -A1/-B1 to inspect the output format. This is exactly how RABBIT_MQ_URL + CONVERT_API leaked during #3754 verification (#4966): pm2 env 0 | grep -i -A1 -B1 'NODE_ENV' matched a harmless key but the -A1 -B1 window captured the adjacent variables’ full values. If you need to see the output format, redact first: append | sed 's/value: .*/value: REDACTED/'.

5c. Legacy v3 endpoint check (verifies pinbox_production / v3MongoUrl)

# Test an endpoint backed by w3_db (adjust path to actual v3 route)
$r2 = Invoke-WebRequest "https://api.w4.pinbox24.com/api/v3/widgets" -TimeoutSec 15 -UseBasicParsing -ErrorAction SilentlyContinue
Write-Host "v3 widget status: $($r2.StatusCode)"
# 200 = pinbox_production / mongojs connection is working
# 422/504 = v3MongoUrl problem

5d. Verify mailgun-v42-prod picked up the same credential (MANDATORY if w4_app was rotated)

Hash comparison — confirms the running container actually loaded the new value (never prints the credential itself):

ssh root@94.23.26.113 "grep '^MONGODB_URL=' /root/mailgun-prod/mailgun-environment.env | sha256sum; docker exec mailgun-v42-prod sh -c `"cat /proc/1/environ | tr '\0' '\n' | grep '^MONGODB_URL=' | sha256sum`""
# Both hashes MUST match. If they differ, the container is still running the OLD credential —
# docker-compose up -d without --force-recreate silently no-ops. Re-run Step 4's mailgun-v42-prod
# command and check again.

Then confirm live traffic is actually processing (not just that the container started):

ssh root@94.23.26.113 "docker exec mailgun-v42-prod pm2 logs mailgun-backend --lines 10 --nostream"
# Good sign: recent "POST /api/v1/integration/add 200 <ms> - 2" lines with real response times
# Bad sign:  "POST /api/v1/integration/add - - ms - -" (no status/duration = request hanging,
#            same silent-hang symptom as the 2026-07-10 incident) or a fresh MongoNetworkError/
#            MongoParseError in the err log after this container's start time

Step 6 — Update SOPS with the new password(s)

After confirming the application is healthy, record the new password(s) in the canonical secret store.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$secretsDir = "C:\code_2026\p24-infra\secrets"
 
$plain = sops --decrypt --input-type dotenv --output-type dotenv "$secretsDir\bms-servers.env.sops"
$tempPath = "$secretsDir\bms-servers-edit.env.sops"
 
$updated = $plain | ForEach-Object {
    if ($_ -match '^W4_APP_MONGODB_PASSWORD=') {
        "W4_APP_MONGODB_PASSWORD=$($env:W4_APP_NEW_PASSWORD)"
    } elseif ($_ -match '^PINBOX_PRODUCTION_MONGODB_PASSWORD=') {
        "PINBOX_PRODUCTION_MONGODB_PASSWORD=$($env:PINBOX_PROD_NEW_PASSWORD)"
    } else { $_ }
}
# Append keys if they do not exist yet
if (-not ($plain | Where-Object { $_ -match '^W4_APP_MONGODB_PASSWORD=' })) {
    $updated += "W4_APP_MONGODB_PASSWORD=$($env:W4_APP_NEW_PASSWORD)"
}
if ($env:PINBOX_PROD_NEW_PASSWORD -and -not ($plain | Where-Object { $_ -match '^PINBOX_PRODUCTION_MONGODB_PASSWORD=' })) {
    $updated += "PINBOX_PRODUCTION_MONGODB_PASSWORD=$($env:PINBOX_PROD_NEW_PASSWORD)"
}
 
[System.IO.File]::WriteAllText($tempPath, ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
$enc = sops --encrypt --input-type dotenv --output-type dotenv $tempPath
[System.IO.File]::WriteAllText("$secretsDir\bms-servers.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
# Canary — MANDATORY before git add
sops --decrypt --input-type dotenv --output-type dotenv "$secretsDir\bms-servers.env.sops" | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS canary FAILED — do NOT commit" }
Write-Host "SOPS canary OK"
 
Remove-Item $tempPath -Force -ErrorAction SilentlyContinue
$env:W4_APP_NEW_PASSWORD = ""
$env:PINBOX_PROD_NEW_PASSWORD = ""

Commit via PR (do not push directly to main):

cd C:\code_2026\p24-infra
git add secrets\bms-servers.env.sops
git commit -m "chore: rotate w4_app MongoDB password (w4.pinbox24.com)"
# Then: gh pr create --base main --title "chore: rotate w4_app MongoDB password"

Step 7 — Log the rotation

Append to docs/secrets-rotation-log.md:

| YYYY-MM-DD HH:MM UTC | w4.pinbox24.com | W4_APP_MONGODB_PASSWORD | <reason> | <your-name> | yes |
| YYYY-MM-DD HH:MM UTC | w4.pinbox24.com (legacy) | PINBOX_PRODUCTION_MONGODB_PASSWORD | <reason> | <your-name> | yes |

Rollback — if Step 5 verification fails

Option A — Restore from env file backup (fastest)

ssh root@94.23.26.113 "ls /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env.bak.*"
# Pick the latest timestamp, e.g. backend-environment.env.bak.20260702_153100
 
ssh root@94.23.26.113 "cp /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env.bak.20260702_153100 /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env"

Then restart the container (Step 4) and verify (Step 5).

Note: restoring the env file does NOT revert the MongoDB password change. The application will work again because the old password is still accepted immediately after changeUserPassword — MongoDB does not invalidate active sessions instantly. Schedule a follow-up rotation with a safe password once the application is stable.

Option B — Full rollback (revert MongoDB password + env file)

Identify the old password from SOPS or the .bak env file.
Connect to MongoDB PRIMARY (Step 0).
Change w4_app / pinbox_production password back to the old value (Steps 2b/2c with old value).
Restore env file from backup (Option A) or rebuild URIs with old encoded password (Step 3d).
Restart container (Step 4).
Verify (Step 5).

Escalation

Symptom	Action
mongosh auth fails for `w4_app` with new password	Check replication lag — connect to SECONDARY and confirm change replicated. If lag > 60 s, wait and retry.
mongosh auth fails for `pinbox_production` with new password	Same — replication lag; `pinbox_production` lives in `w3_db`, verify on the correct database.
Mongoose still disconnecting after env update + restart	Confirm `NEW_MONGODB_URI` was updated: `ssh root@94.23.26.113 "grep -c 'NEW_MONGODB_URI' /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env"` — expect `1`.
422 (not 504) on `/api/i18n/langs`	mongojs (`v3MongoUrl` / `pinbox_production`) is failing — check encoding or auth for `v3MongoUrl`.
504 but legacy endpoints (widgets, i18n from w3_db) respond	Mongoose (`NEW_MONGODB_URI` / `w4_app`) is timing out — check encoding of `NEW_MONGODB_URI`.
”Password contains an illegal unescaped character” in logs	URL-encoding missing — re-run Step 3 with `[System.Uri]::EscapeDataString()` output.
”Operation X buffering timed out after 10000ms” in logs	Mongoose cannot reach MongoDB — check `NEW_MONGODB_URI` encoding AND that the password was changed in the correct database (`w4_db`).
SOPS canary fails after write	Do NOT commit. See `docs/playbooks/sops-windows-crlf.md` for recovery.
rs0 PRIMARY unknown (both hosts return false)	Election in progress — wait 30 s and retry. If persists > 2 min, check network between bms-2, bms-3, bms-4.
`w4_app` user missing from `w4_db` after rotation attempt	Recreate: `use w4_db; db.createUser({user:'w4_app',pwd:'NEWPWD',roles:[{role:'readWrite',db:'w4_db'}]})` on PRIMARY with admin auth.
`pinbox_production` user missing from `w3_db`	Recreate on `w3_db` — coordinate with w3 rotation playbook to avoid conflicts with `w3_app`.

Prevention

Generate passwords without URI-unsafe characters. Use the method in Step 1 or openssl rand -base64 18 | tr -d '+/=' on Linux. Eliminates the URL-encoding requirement entirely.
Decide scope before starting. w4 has two users (w4_app, pinbox_production) in two databases (w4_db, w3_db). Update only the user(s) you are rotating — partial updates cause partial failures.
When rotating pinbox_production, coordinate with the w3 team. pinbox_production on w3_db may be shared with other services; changing its password without updating all consumers will break them.
Back up backend-environment.env before every edit (Step 3c). A timestamped backup makes rollback a one-liner.
Verify with the HTTP endpoint (/api/i18n/langs), not just PM2 logs. PM2 may show “Mongoose connected” while mongojs is still failing, causing 422 on all data endpoints.
Never rely on pm2 restart --update-env as the permanent fix. Env vars injected via docker exec are lost on docker restart. Always edit the env file and recreate the container.
Add a calendar reminder for the next rotation (~90 days) immediately after completing this one.

docs/playbooks/w3-mongodb-credential-rotation.md — w3.pinbox24.com (v32-prod) rotation — same two-connection pattern, different env keys and users
docs/playbooks/mongodb-credential-rotation.md — rs0 admin and prometheus user rotation
docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md — full outage diagnosis (incl. Fix C: two-connection problem)
docs/playbooks/mongodb-exporter-uri-special-chars.md — URI special character encoding for exporters
docs/playbooks/sops-windows-crlf.md — SOPS write corruption on Windows (BOM/CRLF)

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="W4_APP_MONGODB_PASSWORD",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — w4_app MongoDB password rotated on rs0 and v42-prod env updated",
    env="bms-2",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'W4_APP_MONGODB_PASSWORD', 'success', 'Scheduled rotation — w4_app MongoDB password rotated on rs0 and v42-prod env updated', 'bms-2')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

w4-mongodb-credential-rotation