mongodb-credential-rotation

MongoDB rs0 Credential Rotation Playbook

Trigger

Rotate MongoDB credentials when any of the following occur:

• Quarterly scheduled rotation (due ~3 months after last rotation)
• Suspected credential exposure (team member offboarding, accidental log leak, git history exposure)
• MongoDB exporter alert fires with authentication failures
• Any server that had access to admin credentials is decommissioned

---

MongoDB rs0 Users

User	Role	Used by	SOPS key
admin (or configured admin user)	root on admin db	p24-status.py, manual maintenance	MONGODB_RS0_ADMIN_USER / MONGODB_RS0_ADMIN_PASSWORD — mirrored across 5 files: secrets/administration.env.sops, secrets/bms-servers.env.sops (lowercase mongodb_rs0_admin_password), secrets/mongodb-bms.env.sops, secrets/n8n-bms4.env.sops, secrets/role-secret-manager.env.sops
admin_backup	root on admin db — break-glass account, see below	Emergency recovery only, not used by any live service	GH Secrets ONLY (MONGODB_RS0_ADMIN_BACKUP_USER / MONGODB_RS0_ADMIN_BACKUP_PASSWORD) — deliberately NOT in any SOPS file that also holds the main admin credential
w3_app	readWrite on w3_db	v32-prod (api.w3.pinbox24.com) via MONGODB_URL + PMONGODB_URL	W3_APP_MONGODB_PASSWORD in secrets/bms-servers.env.sops
w4_app	readWrite on w4_db	v42-prod (api.w4.pinbox24.com) via NEW_MONGODB_URI	W4_APP_MONGODB_PASSWORD in secrets/bms-servers.env.sops
prometheus	clusterMonitor on admin db	mongodb-exporter-bms2, mongodb-exporter-bms3 containers on bms-4	MONGODB_RS0_PROMETHEUS_PASSWORD in secrets/n8n-bms4.env.sops

Note: The arbiter on bms-4 does not replicate users — the prometheus user only exists on bms-2 and bms-3 (the voting members). mongodb-exporter for bms-4 is intentionally omitted.

Note (2026-08-07, #5859) — the canonical prometheus consumers are the two bms-4 containers ONLY. Prometheus scrapes 54.36.123.110:9216 (bms-2 data) and :9217 (bms-3 data) — the containers listed above — never an exporter running on bms-2/bms-3 (only :9100 node_exporter is scraped there). A legacy mongodb_exporter.service systemd unit on bms-2 was found still running with a stale hardcoded prometheus password (inline in its ExecStart), failing SCRAM auth ~240×/hour and filling mongod.log — but it is scraped by nothing and managed by no ansible/secrets-sync path, so it is redundant, not a monitoring gap. Do NOT credential-manage or add it to any distribution list — decommission it per bms2-mongodb-exporter-decommission.md.

Note (2026-07-09, incident #3494): the table above previously pointed to secrets/monitoring.env.sops, which was stale — the admin credential had already migrated to the 5 files listed. Always verify the current SOPS location with a key-name-only grep (see Step 0b) before trusting this table blindly; SOPS reorganizations (e.g. #3178) can move keys without this doc being updated in the same PR.

---

Backup Admin Account (admin_backup) — Break-Glass Design

Why this exists: on 2026-07-09 (incident #3494), the admin password was rotated and the newly-generated value was lost before it could be persisted anywhere (the shell process holding it exited before the SOPS write ran) — a self-inflicted, total lockout of the ONLY admin-equivalent account. Recovering required temporarily taking bms-2 out of the replica set and running it as a standalone with auth disabled — a materially riskier operation than a routine rotation should ever require.

Design rules — read before touching the admin password:

1. Never rotate the only admin-equivalent credential without a working fallback in hand first. Before touching MONGODB_RS0_ADMIN_PASSWORD, confirm admin_backup authenticates (Step 1-pre below). If it doesn’t, fix admin_backup FIRST — completely separately from the main rotation — and confirm it works before proceeding.
2. admin_backup’s credential lives ONLY in GH Secrets (MONGODB_RS0_ADMIN_BACKUP_USER, MONGODB_RS0_ADMIN_BACKUP_PASSWORD), never co-located in any SOPS file that also holds the main admin password. The two accounts must not be simultaneously exposed by a single leak/compromise of one store (SOPS+age vs. GH Secrets are different security domains — a leaked age key doesn’t hand over admin_backup, and vice versa).
3. Whenever you generate a new password value for either account, persist it to its permanent store (GH Secrets for admin_backup, SOPS for admin) in the SAME shell invocation that generated it — never across separate tool calls / separate terminal sessions. Shell state (variables) does not persist between calls; a value that isn’t written down before the process exits is gone forever. This is exactly how incident #3494 happened, twice, before this rule was written.
4. If you ever end up with NO working admin-equivalent credential (both admin and admin_backup lost/locked out simultaneously — the double-failure case), the only recovery is the standalone-mode procedure in the Escalation table below. Treat it as a last resort, not a routine step.

---

ℹ UPDATE (PR 6c3c5b03) — v32-prod and v42-prod now use dedicated app users, not admin. Since the migration, v32-prod connects as w3_app and v42-prod connects as w4_app. Admin password rotation no longer requires updating Pinbox24 containers (Step 1e is deprecated). To rotate the Pinbox24 app MongoDB passwords, run scripts/rotate/w3_app_mongodb_password.sh and scripts/rotate/w4_app_mongodb_password.sh, or follow docs/playbooks/w3-mongodb-credential-rotation.md.

---

SOPS File Map

SOPS file	Keys
secrets/administration.env.sops	MONGODB_RS0_ADMIN_USER, MONGODB_RS0_ADMIN_PASSWORD
secrets/bms-servers.env.sops	mongodb_rs0_admin_password (lowercase), W3_APP_MONGODB_PASSWORD, mongodb_w3_app_password, mongodb_w4_app_password, mongodb_rs0_prometheus_password
secrets/mongodb-bms.env.sops	MONGODB_RS0_ADMIN_PASSWORD, mongodb_rs0_admin_user, MONGODB_RS0_PROMETHEUS_PASSWORD, MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED
secrets/n8n-bms4.env.sops	MONGODB_RS0_ADMIN_PASSWORD, mongodb_rs0_admin_user, MONGODB_RS0_PROMETHEUS_PASSWORD, MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED
secrets/role-secret-manager.env.sops	MONGODB_RS0_ADMIN_PASSWORD, MONGODB_RS0_ADMIN_USER

GH Secrets (radieu/p24-infra): MONGODB_RS0_ADMIN_PASSWORD (mirror of the current SOPS value, independent recovery path if SOPS/age is ever unavailable), MONGODB_RS0_ADMIN_BACKUP_USER, MONGODB_RS0_ADMIN_BACKUP_PASSWORD (break-glass account — only stored here, never in SOPS alongside the main admin credential).

All five SOPS files must be updated together on every admin rotation — a partial update reproduces the exact stale-credential drift that caused incident #3494’s prerequisite investigation to take longer than it should have (n8n-bms4.env.sops was found holding an old value while the other 4 held the current one).

secrets-sync.yml deploys these to their respective target servers on merge to main.

---

Step 0 — Find the Current PRIMARY

The PRIMARY can shift after elections. Always determine the current PRIMARY before connecting.

# Check bms-2 first
ssh ubuntu@145.239.133.104 "mongosh --quiet --eval 'rs.isMaster().ismaster'"
# Returns: true  → bms-2 is PRIMARY
# Returns: false → bms-3 is PRIMARY, try:
ssh ubuntu@51.68.155.224 "mongosh --quiet --eval 'rs.isMaster().ismaster'"

All password changes must be run on the PRIMARY — they replicate automatically to secondaries.

---

Step 0b — Verify (or Establish) admin_backup FIRST — MANDATORY before touching admin

Do not skip this. Do not proceed to Step 1 until this passes.

# Boolean-only check — never print the password
ssh ubuntu@<PRIMARY_HOST> "mongosh --quiet -u 'admin_backup' -p '<MONGODB_RS0_ADMIN_BACKUP_PASSWORD from GH Secrets>' --authenticationDatabase admin --eval 'db.runCommand({ping:1}).ok'"
# Expected: 1

• If this returns 1: you have a working fallback. Proceed to Step 1 — if the main rotation goes wrong, admin_backup can reset admin via a normal authenticated changeUserPassword call, no standalone-mode dance needed.
• If this fails (account missing or password wrong): fix admin_backup NOW, on its own, before touching the main admin password:
  1. Authenticate as the current (still-working) admin user.
  2. Generate a new backup password and, in the SAME shell invocation, immediately: (a) create-or-update admin_backup with root role on the PRIMARY, (b) verify it authenticates, (c) gh secret set MONGODB_RS0_ADMIN_BACKUP_USER / MONGODB_RS0_ADMIN_BACKUP_PASSWORD — all before the process exits.
  3. Only once admin_backup is confirmed working and persisted, proceed to Step 1.

This inverts the traditional order (previously: rotate main admin, backup was an afterthought). Backup first, always — see “Backup Admin Account” design rules above.

---

Step 0c — Acquire the rotation lock FIRST — MANDATORY (ADR 004)

Changing a MongoDB password (db.changeUserPassword, Steps 1b / 2a) is a non-idempotent live server-side write, so per ADR 004 this rotation MUST hold the per-secret advisory lock before that write. Two sessions rotating the same user at once each set a different password — whoever writes last wins on rs0 while whoever’s secrets-sync.yml ran last wins in SOPS, and they silently diverge (the #5925 race). SOPS-only edits don’t need this; a live password change does.

Acquire one lock per secret, next to the step that rotates it — do not pre-acquire both here (a second acquire for a secret you already hold self-denies with exit 3). This step acquires the admin lock for Step 1; if you are running Step 2, its MONGODB_RS0_PROMETHEUS_PASSWORD lock is acquired in Step 2 → 2-lock below (which stands alone if you rotate only the Prometheus password). Run it from the repo root so the repo-relative script path resolves on the Windows dev machine (the absolute /opt/p24-infra/... path does not exist there):

# Windows dev machine (this playbook's SOPS steps run here). $LOCK_ID captures the pending row id.
$env:LOCK_ID = python scripts/rotation-log-entry.py acquire --secret MONGODB_RS0_ADMIN_PASSWORD --repo p24-infra --reason "#<issue>"
if ($LASTEXITCODE -ne 0) { throw "acquire denied/unverified — STOP, reconcile, do NOT rotate" }

• Exit 3 = another session holds the lock → stop and reconcile, do not blind-overwrite. Exit 4 = Supabase unreachable → fail-closed, do not proceed. $LOCK_ID is the pending row (do not also open one). On a Linux worker run the same command as LOCK_ID=$(python3 scripts/rotation-log-entry.py acquire … ) || { echo denied; exit 1; }.

Then immediately before the live write (db.changeUserPassword in Step 1b / 2a) re-check the lock is still held:

python scripts/rotation-log-entry.py check $env:LOCK_ID
if ($LASTEXITCODE -ne 0) { throw "lock lost — abort the live write and reconcile (do NOT overwrite blind)" }

After distribution completes (Step 3 verification), release the lock: rotation-log-entry.py close "$LOCK_ID" --sops-commit "$(git rev-parse HEAD)" (or fail "$LOCK_ID" --error "…" on failure). CLI reference: secret-manager-rotation-log.md.

---

Step 1 — Rotate the Admin Password

1a. Connect to PRIMARY as current admin

# SSH to the PRIMARY node (replace HOST with 145.239.133.104 or 51.68.155.224)
ssh ubuntu@HOST
 
# Connect with current admin credentials (read from SOPS — never hardcode here)
mongosh --host localhost:27017 -u "<MONGODB_RS0_ADMIN_USER>" -p "<MONGODB_RS0_ADMIN_PASSWORD>" --authenticationDatabase admin

1b. Change the admin password

// In mongosh — replace NEW_PASSWORD_PLACEHOLDER with the new generated password
use admin
db.changeUserPassword("<MONGODB_RS0_ADMIN_USER>", "NEW_PASSWORD_PLACEHOLDER")
// Verify:
db.auth("<MONGODB_RS0_ADMIN_USER>", "NEW_PASSWORD_PLACEHOLDER")
// Returns: { ok: 1 }

1c. Update ALL FIVE SOPS files — in the SAME shell invocation as 1b

Critical: generate the new password, run 1b (live change), and write it to every file below in ONE continuous script/session. Do not split this across separate tool calls or terminal sessions — the value only exists in memory until it’s written, and shell state does not persist between invocations.

Files to update (see SOPS File Map above for exact key casing per file): secrets/administration.env.sops, secrets/bms-servers.env.sops, secrets/mongodb-bms.env.sops, secrets/n8n-bms4.env.sops, secrets/role-secret-manager.env.sops.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
 
# Repeat this block for EACH of the 5 files (adjust $keyName to match that file's casing)
foreach ($pair in @(
    @{file="secrets\administration.env.sops"; key="MONGODB_RS0_ADMIN_PASSWORD"},
    @{file="secrets\bms-servers.env.sops"; key="mongodb_rs0_admin_password"},
    @{file="secrets\mongodb-bms.env.sops"; key="MONGODB_RS0_ADMIN_PASSWORD"},
    @{file="secrets\n8n-bms4.env.sops"; key="MONGODB_RS0_ADMIN_PASSWORD"},
    @{file="secrets\role-secret-manager.env.sops"; key="MONGODB_RS0_ADMIN_PASSWORD"}
)) {
    $plain = sops --decrypt --input-type dotenv --output-type dotenv $pair.file
    $updated = $plain | ForEach-Object {
        if ($_ -match "^$($pair.key)=") { "$($pair.key)=$env:NEW_PASS" } else { $_ }
    }
    [System.IO.File]::WriteAllText("$PWD\$($pair.file)", ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
    sops --encrypt --input-type dotenv --output-type dotenv --in-place $pair.file
    sops --decrypt --input-type dotenv --output-type dotenv $pair.file | Out-Null
    if ($LASTEXITCODE -ne 0) { throw "SOPS canary FAILED on $($pair.file) — do NOT commit" }
    Write-Host "$($pair.file): canary OK"
}

1d. Update GitHub Secret (Tier 1 credentials mirror to GH Secrets too)

gh secret set MONGODB_RS0_ADMIN_PASSWORD --body $env:NEW_PASS --repo radieu/p24-infra
$env:NEW_PASS = ""

1e. Recreate the Pinbox24 app containers on bms-1 — DEPRECATED since PR 6c3c5b03

This step is no longer needed when rotating the admin password. v32-prod now connects as w3_app and v42-prod as w4_app — dedicated app users, not admin. Admin password rotation does NOT affect Pinbox24 w3/w4 availability.

To rotate Pinbox24 MongoDB app credentials, use the autonomous scripts:

• scripts/rotate/w3_app_mongodb_password.sh (W3, v32-prod)
• scripts/rotate/w4_app_mongodb_password.sh (W4, v42-prod)

Full procedure: docs/playbooks/w3-mongodb-credential-rotation.md / docs/playbooks/w4-mongodb-credential-rotation.md

The autonomous scripts use --force-recreate (added 2026-07-03, issue fix for #2702). --force-recreate is required — without it Docker reuses the running container’s cached env (old password still baked in) even after backend-environment.env is updated on disk. The W4 outage on 2026-07-05 was caused by a rotation that updated the env file but did not force-recreate the container. Always use --force-recreate when recreating these containers.

The information below is kept for historical reference only (pre-6c3c5b03).

⚠️ Build-dir paths below can drift (#4985). The authoritative W3 (v32-prod) compose dir is whatever the sync-pinbox24-w3 job in .github/workflows/secrets-sync.yml deploys to — currently the GitLab-CI-runner-owned /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/, not the root-owned /root/builds/pn3C9eHo/... that older revisions of this doc named. That older copy is not the prod serving dir (editing it has zero effect on v32-prod) but is not abandoned either — it is an active W3-staging mount source, so do not delete it. Always ground-truth the live dir before editing anything: ssh root@94.23.26.113 "docker inspect <container> --format '{{index .Config.Labels \"com.docker.compose.project.working_dir\"}}'".

Container	Build dir on bms-1	Env keys to update	Image
v42-prod (w4)	/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/	NEW_MONGODB_URI	563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod
v32-prod (w3)	/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/	MONGODB_URL and PMONGODB_URL (both!)	563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod

# Extract the new admin password silently and URL-encode it (never print the raw value)
$env:NEW_MG_PASS = (sops --decrypt --input-type dotenv --output-type dotenv secrets\bms-servers.env.sops `
  | Select-String "^mongodb_rs0_admin_password=").ToString().Split("=",2)[1]
$encoded = [System.Uri]::EscapeDataString($env:NEW_MG_PASS)
 
# For EACH container: update its backend-environment.env on bms-1 (pipe a Python script via SSH stdin
# so the password never appears in process args), then force-recreate via docker-compose --no-build:
#   v42-prod -> set NEW_MONGODB_URI
#   v32-prod -> set BOTH MONGODB_URL and PMONGODB_URL (updating only one still causes 422/504)
# NOTE: --force-recreate is mandatory -- plain "up -d" reuses the running container's cached env.
ssh root@94.23.26.113 @'
cd /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts
CONTAINER_NAME=v42-prod IMAGE_NAME=563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod docker-compose up -d --no-build --force-recreate backend
'@
ssh root@94.23.26.113 @'
cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
CONTAINER_NAME=v32-prod IMAGE_NAME=563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod docker-compose up -d --no-build --force-recreate backend
'@
$env:NEW_MG_PASS = ""; $encoded = ""

Verify both web apps recover (must return 200):

(Invoke-WebRequest "https://api.w4.pinbox24.com/api/i18n/langs" -TimeoutSec 10 -UseBasicParsing).StatusCode  # w4
(Invoke-WebRequest "https://api.w3.pinbox24.com/api/i18n/langs" -TimeoutSec 10 -UseBasicParsing).StatusCode  # w3

Full env-file update procedure, the two-connection gotcha for v32-prod, and the pm2 env 0 credential-leak warning are documented in docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md (Fix B / Fix C).

Do NOT use docker exec ... pm2 restart --update-env as the permanent fix — that change is lost on docker restart. Always update the env file and recreate the container as above.

---

Step 2 — Rotate the Prometheus Monitoring Password

2-lock. Acquire the rotation lock FIRST — MANDATORY (ADR 004)

db.changeUserPassword("prometheus", …) below is a non-idempotent live server-side write, so it needs its own lock — acquire it here even if you did NOT run Step 1 (rotating only the Prometheus password is a supported standalone path). Do not reuse Step 1’s admin lock; the lock is keyed per secret name:

# Distinct variable so it does not clobber Step 1's admin $LOCK_ID if you run both in one session.
$env:PROM_LOCK_ID = python scripts/rotation-log-entry.py acquire --secret MONGODB_RS0_PROMETHEUS_PASSWORD --repo p24-infra --reason "#<issue>"
if ($LASTEXITCODE -ne 0) { throw "acquire denied/unverified — STOP, reconcile, do NOT rotate prometheus" }

Exit 3 = another session holds it (reconcile, do not blind-overwrite); exit 4 = Supabase unreachable (fail-closed). $PROM_LOCK_ID is the pending row (do not also open one). Linux worker: use python3 and || exit 1. Release with close/fail after distribution (Step 3), same as Step 1.

2a. Connect to PRIMARY and change password

ssh ubuntu@HOST  # PRIMARY node
 
mongosh --host localhost:27017 -u "<MONGODB_RS0_ADMIN_USER>" -p "<MONGODB_RS0_ADMIN_PASSWORD>" --authenticationDatabase admin

Immediately before the changeUserPassword write, re-check the lock is still held (Layer 2):

python scripts/rotation-log-entry.py check $env:PROM_LOCK_ID
if ($LASTEXITCODE -ne 0) { throw "lock lost — abort the live write and reconcile (do NOT overwrite blind)" }

// In mongosh
use admin
db.changeUserPassword("prometheus", "NEW_PROMETHEUS_PASSWORD_PLACEHOLDER")
// Verify:
db.auth("prometheus", "NEW_PROMETHEUS_PASSWORD_PLACEHOLDER")
// Returns: { ok: 1 }

2b. Update SOPS for n8n-bms4

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
 
$plain = sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops
$tempPath = "$PWD\secrets\n8n-bms4-edit.env.sops"
 
$updated = $plain | ForEach-Object {
    if ($_ -match '^MONGODB_RS0_PROMETHEUS_PASSWORD=') {
        "MONGODB_RS0_PROMETHEUS_PASSWORD=NEW_PROMETHEUS_PASSWORD_PLACEHOLDER"
    } else { $_ }
}
[System.IO.File]::WriteAllText($tempPath, ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
$enc = sops --encrypt --input-type dotenv --output-type dotenv $tempPath
[System.IO.File]::WriteAllText("$PWD\secrets\n8n-bms4.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS canary FAILED — do NOT commit" }
Write-Host "Canary OK"
[System.IO.File]::Delete($tempPath)

Warning — special characters in URI: If the new MONGODB_RS0_PROMETHEUS_PASSWORD contains @, #, /, or ?, the mongodb URI parser will break. See docs/playbooks/mongodb-exporter-uri-special-chars.md for the URL-encoding workaround. Generate passwords that avoid these characters (alphanumeric + -_ only).

2c. Commit and push to trigger secrets-sync

git add secrets\n8n-bms4.env.sops secrets\monitoring.env.sops
git commit -m "chore: rotate MongoDB rs0 credentials"
git push
# Open PR targeting main — secrets-sync.yml deploys on merge

2d. Restart mongodb-exporter containers on bms-4

After secrets-sync.yml runs and updates /opt/p24-infra/bms-4/.env on bms-4:

ssh root@54.36.123.110
cd /opt/p24-infra/bms-4
docker compose up -d --no-deps mongodb-exporter-bms2 mongodb-exporter-bms3

---

Step 3 — Verification

Verify admin access

ssh ubuntu@HOST
mongosh --host localhost:27017 -u "<MONGODB_RS0_ADMIN_USER>" -p "<new-password>" \
  --authenticationDatabase admin --eval "rs.status().ok"
# Expected: 1

Verify prometheus monitoring

Check Grafana or Prometheus for mongodb_exporter metrics:

# On bms-4 — check exporter health
curl -s http://localhost:9216/metrics | grep -c "mongodb_"
curl -s http://localhost:9217/metrics | grep -c "mongodb_"
# Both should return > 0 lines

Check Prometheus targets at https://prometheus.vps-i1.infra.zintegrowana.online/targets — both mongodb-exporter-bms2 and mongodb-exporter-bms3 should show UP.

Verify no authentication errors

# On bms-2 (PRIMARY)
ssh ubuntu@145.239.133.104 "mongosh --quiet --eval 'db.adminCommand({getLog:\"global\"}).log.slice(-20).join(\"\n\")'" \
  | grep -i "auth"
# Should show no recent authentication failures

---

Step 4 — Log the Rotation

Append a row to docs/secrets-rotation-log.md:

| YYYY-MM-DD HH:MM UTC | — | MONGODB_RS0_ADMIN_PASSWORD | Scheduled quarterly | <your-name> | yes |
| YYYY-MM-DD HH:MM UTC | — | MONGODB_RS0_PROMETHEUS_PASSWORD | Scheduled quarterly | <your-name> | yes |

---

Escalation

Symptom	Action
mongosh auth fails after rotation, but admin_backup still works	This is the expected, low-risk case. Authenticate as admin_backup (normal mongosh -u admin_backup -p <from GH Secrets> --authenticationDatabase admin) and run db.changeUserPassword("<admin_user>", "<new-password>") as a completely normal authenticated operation — no mongod restart, no standalone mode, no replica set disruption. Then persist to all 5 SOPS files per Step 1c.
mongosh auth fails for BOTH admin and admin_backup (double lockout)	Full standalone-mode recovery required — see “Double-Lockout Recovery” below. This is a last resort; it takes a live PRIMARY out of the replica set temporarily (bms-2/bms-3 will elect a new PRIMARY during the window — a normal, designed-for failover, not data loss).
mongodb-exporter containers crash after restart	Check logs: docker compose logs mongodb-exporter-bms2. URI special char issue — see docs/playbooks/mongodb-exporter-uri-special-chars.md.
rs.isMaster() returns false on both bms-2 and bms-3	Replica set election in progress — wait 30s and retry. If persists > 2 min, check network between bms-2, bms-3, bms-4 (arbiter).
SOPS canary fails after write	Do NOT commit. See docs/playbooks/sops-windows-crlf.md for recovery.

Double-Lockout Recovery (both admin and admin_backup unavailable)

Tested and verified 2026-07-09 during incident #3494 recovery. The MongoDB “localhost exception” referenced in older versions of this doc does NOT work here — it only applies when NO users exist yet on the deployment, and this cluster has users. Disabling only security.authorization while leaving security.keyFile + replication.replSetName in place also does not work — the node can no longer participate in replica-set consensus and never becomes writable again until reverted. The only procedure that actually works is running the node as a genuine standalone (replication disabled too), fully disconnected from the replica set for the duration.

1. On the current PRIMARY (find with rs.isMaster().ismaster, Step 0), back up /etc/mongod.conf.
2. Comment out both the replication:/replSetName: lines and the security:/keyFile:/authorization: lines. (Disabling security alone is not sufficient — see note above.)
3. sudo systemctl restart mongod. The node now runs as a standalone with no auth. The rest of the replica set (other data-bearing member + arbiter) elects a new PRIMARY during this window — expected, not an incident.
4. Wrap ALL of the following in a trap ... EXIT on the remote shell that restores the backed-up config and restarts mongod, so a mid-script failure can never leave the node running without auth:
   • db.getSiblingDB('admin').runCommand({ping:1}) — confirm standalone unauthenticated access works.
   • Reset/create admin_backup FIRST (create-if-missing, else changeUserPassword) — see Design Rule #1 above; never reset the main admin account before a working fallback exists, even during emergency recovery.
   • Only then reset admin’s password.
5. Let the trap restore the original config and restart mongod. Poll rs.status().myState (needs auth now — use the NEW backup password) until it returns 1 or 2 (rejoined as PRIMARY or SECONDARY).
6. In the SAME shell session (before it exits): write the new admin password to all 5 SOPS files (Step 1c) AND the new admin_backup credential to GH Secrets ONLY (never to the same SOPS files as admin). Canary-verify every SOPS write.
7. Verify: fresh mongosh auth as both admin and admin_backup, plus rs.status().members shows all 3 nodes healthy (PRIMARY/SECONDARY/ARBITER).
8. File a GH issue documenting the double-lockout (what caused it, which accounts were reset) — this should never happen if Step 0b is followed on every rotation.

---

Prevention

• Never rotate the sole admin-equivalent credential without a working admin_backup fallback confirmed first (Step 0b). This is the single biggest lesson from incident #3494.
• Generate → use remotely → persist to permanent store, all in ONE shell invocation. A password value held only in a shell variable is gone the moment that process exits — there is no way to read it back from MongoDB (password hashes are one-way) or from a terminated process.
• Generate passwords with openssl rand -base64 24 | tr -dc 'A-Za-z0-9' (alphanumeric-only avoids both URI-unsafe characters AND shell-quoting edge cases).
• Always rotate both users together on the same maintenance window to avoid partial rotation state.
• Update all 5 SOPS files in the same operation — a partial update silently reintroduces the class of stale-credential drift found during incident #3494 (n8n-bms4.env.sops held an old value while the other 4 held current).
• When extracting a value from a SOPS/env file by key name, prefer exact-match tools (sops --output-type json + a JSON-key lookup, or awk -F= '$1==key') over grep '^KEY=' — an unanchored or substring-prone pattern can match a similarly-named key (e.g. MONGODB_URL= matching inside PMONGODB_URL=) and leak an unrelated value. This exact bug caused the credential exposure that triggered incident #3494 in the first place.
• Step 1e is deprecated — since PR 6c3c5b03, v32-prod and v42-prod use w3_app/w4_app dedicated users, not the admin credential. Admin rotation no longer affects Pinbox24 availability. To rotate Pinbox24 app MongoDB passwords, use scripts/rotate/w3_app_mongodb_password.sh / w4_app_mongodb_password.sh.
• The secrets-sync.yml workflow auto-deploys SOPS changes to servers on merge to dev or main — no manual scp needed.
• Add a calendar reminder for the next rotation (~3 months from now) immediately after completing this one.

---

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="MONGODB_RS0_ADMIN_PASSWORD",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — MongoDB rs0 admin password rotated across bms-2/bms-3",
    env="bms-2",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'MONGODB_RS0_ADMIN_PASSWORD', 'success', 'Scheduled rotation — MongoDB rs0 admin password rotated across bms-2/bms-3', 'bms-2')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''