p24-infra Docs

mongodb-admin-password-recovery

8 min read

Playbook: MongoDB rs0 Admin Password Recovery

Applies to: MONGODB_RS0_ADMIN_PASSWORD in secrets/mongodb-bms.env.sops
Trigger: Admin auth fails on both PRIMARY and SECONDARY; SOPS value doesn’t match any server

Access note (verified 2026-10-05). All rs0 nodes are reached as root@<ip> by SSH key only (C:\Users\konar\.ssh\id_ed25519) — there is no ubuntu user. MongoDB rs0 admin + prometheus credentials live in secrets/mongodb-bms.env.sops (credential-isolation split, #2728), not secrets/monitoring.env.sops / secrets/n8n-bms4.env.sops. The mongodb-exporter is two Docker containers on bms-4 (see Step 9), not a systemd unit on the data nodes.

What triggers this problem

  • MongoDB admin auth fails with Authentication failed on all rs0 members
  • SOPS value was rotated but MongoDB password change never propagated
  • Previous recovery attempt set an unknown/lost password (e.g. stored only in a dropped git stash)
  • Standalone mongod was used for recovery but the change bypassed the oplog, causing split-brain

Key facts about this architecture

NodeIPRoledata dir
bms-2145.239.133.104Full member (priority 1)/var/lib/mongodb
bms-351.68.155.224Full member (priority 1)/var/lib/mongodb
bms-454.36.123.110ARBITERn/a (no data)

WARNING — standalone mode creates oplog split-brain. If you use mongod --noauth to reset a password directly against the data directory, that change bypasses the oplog. When the node rejoins as SECONDARY, it applies oplog from the PRIMARY — which does NOT include your standalone change. The result: two nodes with different passwords that both claim to be “up to date”. Fix by running changeUserPassword on the PRIMARY through the authenticated shell (after you recover auth on the PRIMARY).

WARNING — keyFile disables authorization: disabled. If security.keyFile is set in mongod.conf, MongoDB enforces auth even if you add authorization: disabled. Use mongod --noauth --port <different-port> to bypass.

WARNING — file ownership. If you run sudo mongod as root against the data directory, it creates files owned by root. The service runs as mongodb user and will fail with Permission denied. Fix: sudo chown -R mongodb:mongodb /var/lib/mongodb before starting the service.

Recovery procedure

Step 0 — Identify which node has auth you can test

$key = "C:\Users\konar\.ssh\id_ed25519"
$SO = @("-o","StrictHostKeyChecking=no","-o","BatchMode=yes","-i",$key)
 
# Try SOPS value on each node
$env:_MADM = (sops --decrypt --input-type dotenv --output-type dotenv secrets\mongodb-bms.env.sops | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").Line.Split("=",2)[1]
ssh @SO root@145.239.133.104 "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'db.version()' 2>&1 | tail -1"
ssh @SO root@51.68.155.224 "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'db.version()' 2>&1 | tail -1"
$env:_MADM = ""
 
# Also try rs.isMaster (unauthenticated) to find current PRIMARY
ssh @SO root@145.239.133.104 "mongosh --quiet localhost/admin --eval 'rs.isMaster().primary' 2>&1 | tail -1"

Case A: Both nodes fail auth (total lockout)

Use standalone recovery on ONE node (prefer bms-2 if it’s PRIMARY, otherwise whichever):

Step 1: Stop bms-3

ssh root@51.68.155.224 "sudo systemctl stop mongod && echo stopped"

Step 2: Stop bms-2 main mongod

ssh root@145.239.133.104 "sudo systemctl stop mongod && echo stopped"

Step 3: Start standalone mongod on bms-2 (different port, —noauth)

ssh root@145.239.133.104 "sudo mongod --port 27018 --noauth --dbpath /var/lib/mongodb --fork --logpath /tmp/sa.log && echo standalone-started"

Wait 5 seconds, then verify it’s running:

ssh root@145.239.133.104 "pgrep -a mongod | grep 27018"

Step 4: Reset admin password on standalone

Write the script to a local temp file to avoid quoting issues:

# chpass_recovery.py
import base64, subprocess, os
admin_new = base64.b64decode(open("/tmp/._mn").read().strip()).decode()
r = subprocess.run(
    ["mongosh","--quiet","127.0.0.1:27018/admin","--eval",
     'db.changeUserPassword("admin", "' + admin_new + '")'],
    capture_output=True, text=True
)
print("OK" if r.returncode == 0 else "FAIL: " + r.stderr[:200])
os.remove("/tmp/._mn")
# On Windows: encode new password as b64, send to server
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:_MADM = (sops --decrypt --input-type dotenv --output-type dotenv secrets\mongodb-bms.env.sops | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").Line.Split("=",2)[1]
$b64 = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($env:_MADM))
ssh @SO root@145.239.133.104 "echo '$b64' > /tmp/._mn && echo written"
scp @SO chpass_recovery.py root@145.239.133.104:/tmp/._cr.py
ssh @SO root@145.239.133.104 "python3 /tmp/._cr.py; rm -f /tmp/._cr.py"
$env:_MADM = ""

Step 5: Kill standalone, fix ownership, start main mongod on bms-2

ssh root@145.239.133.104 "sudo pkill -f 'mongod.*27018' && sleep 2 && sudo chown -R mongodb:mongodb /var/lib/mongodb && sudo systemctl start mongod && sleep 8 && systemctl is-active mongod"

Step 6: Start bms-3

ssh root@51.68.155.224 "sudo systemctl start mongod && sleep 5 && systemctl is-active mongod"

Step 7: Determine which node is PRIMARY after election

rs.isMaster() is unauthenticated:

ssh root@145.239.133.104 "mongosh --quiet localhost/admin --eval 'rs.isMaster().primary' 2>&1 | tail -1"

Case B: One node has auth, other doesn’t (split-brain from standalone)

This happens when standalone changed bms-X but the change didn’t go through the oplog.

If the working node is PRIMARY: Run changeUserPassword on it — the oplog entry replicates to secondary automatically:

# Write Python script locally, SCP to PRIMARY
# Script: read b64 tokens from files, run changeUserPassword on PRIMARY
# See Case A Step 4 for script pattern, but connect to port 27017 with current working password

Wait 20–30 seconds, then test auth on the other node.

If the working node is SECONDARY: Stop the NON-WORKING primary so the working secondary can become PRIMARY:

ssh root@51.68.155.224 "sudo systemctl stop mongod"  # stop the node that has unknown password
# Wait 20s for election
ssh root@145.239.133.104 "mongosh --quiet localhost/admin --eval 'rs.isMaster().ismaster' 2>&1 | tail -1"
# Should return: true
# Now run changeUserPassword on the new PRIMARY (the node that was SECONDARY)
# Then restart the stopped node — it replicates the new password
ssh root@51.68.155.224 "sudo systemctl start mongod"

Step 8: Normalize password across the RS (always run after any recovery)

After any recovery, ensure the SOPS password is set on the PRIMARY via authenticated shell — this creates an oplog entry that propagates to all secondaries:

# Get SOPS password + run changeUserPassword on confirmed PRIMARY
$env:_MADM = (sops --decrypt --input-type dotenv --output-type dotenv secrets\mongodb-bms.env.sops | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").Line.Split("=",2)[1]
$b64 = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($env:_MADM))
ssh @SO root@<PRIMARY_IP> "echo '$b64' > /tmp/._mn && echo written"
# SCP + run a script that reads /tmp/._mn and runs db.changeUserPassword("admin", value)
$env:_MADM = ""

Wait 20s, then verify auth on all nodes.

Step 9: Rotate prometheus user if needed

After admin is working, also rotate the prometheus MongoDB user and re-point the exporters at it.

Exporter topology (verified 2026-10-05). mongodb-exporter is NOT a systemd unit on the data nodes. It runs as two Docker containers on bms-4 (percona/mongodb_exporter:0.52.0), defined in bms-4/docker-compose.yml:

ContainerScrapesLocal port on bms-4
bms-4-mongodb-exporter-bms2-1bms-2 (145.239.133.104:27017):9216
bms-4-mongodb-exporter-bms3-1bms-3 (51.68.155.224:27017):9217

There is no mongodb_exporter.service systemd unit and nothing serves :9216 on bms-2/bms-3. Each container builds its URI from MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED in /opt/p24-infra/bms-4/.env at docker compose up time. See mongonoprimary-bms4-exporter-link-loss.md (#7209) for the full model and the exporter-password footgun.

  1. Change the password on the PRIMARY (authenticated), reading the new value from SOPS:

    $env:_MPRO = (sops --decrypt --input-type dotenv --output-type dotenv secrets\mongodb-bms.env.sops | Select-String "^MONGODB_RS0_PROMETHEUS_PASSWORD=").Line.Split("=",2)[1]
# SCP a script to the PRIMARY and run db.changeUserPassword("prometheus", <new_value>) there.
$env:_MPRO = ""

  2. Re-point the bms-4 exporter containers at the new password (operational SOPS read, no SOPS write — run on the bms-4 host; a sys-admin/infra-task operation, not a code change):

    cd /opt/p24-infra/bms-4
export MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED=$(sops -d --input-type dotenv --output-type dotenv \
  <repo>/secrets/mongodb-bms.env.sops | grep '^MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED=' | cut -d= -f2- | tr -d '"')
docker compose up -d --force-recreate mongodb-exporter-bms2 mongodb-exporter-bms3
unset MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED

    Ensure MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED is present in /opt/p24-infra/bms-4/.env (secrets-sync target) so a plain docker compose up --force-recreate does not bake an empty password into the exporter — that is a known footgun (#7209).

Verification

# Admin auth on both nodes
foreach ($ip in @("145.239.133.104","51.68.155.224")) {
    $v = ssh @SO root@$ip "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'db.version()' 2>&1 | tail -1"
    Write-Host "$ip : $v"
}
 
# RS status — run from the PRIMARY, authenticated. NOTE: on the data nodes (bms-2/bms-3)
# `rs.status()` requires auth; only `rs.isMaster()` / `db.hello()` work unauthenticated there.
# So always pass `-u admin -p` for rs.status(), or use rs.isMaster() if you have no working creds.
ssh @SO root@145.239.133.104 "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'JSON.stringify(rs.status().members.map(function(m){return m.name+\":\"+m.stateStr}))' 2>&1 | tail -1"
 
# Exporter endpoints — the two exporters run on bms-4 (NOT the data nodes): :9216 scrapes bms-2,
# :9217 scrapes bms-3. Check from the bms-4 host.
ssh @SO root@54.36.123.110 "curl -s http://localhost:9216/metrics | grep -m1 mongodb_up"   # bms-2 target
ssh @SO root@54.36.123.110 "curl -s http://localhost:9217/metrics | grep -m1 mongodb_up"   # bms-3 target

Escalation

If standalone recovery fails (data corruption, keyFile mismatch, WiredTiger errors):

  1. Check /var/log/mongodb/mongod.log for the specific error
  2. For Permission denied on WiredTiger.turtle: sudo chown -R mongodb:mongodb /var/lib/mongodb then retry start
  3. For keyFile mismatch: verify /etc/mongodb-keyfile exists and is identical on both nodes (copy from one to the other)
  4. For data corruption: last resort is restoring from Wasabi S3 backup (Thanos stores metrics, not MongoDB data — check separate backup scripts)

Prevention

  • The prometheus user had a weak known password (p24@pro_Pass) — this triggered a rotation as part of incident #1506
  • Always store new passwords in SOPS BEFORE applying to MongoDB servers
  • Run changeUserPassword on the PRIMARY (not standalone) whenever possible
  • After any maintenance that involves standalone mode, always run Step 8 to normalize via oplog

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="MONGODB_RS0_ADMIN_PASSWORD",
    result="success",  # "success" | "failed" | "skipped"
    detail="Emergency recovery — MongoDB admin password reset and SOPS updated",
    env="bms-2",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'MONGODB_RS0_ADMIN_PASSWORD', 'success', 'Emergency recovery — MongoDB admin password reset and SOPS updated', 'bms-2')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks