p24-infra Docs

mongodb-admin-password-recovery

7 min read

Playbook: MongoDB rs0 Admin Password Recovery

Applies to: MONGODB_RS0_ADMIN_PASSWORD in secrets/monitoring.env.sops
Trigger: Admin auth fails on both PRIMARY and SECONDARY; SOPS value doesn’t match any server

What triggers this problem

  • MongoDB admin auth fails with Authentication failed on all rs0 members
  • SOPS value was rotated but MongoDB password change never propagated
  • Previous recovery attempt set an unknown/lost password (e.g. stored only in a dropped git stash)
  • Standalone mongod was used for recovery but the change bypassed the oplog, causing split-brain

Key facts about this architecture

NodeIPRoledata dir
bms-2145.239.133.104Full member (priority 1)/var/lib/mongodb
bms-351.68.155.224Full member (priority 1)/var/lib/mongodb
bms-454.36.123.110ARBITERn/a (no data)

WARNING — standalone mode creates oplog split-brain. If you use mongod --noauth to reset a password directly against the data directory, that change bypasses the oplog. When the node rejoins as SECONDARY, it applies oplog from the PRIMARY — which does NOT include your standalone change. The result: two nodes with different passwords that both claim to be “up to date”. Fix by running changeUserPassword on the PRIMARY through the authenticated shell (after you recover auth on the PRIMARY).

WARNING — keyFile disables authorization: disabled. If security.keyFile is set in mongod.conf, MongoDB enforces auth even if you add authorization: disabled. Use mongod --noauth --port <different-port> to bypass.

WARNING — file ownership. If you run sudo mongod as root against the data directory, it creates files owned by root. The service runs as mongodb user and will fail with Permission denied. Fix: sudo chown -R mongodb:mongodb /var/lib/mongodb before starting the service.

Recovery procedure

Step 0 — Identify which node has auth you can test

$key = "C:\Users\konar\.ssh\id_ed25519"
$SO = @("-o","StrictHostKeyChecking=no","-o","BatchMode=yes","-i",$key)
 
# Try SOPS value on each node
$env:_MADM = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").Line.Split("=",2)[1]
ssh @SO ubuntu@145.239.133.104 "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'db.version()' 2>&1 | tail -1"
ssh @SO ubuntu@51.68.155.224 "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'db.version()' 2>&1 | tail -1"
$env:_MADM = ""
 
# Also try rs.isMaster (unauthenticated) to find current PRIMARY
ssh @SO ubuntu@145.239.133.104 "mongosh --quiet localhost/admin --eval 'rs.isMaster().primary' 2>&1 | tail -1"

Case A: Both nodes fail auth (total lockout)

Use standalone recovery on ONE node (prefer bms-2 if it’s PRIMARY, otherwise whichever):

Step 1: Stop bms-3

ssh ubuntu@51.68.155.224 "sudo systemctl stop mongod && echo stopped"

Step 2: Stop bms-2 main mongod

ssh ubuntu@145.239.133.104 "sudo systemctl stop mongod && echo stopped"

Step 3: Start standalone mongod on bms-2 (different port, —noauth)

ssh ubuntu@145.239.133.104 "sudo mongod --port 27018 --noauth --dbpath /var/lib/mongodb --fork --logpath /tmp/sa.log && echo standalone-started"

Wait 5 seconds, then verify it’s running:

ssh ubuntu@145.239.133.104 "pgrep -a mongod | grep 27018"

Step 4: Reset admin password on standalone

Write the script to a local temp file to avoid quoting issues:

# chpass_recovery.py
import base64, subprocess, os
admin_new = base64.b64decode(open("/tmp/._mn").read().strip()).decode()
r = subprocess.run(
    ["mongosh","--quiet","127.0.0.1:27018/admin","--eval",
     'db.changeUserPassword("admin", "' + admin_new + '")'],
    capture_output=True, text=True
)
print("OK" if r.returncode == 0 else "FAIL: " + r.stderr[:200])
os.remove("/tmp/._mn")
# On Windows: encode new password as b64, send to server
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:_MADM = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").Line.Split("=",2)[1]
$b64 = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($env:_MADM))
ssh @SO ubuntu@145.239.133.104 "echo '$b64' > /tmp/._mn && echo written"
scp @SO chpass_recovery.py ubuntu@145.239.133.104:/tmp/._cr.py
ssh @SO ubuntu@145.239.133.104 "python3 /tmp/._cr.py; rm -f /tmp/._cr.py"
$env:_MADM = ""

Step 5: Kill standalone, fix ownership, start main mongod on bms-2

ssh ubuntu@145.239.133.104 "sudo pkill -f 'mongod.*27018' && sleep 2 && sudo chown -R mongodb:mongodb /var/lib/mongodb && sudo systemctl start mongod && sleep 8 && systemctl is-active mongod"

Step 6: Start bms-3

ssh ubuntu@51.68.155.224 "sudo systemctl start mongod && sleep 5 && systemctl is-active mongod"

Step 7: Determine which node is PRIMARY after election

rs.isMaster() is unauthenticated:

ssh ubuntu@145.239.133.104 "mongosh --quiet localhost/admin --eval 'rs.isMaster().primary' 2>&1 | tail -1"

Case B: One node has auth, other doesn’t (split-brain from standalone)

This happens when standalone changed bms-X but the change didn’t go through the oplog.

If the working node is PRIMARY: Run changeUserPassword on it — the oplog entry replicates to secondary automatically:

# Write Python script locally, SCP to PRIMARY
# Script: read b64 tokens from files, run changeUserPassword on PRIMARY
# See Case A Step 4 for script pattern, but connect to port 27017 with current working password

Wait 20–30 seconds, then test auth on the other node.

If the working node is SECONDARY: Stop the NON-WORKING primary so the working secondary can become PRIMARY:

ssh ubuntu@51.68.155.224 "sudo systemctl stop mongod"  # stop the node that has unknown password
# Wait 20s for election
ssh ubuntu@145.239.133.104 "mongosh --quiet localhost/admin --eval 'rs.isMaster().ismaster' 2>&1 | tail -1"
# Should return: true
# Now run changeUserPassword on the new PRIMARY (the node that was SECONDARY)
# Then restart the stopped node — it replicates the new password
ssh ubuntu@51.68.155.224 "sudo systemctl start mongod"

Step 8: Normalize password across the RS (always run after any recovery)

After any recovery, ensure the SOPS password is set on the PRIMARY via authenticated shell — this creates an oplog entry that propagates to all secondaries:

# Get SOPS password + run changeUserPassword on confirmed PRIMARY
$env:_MADM = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "^MONGODB_RS0_ADMIN_PASSWORD=").Line.Split("=",2)[1]
$b64 = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($env:_MADM))
ssh @SO ubuntu@<PRIMARY_IP> "echo '$b64' > /tmp/._mn && echo written"
# SCP + run a script that reads /tmp/._mn and runs db.changeUserPassword("admin", value)
$env:_MADM = ""

Wait 20s, then verify auth on all nodes.

Step 9: Rotate prometheus user if needed

After admin is working, also rotate the prometheus MongoDB user:

$env:_MPRO = (sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops | Select-String "^MONGODB_RS0_PROMETHEUS_PASSWORD=").Line.Split("=",2)[1]
# SCP script to PRIMARY, run changeUserPassword("prometheus", new_value)
# Update /etc/systemd/system/mongodb_exporter.service on bms-2 AND bms-3:
$encoded = [Uri]::EscapeDataString($env:_MPRO)
ssh @SO ubuntu@145.239.133.104 "sudo sed -i 's|prometheus:[^@]*@localhost|prometheus:$encoded@localhost|' /etc/systemd/system/mongodb_exporter.service && sudo systemctl daemon-reload && sudo systemctl restart mongodb_exporter"
ssh @SO ubuntu@51.68.155.224 "sudo sed -i 's|prometheus:[^@]*@localhost|prometheus:$encoded@localhost|' /etc/systemd/system/mongodb_exporter.service && sudo systemctl daemon-reload && sudo systemctl restart mongodb_exporter"
$env:_MPRO = ""

Verification

# Admin auth on both nodes
foreach ($ip in @("145.239.133.104","51.68.155.224")) {
    $v = ssh @SO ubuntu@$ip "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'db.version()' 2>&1 | tail -1"
    Write-Host "$ip : $v"
}
 
# RS status (from PRIMARY)
ssh @SO ubuntu@145.239.133.104 "mongosh --quiet localhost/admin -u admin -p '$($env:_MADM)' --eval 'JSON.stringify(rs.status().members.map(function(m){return m.name+\":\"+m.stateStr}))' 2>&1 | tail -1"
 
# Exporter endpoints
foreach ($ip in @("145.239.133.104","51.68.155.224")) {
    ssh @SO ubuntu@$ip "curl -s http://localhost:9216/metrics | grep -m1 rs_nm=" 2>/dev/null
}

Escalation

If standalone recovery fails (data corruption, keyFile mismatch, WiredTiger errors):

  1. Check /var/log/mongodb/mongod.log for the specific error
  2. For Permission denied on WiredTiger.turtle: sudo chown -R mongodb:mongodb /var/lib/mongodb then retry start
  3. For keyFile mismatch: verify /etc/mongodb-keyfile exists and is identical on both nodes (copy from one to the other)
  4. For data corruption: last resort is restoring from Wasabi S3 backup (Thanos stores metrics, not MongoDB data — check separate backup scripts)

Prevention

  • The prometheus user had a weak known password (p24@pro_Pass) — this triggered a rotation as part of incident #1506
  • Always store new passwords in SOPS BEFORE applying to MongoDB servers
  • Run changeUserPassword on the PRIMARY (not standalone) whenever possible
  • After any maintenance that involves standalone mode, always run Step 8 to normalize via oplog

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="MONGODB_RS0_ADMIN_PASSWORD",
    result="success",  # "success" | "failed" | "skipped"
    detail="Emergency recovery — MongoDB admin password reset and SOPS updated",
    env="bms-2",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'MONGODB_RS0_ADMIN_PASSWORD', 'success', 'Emergency recovery — MongoDB admin password reset and SOPS updated', 'bms-2')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks