Playbook: MongoDB rs0 — Full Restore from Wasabi Backup

Full disaster-recovery runbook for MongoDB replica set rs0 in the scenario where bms-2 AND bms-3 are both lost (or corrupted beyond repair). This covers Scenario B from bms1-dr-plan.md — the step-by-step rebuild that playbook left incomplete.

RPO: up to 7 days (tygodniowy backup) — or up to 1 day after issue #2367 is resolved (daily cron). RTO: 2–4 hours (estimated; first measured drill pending — see issue #2148).

rs0 member map:

Role	Host	IP
PRIMARY (rebuild target)	bms-2	145.239.133.104
SECONDARY (rebuild target)	bms-3	51.68.155.224
ARBITER (survives, config may be stale)	bms-4	54.36.123.110

When to use this playbook

Both bms-2 and bms-3 are unrecoverable (hardware failure, data corruption, OVH loss)
rs0 has no PRIMARY and ARBITER alone cannot elect one (no quorum)
mongosh --eval "rs.status()" returns "stateStr": "REMOVED" or connection refused on both bms-2 and bms-3

Confirm disaster before starting:

mongosh --host 145.239.133.104:27017 --eval "rs.status().myState" # Should return 1 (PRIMARY) — if offline → disaster
mongosh --host 51.68.155.224:27017  --eval "rs.status().myState" # Should return 2 (SECONDARY) — if offline → disaster

Prerequisites

Before starting the restore:

New bms-2 replacement server provisioned (OVH Kimsufi — same specs or better)
New bms-3 replacement server provisioned (optional — can restore single-node first, then re-add)
SSH access to new servers as root
sops -d secrets/bms-servers.env.sops accessible (age key on dev machine)
Wasabi credentials from secrets/monitoring.env.sops (WASABI_ACCESS_KEY_BACKUPS, WASABI_SECRET_KEY_BACKUPS)
MongoDB admin password from secrets/bms-servers.env.sops (mongodb_rs0_admin_password)

Step 1 — Get credentials

On dev machine (Windows):

# Silently load into env — NEVER print values
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets/bms-servers.env.sops | Out-Null
# Extract individually:
$env:MONGO_PASS = (sops --decrypt --input-type dotenv --output-type dotenv secrets/bms-servers.env.sops | Select-String "^mongodb_rs0_admin_password=").ToString().Split("=",2)[1]
$env:WASABI_KEY = (sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops | Select-String "^WASABI_ACCESS_KEY_BACKUPS=").ToString().Split("=",2)[1]
$env:WASABI_SECRET = (sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops | Select-String "^WASABI_SECRET_KEY_BACKUPS=").ToString().Split("=",2)[1]

Step 2 — Find latest backup on Wasabi

# On bms-4 or any server with aws CLI + Wasabi credentials
AWS_ACCESS_KEY_ID=$WASABI_KEY AWS_SECRET_ACCESS_KEY=$WASABI_SECRET \
  aws s3 ls s3://p24-infra/mongodb/full/ \
  --endpoint-url https://s3.eu-central-2.wasabisys.com | sort | tail -5
# Note the latest: mongodb-full-YYYY-MM-DD.tar.gz

Step 3 — Install MongoDB 6.x on new bms-2

# On new bms-2 (Ubuntu 22.04 recommended)
curl -fsSL https://www.mongodb.org/static/pgp/server-6.0.asc | gpg --dearmor -o /usr/share/keyrings/mongodb-server-6.0.gpg
echo "deb [ arch=amd64,arm64 signed-by=/usr/share/keyrings/mongodb-server-6.0.gpg ] https://repo.mongodb.org/apt/ubuntu jammy/mongodb-org/6.0 multiverse" | tee /etc/apt/sources.list.d/mongodb-org-6.0.list
apt-get update && apt-get install -y mongodb-org
 
# Create data dir, set ownership
mkdir -p /var/lib/mongodb /var/log/mongodb
chown -R mongodb:mongodb /var/lib/mongodb /var/log/mongodb

Step 4 — Set up keyFile (inter-node auth)

# On new bms-2: generate keyFile
openssl rand -base64 756 > /etc/mongodb/keyFile
chmod 400 /etc/mongodb/keyFile
chown mongodb:mongodb /etc/mongodb/keyFile
# Copy same keyFile to bms-3 and bms-4 (must be identical on all rs0 members)
scp /etc/mongodb/keyFile root@51.68.155.224:/etc/mongodb/keyFile
scp /etc/mongodb/keyFile root@54.36.123.110:/etc/mongodb/keyFile

Step 5 — Restore from Wasabi backup (standalone, no auth)

# On new bms-2 — start mongod WITHOUT auth first (standalone, no replSet)
cat > /etc/mongod-standalone.conf << 'EOF'
storage:
  dbPath: /var/lib/mongodb
net:
  bindIp: 127.0.0.1
  port: 27017
processManagement:
  fork: true
  pidFilePath: /tmp/mongod-standalone.pid
systemLog:
  destination: file
  path: /var/log/mongodb/mongod-restore.log
EOF
 
mongod --config /etc/mongod-standalone.conf
# Wait for start:
until mongosh --quiet --eval 'db.adminCommand({ping:1}).ok' 2>/dev/null | grep -q 1; do sleep 2; done
 
# Download backup
cd /tmp
AWS_ACCESS_KEY_ID=$WASABI_KEY AWS_SECRET_ACCESS_KEY=$WASABI_SECRET \
  aws s3 cp s3://p24-infra/mongodb/full/mongodb-full-YYYY-MM-DD.tar.gz . \
  --endpoint-url https://s3.eu-central-2.wasabisys.com
tar -xzf mongodb-full-YYYY-MM-DD.tar.gz
 
# Restore (preserveUUID required for consistent rs0 identity)
mongorestore --host 127.0.0.1:27017 --drop --preserveUUID --oplogReplay ./dump/
 
# Verify document counts
mongosh --quiet --eval 'use w3_db; print("regRecords:", db.regRecords.countDocuments())'
mongosh --quiet --eval 'use w3_db; print("files:", db.files.countDocuments())'

Step 6 — Create admin user and enable auth

# On new bms-2 (standalone still running)
mongosh --quiet --eval "
  db.getSiblingDB('admin').createUser({
    user: 'admin',
    pwd: '$MONGO_PASS',
    roles: [{role:'root', db:'admin'}]
  });
"
 
# Stop standalone
mongod --shutdown --dbpath /var/lib/mongodb
sleep 5

Step 7 — Start mongod with rs0 config and keyFile

# On new bms-2
cat > /etc/mongod.conf << 'EOF'
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
net:
  bindIp: 0.0.0.0
  port: 27017
security:
  authorization: enabled
  keyFile: /etc/mongodb/keyFile
replication:
  replSetName: rs0
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
EOF
 
systemctl enable mongod
systemctl start mongod
sleep 10

Step 8 — Initiate rs0 as single-node PRIMARY

mongosh --host 127.0.0.1:27017 -u admin -p "$MONGO_PASS" --authenticationDatabase admin --eval "
  rs.initiate({
    _id: 'rs0',
    version: 1,
    members: [
      { _id: 0, host: '145.239.133.104:27017', priority: 10 }
    ]
  });
"
# Wait for PRIMARY
sleep 15
mongosh --host 145.239.133.104:27017 -u admin -p "$MONGO_PASS" --authenticationDatabase admin \
  --eval "rs.status().members.map(m => m.stateStr)"

Step 9 — Install MongoDB on new bms-3 and add as SECONDARY

# Repeat Step 3 on bms-3 (new server), then:
# Copy keyFile from bms-2:
scp root@145.239.133.104:/etc/mongodb/keyFile /etc/mongodb/keyFile
chmod 400 /etc/mongodb/keyFile && chown mongodb:mongodb /etc/mongodb/keyFile
 
# Use same mongod.conf (Step 7) on bms-3, then start:
systemctl enable mongod && systemctl start mongod
 
# Add bms-3 as SECONDARY from bms-2:
mongosh --host 145.239.133.104:27017 -u admin -p "$MONGO_PASS" --authenticationDatabase admin --eval "
  rs.add({ host: '51.68.155.224:27017', priority: 1 });
"
# Wait for SECONDARY sync (may take minutes for large datasets):
mongosh --host 51.68.155.224:27017 -u admin -p "$MONGO_PASS" --authenticationDatabase admin \
  --eval "rs.status().members.map(m => m.stateStr + ':' + m.host)"

Step 10 — Re-add bms-4 ARBITER

# On bms-4, stop current mongod (which is probably confused about rs0)
systemctl stop mongod
# Clear data dir (arbiter has no data)
rm -rf /var/lib/mongodb/*
# Update keyFile if changed in Step 4
# Start mongod (same conf as before, keyFile path must match)
systemctl start mongod
 
# Add arbiter from bms-2:
mongosh --host 145.239.133.104:27017 -u admin -p "$MONGO_PASS" --authenticationDatabase admin --eval "
  rs.addArb('54.36.123.110:27017');
"

Step 11 — Verify rs0 health

mongosh --host 145.239.133.104:27017 -u admin -p "$MONGO_PASS" --authenticationDatabase admin --eval "
  var s = rs.status();
  s.members.forEach(m => print(m.stateStr, m.host, 'lag:', m.optimeDate));
"
# Expected: PRIMARY (bms-2), SECONDARY (bms-3), ARBITER (bms-4)

Step 12 — Update MONGODB_URL in bms-1 env files

After rs0 is healthy, update all application env files on bms-1:

# On bms-1 — update backend-environment.env
sed -i "s|mongodb://admin:[^@]*@[^/]*/|mongodb://admin:$MONGO_PASS@145.239.133.104,51.68.155.224/|g" \
  /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env
sed -i "s|PMONGODB_URL=.*|PMONGODB_URL=mongodb://admin:$MONGO_PASS@145.239.133.104,51.68.155.224/w3_db?replicaSet=rs0\&authSource=admin|" \
  /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env
 
# Update s3-environment.env
sed -i "s|DB_URI=.*|DB_URI=mongodb://admin:$MONGO_PASS@145.239.133.104,51.68.155.224/w3_db?replicaSet=rs0\&authSource=admin|" \
  /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/s3-environment.env
 
# Force-recreate containers
cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
docker-compose up -d --force-recreate backend s3

Step 13 — Update SOPS with new IPs (if servers changed)

If new bms-2/bms-3 IPs are different from original:

Update secrets/bms-servers.env.sops if password changed
Update secrets/monitoring.env.sops Prometheus scrape targets
Update monitoring/prometheus/prometheus.yml scrape configs
Update Prometheus alerting rules with new IPs

UFW rules (if new servers)

# On new bms-2 — allow MongoDB from bms-3, bms-4, bms-1, vps-i1
ufw allow from 51.68.155.224 to any port 27017
ufw allow from 54.36.123.110 to any port 27017
ufw allow from 94.23.26.113  to any port 27017
ufw allow from 217.154.82.162 to any port 27017  # vps-i1 (Prometheus)
ufw deny 27017

docs/playbooks/bms1-dr-plan.md — Scenario A (bms-1 loss only, rs0 intact)
scripts/mongodb-restore-drill.sh — automated drill (runs quarterly on bms-4)
scripts/mongodb-backup.sh — backup script on bms-3 (Wasabi, weekly → daily after #2367)
docs/05-backups-dr/README.md — backup coverage overview
Issue #2148 — backup gap tracking
Issue #2367 — bms-3 SSH + daily cron fix

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="other",
    resource="mongodb-rs0",
    result="success",  # "success" | "failed" | "skipped"
    detail="MongoDB rs0 full restore executed from backup",
    env="bms-2",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'other', 'mongodb-rs0', 'success', 'MongoDB rs0 full restore executed from backup', 'bms-2')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

mongodb-rs0-full-restore

Playbook: MongoDB rs0 — Full Restore from Wasabi Backup

When to use this playbook

Prerequisites

Step 1 — Get credentials

Step 2 — Find latest backup on Wasabi

Step 3 — Install MongoDB 6.x on new bms-2

Step 4 — Set up keyFile (inter-node auth)

Step 5 — Restore from Wasabi backup (standalone, no auth)

Step 6 — Create admin user and enable auth

Step 7 — Start mongod with rs0 config and keyFile

Step 8 — Initiate rs0 as single-node PRIMARY

Step 9 — Install MongoDB on new bms-3 and add as SECONDARY

Step 10 — Re-add bms-4 ARBITER

Step 11 — Verify rs0 health

Step 12 — Update MONGODB_URL in bms-1 env files

Step 13 — Update SOPS with new IPs (if servers changed)

UFW rules (if new servers)

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

mongodb-rs0-full-restore

Playbook: MongoDB rs0 — Full Restore from Wasabi Backup

When to use this playbook

Prerequisites

Step 1 — Get credentials

Step 2 — Find latest backup on Wasabi

Step 3 — Install MongoDB 6.x on new bms-2

Step 4 — Set up keyFile (inter-node auth)

Step 5 — Restore from Wasabi backup (standalone, no auth)

Step 6 — Create admin user and enable auth

Step 7 — Start mongod with rs0 config and keyFile

Step 8 — Initiate rs0 as single-node PRIMARY

Step 9 — Install MongoDB on new bms-3 and add as SECONDARY

Step 10 — Re-add bms-4 ARBITER

Step 11 — Verify rs0 health

Step 12 — Update MONGODB_URL in bms-1 env files

Step 13 — Update SOPS with new IPs (if servers changed)

UFW rules (if new servers)

Related

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks