p4-ovh-bms-2-ns3087638 — Operations Workbook

Label: p4-ovh-bms-2-ns3087638 Host: 145.239.133.104 Hostname: ns3087638 Provider: OVH / Kimsufi (ns3087638.ip-145-239-133.eu) Hardware: 8 vCPU · 32 GB RAM · 410 GB RAID1 (2× NVMe nvme0n1 + nvme1n1 → /dev/md3) OS: Ubuntu 24.04.3 LTS Role: MongoDB rs0 full voting member (currently PRIMARY) Inventoried: 2026-06-14

Server Role

MongoDB rs0 full voting member (priority 1, votes 1). Currently PRIMARY (verified 2026-06-15).

No Claude agent installed on this server. Claude agents run on bms-4 (AI-Dev-BMS4-1).

SSH Access

Method	Command
Human (radieu)	`ssh ubuntu@145.239.133.104` (uses `~/.ssh/id_ed25519`)
Claude agent	`ssh claude-admin@145.239.133.104` (uses `VPS_SSH_PRIVATE_KEY`)
PuTTY saved session	`ns3087638.ip-145-239-133.eu`
Password fallback	`ubuntu` + password in `.env.local` → `bare_metal_server_2_root_password`

Keys installed:

id_ed25519 (radieu) → /home/ubuntu/.ssh/authorized_keys — 2026-06-14
VPS_SSH_PRIVATE_KEY (claude-admin) → /home/claude-admin/.ssh/authorized_keys — 2026-06-14

MongoDB

Setting	Value
Version	7.0.25
Replica set	`rs0`
Role	Full voting member — priority 1, votes 1, currently PRIMARY
Port	`27017` (bindIp `0.0.0.0`)
Data dir	`/var/lib/mongodb`
Log	`/var/log/mongodb/mongod.log`
Auth	keyFile `/etc/mongodb-keyfile` + `authorization: enabled`
Status	Active since 2026-06-10

Replica Set Members (verified 2026-06-15)

Member	IP	Role	priority	votes
ns3087638 (bms-2)	`145.239.133.104:27017`	PRIMARY	1	1
ns3129867 (bms-3)	`51.68.155.224:27017`	SECONDARY	1	1
ns3101999 (bms-4)	`54.36.123.110:27017`	ARBITER	0	1

Dead arbiter 51.83.132.99 confirmed removed.

Check replica set status

mongosh --quiet -u "$mongodb_rs0_admin_user" -p "$mongodb_rs0_admin_password" \
  --authenticationDatabase admin \
  --eval 'rs.status().members.forEach(m => print(m.name, m.stateStr, m.health))'

MongoDB admin credentials

Stored in .env.local (local workstation):

Variable: mongodb_rs0_admin_user (username)
Variable: mongodb_rs0_admin_password (password)

Disk Layout

/dev/nvme0n1  419.2G  (NVMe disk 1)
/dev/nvme1n1  419.2G  (NVMe disk 2)
/dev/md3      410G    mounted at /  — ~21.7 GB used by MongoDB (watch OOM at high load)

RAM: MongoDB ~21.7 GB resident. Monitor for OOM if workload increases.

Monitoring

prom/node-exporter v1.11.1 (Docker, --restart=always), port :9100. Upgraded from bare-metal 1.7.0 systemd on 2026-06-28 (#1922).

Prometheus scrape target: monitoring/prometheus/prometheus.yml node job — 145.239.133.104:9100.

node-exporter Upgrade (1.7.0 → 1.11.1)

Issue: #1922

Current state

prometheus-node-exporter 1.7.0 installed bare-metal via apt/systemd — not tracked by this repo or scanned by image-scan.yml. The weekly Trivy matrix pins prom/node-exporter:v1.11.1; the version mismatch means bms-2 is never CVE-scanned.

Target state

prom/node-exporter:v1.11.1 running as a Docker container (managed), port :9100. Matches image-scan.yml matrix — covered by weekly Trivy scans going forward.

Pre-upgrade verification

# Run from local workstation — confirms current metrics are flowing
curl -s http://145.239.133.104:9100/metrics | head -5
 
# Confirm current version
curl -s http://145.239.133.104:9100/metrics | grep node_exporter_build_info

Upgrade runbook (human SSH required — bms-2 is rs0 PRIMARY)

ssh ubuntu@145.239.133.104
 
# 1. Confirm mongod is healthy before touching anything
mongosh --quiet --eval 'rs.isMaster().ismaster'
# Expected: true
 
# 2. Stop and disable the old systemd service
sudo systemctl stop node_exporter
sudo systemctl disable node_exporter
 
# 3. Detect install type and remove old binary
#    dpkg -l | grep node-exporter
#    If apt-managed:
sudo apt-get remove -y prometheus-node-exporter
#    If direct binary only (not apt):
#    sudo rm -f /usr/local/bin/node_exporter
 
# 4. Remove old systemd unit file if present
#    sudo rm -f /etc/systemd/system/node_exporter.service
sudo systemctl daemon-reload
 
# 5. Install v1.11.1 via Docker (matches image-scan.yml matrix)
docker run -d \
  --name node-exporter \
  --restart=always \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  prom/node-exporter:v1.11.1 \
  --path.rootfs=/host
 
# 6. Verify metrics are flowing on :9100
curl -s http://localhost:9100/metrics | grep node_exporter_build_info
# Expected: node_exporter_build_info{...,version="1.11.1",...} 1
 
# 7. Confirm mongod is still PRIMARY (should be — we never touched it)
mongosh --quiet --eval 'rs.isMaster().ismaster'
# Expected: true

Post-upgrade verification (from local workstation)

# Metrics flowing externally
curl -s http://145.239.133.104:9100/metrics | grep node_exporter_build_info
 
# Check Prometheus target is UP
# Open: https://prometheus.vps-i1.infra.zintegrowana.online/targets
# Look for: 145.239.133.104:9100 — should show "UP"

Rollback

If Docker node-exporter fails to start:

# Re-enable the old systemd service (if binary still present)
sudo systemctl enable node_exporter
sudo systemctl start node_exporter
# Or reinstall: sudo apt-get install -y prometheus-node-exporter

MongoDB `admin` Database Cleanup (2026-06-17)

The legacy Pinbox24 admin application database was cleaned up on 2026-06-17 (issue #539).

What was done:

Verified Wasabi backup: s3://p24-infra/mongodb-backups/admin-freeze-2026-06-15/mongo-admin-freeze.tar.gz (3007 MB compressed) — confirmed present before drop
Verified no data operations from bms-1 (94.23.26.113) against admin collections (only MongoDB driver hello heartbeats, not application queries)
Dropped 76 Pinbox24 application collections from the admin database on the PRIMARY (bms-2)
Verified replication to bms-3 SECONDARY: only 4 system collections remain
Verified bms-1 Pinbox24 containers still running normally after cleanup

Collections dropped (76 total): regRecords (11.2M docs), process.instances (1.3M docs), files (201k docs), forms (24k docs), businessaddressbooks (56k docs), and 71 more — all Pinbox24 legacy application data.

System collections preserved (4):

system.users — MongoDB user accounts (CRITICAL)
system.keys — MongoDB internal keyFile keys (CRITICAL)
system.version — MongoDB metadata
system.views — MongoDB view definitions

Before cleanup: admin DB = 8439 MB disk
After cleanup: admin DB = 0 MB disk (disk space reclaimed by WiredTiger background compaction)

Rollback: Restore from s3://p24-infra/mongodb-backups/admin-freeze-2026-06-15/mongo-admin-freeze.tar.gz

Tasks

rs0 topology verified: bms-2 is PRIMARY (priority 1, votes 1) — 2026-06-15
node_exporter 1.7.0 active on :9100; added to monitoring/prometheus/prometheus.yml — 2026-06-15
Upgrade node-exporter to prom/node-exporter:v1.11.1 (Docker) — done 2026-06-28 (#1922)
dev_r_services rows updated with PRIMARY role + correct credential names — 2026-06-15
compliance_workbook = 'yes' and workbook_url set in dev_r_services — 2026-06-15
Legacy Pinbox24 admin DB application collections dropped (76 collections, ~8.4 GB) — 2026-06-17 (#539)

Provisioning Log

Date	Action	By
2026-06-10	MongoDB 7.0.25 installed, joined `rs0` as voting member	Claude Code
2026-06-14	Server inventoried, ops doc created	Claude Code
2026-06-15	Verified as PRIMARY (priority 1, votes 1); doc corrected from “observer”	Claude Code
2026-06-17	Dropped 76 legacy Pinbox24 application collections from `admin` DB (#539)	Claude Code
2026-06-28	Added upgrade runbook for node-exporter 1.7.0 → 1.11.1 (#1922)	Claude Code

p24-infra Docs

Explorer

p4-ovh-bms-2-ns3087638-operations

p4-ovh-bms-2-ns3087638 — Operations Workbook

Server Role

SSH Access

MongoDB

Replica Set Members (verified 2026-06-15)

Check replica set status

MongoDB admin credentials

Disk Layout

Monitoring

node-exporter Upgrade (1.7.0 → 1.11.1)

Current state

Target state

Pre-upgrade verification

Upgrade runbook (human SSH required — bms-2 is rs0 PRIMARY)

Post-upgrade verification (from local workstation)

Rollback

MongoDB `admin` Database Cleanup (2026-06-17)

Tasks

Provisioning Log

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

p4-ovh-bms-2-ns3087638-operations

p4-ovh-bms-2-ns3087638 — Operations Workbook

Server Role

SSH Access

MongoDB

Replica Set Members (verified 2026-06-15)

Check replica set status

MongoDB admin credentials

Disk Layout

Monitoring

node-exporter Upgrade (1.7.0 → 1.11.1)

Current state

Target state

Pre-upgrade verification

Upgrade runbook (human SSH required — bms-2 is rs0 PRIMARY)

Post-upgrade verification (from local workstation)

Rollback

MongoDB admin Database Cleanup (2026-06-17)

Tasks

Provisioning Log

Graph View

Table of Contents

Backlinks

MongoDB `admin` Database Cleanup (2026-06-17)