p24-infra Docs

p4-ovh-bms-2-ns3087638-operations

3 min read

p4-ovh-bms-2-ns3087638 — Operations Workbook

Label: p4-ovh-bms-2-ns3087638 Host: 145.239.133.104 Hostname: ns3087638 Provider: OVH / Kimsufi (ns3087638.ip-145-239-133.eu) Hardware: 8 vCPU · 32 GB RAM · 410 GB RAID1 (2× NVMe nvme0n1 + nvme1n1 → /dev/md3) OS: Ubuntu 24.04.3 LTS Role: MongoDB rs0 full voting member (currently PRIMARY) Inventoried: 2026-06-14

Server Role

MongoDB rs0 full voting member (priority 1, votes 1). Currently PRIMARY (verified 2026-06-15).

No Claude agent installed on this server. Claude agents run on bms-4 (AI-Dev-BMS4-1).

SSH Access

MethodCommand
Human (radieu)ssh ubuntu@145.239.133.104 (uses ~/.ssh/id_ed25519)
Claude agentssh claude-admin@145.239.133.104 (uses VPS_SSH_PRIVATE_KEY)
PuTTY saved sessionns3087638.ip-145-239-133.eu
Password fallbackubuntu + password in .env.local → bare_metal_server_2_root_password

Keys installed:

  • id_ed25519 (radieu) → /home/ubuntu/.ssh/authorized_keys — 2026-06-14
  • VPS_SSH_PRIVATE_KEY (claude-admin) → /home/claude-admin/.ssh/authorized_keys — 2026-06-14

MongoDB

SettingValue
Version7.0.25
Replica setrs0
RoleFull voting member — priority 1, votes 1, currently PRIMARY
Port27017 (bindIp 0.0.0.0)
Data dir/var/lib/mongodb
Log/var/log/mongodb/mongod.log
AuthkeyFile /etc/mongodb-keyfile + authorization: enabled
StatusActive since 2026-06-10

Replica Set Members (verified 2026-06-15)

MemberIPRolepriorityvotes
ns3087638 (bms-2)145.239.133.104:27017PRIMARY11
ns3129867 (bms-3)51.68.155.224:27017SECONDARY11
ns3101999 (bms-4)54.36.123.110:27017ARBITER01

Dead arbiter 51.83.132.99 confirmed removed.

Check replica set status

mongosh --quiet -u "$mongodb_rs0_admin_user" -p "$mongodb_rs0_admin_password" \
  --authenticationDatabase admin \
  --eval 'rs.status().members.forEach(m => print(m.name, m.stateStr, m.health))'

MongoDB admin credentials

Stored in .env.local (local workstation):

  • Variable: mongodb_rs0_admin_user (username)
  • Variable: mongodb_rs0_admin_password (password)

Disk Layout

/dev/nvme0n1  419.2G  (NVMe disk 1)
/dev/nvme1n1  419.2G  (NVMe disk 2)
/dev/md3      410G    mounted at /  — ~21.7 GB used by MongoDB (watch OOM at high load)

RAM: MongoDB ~21.7 GB resident. Monitor for OOM if workload increases.

Monitoring

prometheus-node-exporter 1.7.0 active (systemd), port :9100.

Prometheus scrape target: monitoring/prometheus/prometheus.yml node job — 145.239.133.104:9100.

MongoDB admin Database Cleanup (2026-06-17)

The legacy Pinbox24 admin application database was cleaned up on 2026-06-17 (issue #539).

What was done:

  • Verified Wasabi backup: s3://p24-infra/mongodb-backups/admin-freeze-2026-06-15/mongo-admin-freeze.tar.gz (3007 MB compressed) — confirmed present before drop
  • Verified no data operations from bms-1 (94.23.26.113) against admin collections (only MongoDB driver hello heartbeats, not application queries)
  • Dropped 76 Pinbox24 application collections from the admin database on the PRIMARY (bms-2)
  • Verified replication to bms-3 SECONDARY: only 4 system collections remain
  • Verified bms-1 Pinbox24 containers still running normally after cleanup

Collections dropped (76 total): regRecords (11.2M docs), process.instances (1.3M docs), files (201k docs), forms (24k docs), businessaddressbooks (56k docs), and 71 more — all Pinbox24 legacy application data.

System collections preserved (4):

  • system.users — MongoDB user accounts (CRITICAL)
  • system.keys — MongoDB internal keyFile keys (CRITICAL)
  • system.version — MongoDB metadata
  • system.views — MongoDB view definitions

Before cleanup: admin DB = 8439 MB disk
After cleanup: admin DB = 0 MB disk (disk space reclaimed by WiredTiger background compaction)

Rollback: Restore from s3://p24-infra/mongodb-backups/admin-freeze-2026-06-15/mongo-admin-freeze.tar.gz

Tasks

  • rs0 topology verified: bms-2 is PRIMARY (priority 1, votes 1) — 2026-06-15
  • node_exporter 1.7.0 active on :9100; added to monitoring/prometheus/prometheus.yml — 2026-06-15
  • dev_r_services rows updated with PRIMARY role + correct credential names — 2026-06-15
  • compliance_workbook = 'yes' and workbook_url set in dev_r_services — 2026-06-15
  • Legacy Pinbox24 admin DB application collections dropped (76 collections, ~8.4 GB) — 2026-06-17 (#539)

Provisioning Log

DateActionBy
2026-06-10MongoDB 7.0.25 installed, joined rs0 as voting memberClaude Code
2026-06-14Server inventoried, ops doc createdClaude Code
2026-06-15Verified as PRIMARY (priority 1, votes 1); doc corrected from “observer”Claude Code
2026-06-17Dropped 76 legacy Pinbox24 application collections from admin DB (#539)Claude Code

Graph View

Backlinks