p24-infra Docs

bms3-audit-2026-06-23

4 min read

Playbook: bms-3 Full Server Audit (2026-06-23)

Reference playbook for the full audit of bms-3 (p4-ovh-bms-3-ns3129867 · 51.68.155.224 · Ubuntu 22.04 · OVH Kimsufi) performed 2026-06-23. Use it to understand what was changed live on the server, and to reproduce each hardening/cleanup step on this or another bare-metal node.

All changes below were applied live on bms-3 during the audit session — there is no open PR to merge for the server state itself. This doc + the spec updates in CLAUDE.md and the bms-3 ops workbook are the documentation catch-up.

What triggers this

  • Periodic / on-demand full server audit of a bare-metal node (software inventory, active services, Docker containers, network, cron, security, MongoDB replication state).
  • Suspicion of orphaned/EOL services, runaway CPU, or unprotected SSH.

Audit scope (what was checked)

Installed packages, active systemd services, Docker containers, open ports, cron jobs, SSH/security posture, and MongoDB rs0 replication health.

Changes applied live — summary

#ChangeResult
1fail2ban installed — SSH jail active (maxretry=3, ban 24h, admin 95.91.246.219 whitelisted)SSH brute-force protection on
2GitLab Runner unregistered from gitlab.com (runner dev, id 50979544, Pinbox24 v3.2 project); service stopped + disabledOrphaned 60s polling stopped
3mongodb_exporter — fixed prometheus user permissions on admin DB (bms-3 and bms-2); replaced --collect-all with selective collectors~1M historical auth errors stopped
4v32-stage container — stopped + removed (PM2 crash-restart loop ~4 months, ~124% CPU, 1.2M restarts, no active deployments)Parasitic CPU → 0%
5RabbitMQ 3.9.27 — decommissioned (0 queues, 0 messages, EOL since Jan 2024); removed 37 erlang packagesDisk 43% → 22%; ports 5672/15672/25672 closed

Before / after

MetricBeforeAfter
Disk usage43% (167G/410G)22% (82G/410G)
Parasitic CPU~124% (v32-stage PM2 loop)0%
RabbitMQ open ports5672, 15672, 25672closed
mongodb_exporter auth errors~1M historical, continuousstopped
SSH brute-force protectionnonefail2ban active
GitLab runner (orphaned)polling gitlab.com every 60sremoved

How to reproduce each step

1. fail2ban — SSH jail

sudo apt-get update && sudo apt-get install -y fail2ban
sudo tee /etc/fail2ban/jail.local >/dev/null <<'EOF'
[DEFAULT]
ignoreip = 127.0.0.1/8 ::1 95.91.246.219
bantime  = 86400
maxretry = 3
 
[sshd]
enabled = true
EOF
sudo systemctl enable --now fail2ban
sudo systemctl reload fail2ban
 
# Verify / operate
sudo fail2ban-client status sshd            # banned IPs + fail count
sudo fail2ban-client set sshd unbanip <IP>  # release an IP

2. GitLab Runner — unregister + disable

gitlab-runner list                                   # find runner name/token
sudo gitlab-runner unregister --name dev             # or --url/--token
sudo systemctl stop gitlab-runner
sudo systemctl disable gitlab-runner

3. mongodb_exporter — fix prometheus user permissions

Auth errors flood the mongod log when the exporter’s prometheus user lacks read on the admin DB. Grant the monitoring roles and stop using --collect-all:

// mongosh, authenticated as an admin user
use admin
db.grantRolesToUser("prometheus", [
  { role: "clusterMonitor", db: "admin" },
  { role: "read",          db: "local" }
])
# Exporter flags: replace --collect-all with selective collectors
mongodb_exporter \
  --mongodb.uri="mongodb://prometheus:<pw>@localhost:27017/admin" \
  --collector.diagnosticdata \
  --collector.replicasetstatus \
  --collector.dbstats

Apply the same grant on bms-2 (and check vps-i1). See also mongodb-exporter-uri-special-chars.md for the related URI-encoding gotcha when the password contains special characters.

4. Remove a crash-looping container

docker ps -a --format '{{.Names}}\t{{.Status}}' | grep -i restart   # spot crash loops
docker logs --tail 50 v32-stage                                     # confirm no live traffic
docker stop v32-stage && docker rm v32-stage

⚠️ The S3 companion s3-v32-stage was left running (peer now gone, ~1% CPU). Stopping it is an open decision — see the ops workbook Tasks.

5. Decommission RabbitMQ (EOL, unused)

# Confirm it is truly idle FIRST
sudo rabbitmqctl list_queues   # expect 0 queues / 0 messages
sudo rabbitmqctl list_connections
 
sudo systemctl stop rabbitmq-server
sudo systemctl disable rabbitmq-server
sudo apt-get purge -y rabbitmq-server 'erlang*'
sudo apt-get autoremove -y
# Ports 5672/15672/25672 now closed; ufw rules referencing them can be dropped

Verification

df -h /                                   # disk should reflect cleanup
docker ps --format '{{.Names}}'           # v32-stage gone, no crash loops
sudo fail2ban-client status sshd          # jail active
ss -tlnp | grep -E '5672|15672|25672'     # RabbitMQ ports closed (no output)
mongosh --eval 'rs.status().myState'      # MongoDB rs0 still healthy

Documentation follow-up (this issue, #1194)

  • CLAUDE.md bms-3 spec updated — RabbitMQ/GitLab Runner removed, fail2ban added, disk 22%, container list trimmed
  • bms-3 ops workbook updated — Security section, container/disk changes, audit note
  • This playbook written
  • dev_r_services — fail2ban-bms3 row added; no RabbitMQ/GitLab Runner rows existed to remove

Open decisions (not actioned in the audit)

  • s3-v32-stage — orphaned S3 companion, ~1% CPU; stop?
  • portainer/portainer v1 CE — deprecated, random high ports; upgrade to portainer-ce or disable?
  • jwilder/nginx-proxy — deprecated image; replace?

Notes for the next session

  • bms-servers.env.sops holds LOGDNA_API_KEY (Mezmo) — needed if Pinbox24 staging containers become active again.
  • The audit was run by a Claude Code p24-infra session on 2026-06-23.

Graph View

Backlinks