p24-infra Docs

p4-ovh-bms-3-ns3129867-operations

4 min read

p4-ovh-bms-3-ns3129867 — Operations Workbook

Label: p4-ovh-bms-3-ns3129867
Host: 51.68.155.224
Hostname: ns3129867
Provider: OVH / Kimsufi (ns3129867.ip-51-68-155.eu)
Hardware: 8 vCPU · 32 GB RAM · 410 GB RAID1 (2× NVMe → /dev/md3)
OS: Ubuntu 22.04.5 LTS
Role: Pinbox24 staging/dev server + MongoDB replica set member (rs0)
Inventoried: 2026-06-14 · Last audit: 2026-06-23 (see playbook)

Server Role

Dual-purpose server:

  1. MongoDB — member of rs0 replica set (running since 2025-12-19; SECONDARY as of 2026-06-28, bms-2 is PRIMARY)
  2. Staging — hosts all active Pinbox24 staging versions (v31/v32/v41/v42) via Docker + nginx-proxy

⚠️ MongoDB is consuming 21.7 GB RAM — leaves ~12 GB for containers. Watch for OOM.

2026-06-23 audit — RabbitMQ 3.9.27 (EOL, 0 queues) and an orphaned GitLab Runner were decommissioned; the crash-looping v32-stage container was removed; fail2ban was installed (SSH jail). Disk dropped 44% → 22%. Full details in the bms-3 audit playbook.

SSH Access

MethodCommand
Human (radieu)ssh ubuntu@51.68.155.224 (uses ~/.ssh/id_ed25519)
Claude agentssh claude-admin@51.68.155.224 (uses VPS_SSH_PRIVATE_KEY)
Password fallbackubuntu + password in .env.local → bare_metal_server_3_root_password

Keys installed:

  • id_ed25519 (radieu) → /home/ubuntu/.ssh/authorized_keys — 2026-06-14
  • VPS_SSH_PRIVATE_KEY (claude-admin) → /home/claude-admin/.ssh/authorized_keys — to be set up

claude-admin setup (run once as ubuntu):

sudo useradd -m -s /bin/bash claude-admin
sudo mkdir -p /home/claude-admin/.ssh
echo "<VPS_SSH_PRIVATE_KEY public part>" | sudo tee /home/claude-admin/.ssh/authorized_keys
sudo chmod 700 /home/claude-admin/.ssh && sudo chmod 600 /home/claude-admin/.ssh/authorized_keys
sudo chown -R claude-admin:claude-admin /home/claude-admin/.ssh
echo "claude-admin ALL=(ALL) NOPASSWD: /usr/bin/docker, /bin/systemctl, /bin/mkdir, /bin/chown, /bin/cp, /usr/bin/tee" \
  | sudo tee /etc/sudoers.d/claude-admin

MongoDB

SettingValue
Version7.0.26
Replica setrs0
Port27017 (bindIp 0.0.0.0)
Data dir/var/lib/mongodb
Log/var/log/mongodb/mongod.log
AuthkeyFile /etc/mongodb-keyfile + authorization: enabled
StatusActive since 2025-12-19 (5+ months)
RAM usage~21.7 GB
rs0 roleSECONDARY — bms-2 (145.239.133.104) is PRIMARY (verified 2026-06-28)

regRecords search index operations (issue #1198)

The Pinbox24 “search records” feature on w3_db/w4_db regRecords used to issue 60–223 s $or regex scans (RESO EUROPA + Keller). Compound wildcard search indexes were created on the PRIMARY (bms-2) on 2026-06-24 and replicate here automatically; slow-query volume has since dropped from ~1,271/day to 1–10/day.

CollectionSearch index (present, verified 2026-06-28)
w3_db.regRecordsp24_search_compound_wildcard {officeId:1, regId:1, $**:1}
w4_db.regRecordsp24_search_recorddata_wildcard {officeId:1, regId:1, deleted:1, recordData.$**:1}

Running Docker Containers

ContainerImageStatus
v42-stageECR v42-stageUp 3 months
s3-v42-stageECR v4-s3Up 3 months
v41-stageECR v41-stageUp 6 months
s3-v32-stageECR old-s3Up — ⚠️ orphaned (peer v32-stage removed 2026-06-23, ~1% CPU; decommission TBD)
v31-stageECR v31-stage:latestUp 5 months
traccartraccar/traccar:latestUp 4 months
mt5mt5Up 5 months
portainer-pinbox24portainer/portainerUp — ⚠️ deprecated v1 CE on random high ports; upgrade to portainer-ce or disable (open decision)
nginx-proxyjwilder/nginx-proxyUp — ⚠️ deprecated image (open decision)
nginx-proxy-letsencryptjrcs/letsencrypt-nginx-proxy-companionUp 6 months

Registry: 563740926945.dkr.ecr.eu-central-1.amazonaws.com

ECR Login (required after 12h token expiry)

aws ecr get-login-password --region eu-central-1 \
  | docker login --username AWS --password-stdin \
    563740926945.dkr.ecr.eu-central-1.amazonaws.com

Disk Layout

/dev/nvme0n1  419.2G  (NVMe disk 1)
/dev/nvme1n1  419.2G  (NVMe disk 2)
/dev/md3      410G    mounted at /  — 82G used (22%, after 2026-06-23 audit cleanup)

✅ 22% disk used after the 2026-06-23 audit (was 44% — freed by RabbitMQ/erlang removal and stopping the v32-stage crash loop). Staging logs and old Docker images can still fill this fast — keep monitoring.

Disk cleanup

# Remove unused Docker images
docker system prune -f
 
# Check largest directories
du -sh /var/lib/mongodb /var/lib/docker /var/log

Security

fail2ban (SSH brute-force protection) — added 2026-06-23

SettingValue
Jailsshd (enabled)
maxretry3
bantime24h (86400)
Whitelistadmin workstation 95.91.246.219 (ignoreip)
# Status of the SSH jail (banned IPs, fail count)
sudo fail2ban-client status sshd
 
# Unban an IP
sudo fail2ban-client set sshd unbanip <IP>
 
# Config: /etc/fail2ban/jail.local  (edit then reload)
sudo systemctl reload fail2ban

ufw — port 27017 allowlist

MongoDB 27017 is restricted to the rs0 peers + monitoring + admin. See CLAUDE.md → bms-3 ufw rules. fail2ban guards SSH (22).

RabbitMQ / GitLab Runner — decommissioned 2026-06-23

RabbitMQ (ports 5672/15672/25672) and an orphaned GitLab Runner were removed during the 2026-06-23 audit. Do not re-add them. Reproduction steps and rationale: bms-3 audit playbook.

Monitoring

Not yet connected to Prometheus. node_exporter not installed.

To add Prometheus scrape:

  1. Install node_exporter on this server
  2. Add scrape target to monitoring/prometheus/prometheus.yml under the node job

Tasks

  • Install node_exporter and add to Prometheus
  • Monitor disk — 22% used after audit; set alert at 70%
  • Monitor RAM — MongoDB using 21.7GB; evaluate if it should be moved to dedicated node
  • Add ops doc URL to dev_r_services and set compliance_workbook = 'yes'
  • Open decision: stop orphaned s3-v32-stage container (peer removed 2026-06-23)
  • Open decision: upgrade deprecated portainer/portainer v1 → portainer-ce, or disable
  • Open decision: replace deprecated jwilder/nginx-proxy image

Graph View

Backlinks