p24-infra Docs

bms-3

3 min read

OVH Kimsufi bms-3 (ns3129867) — MongoDB rs0 SECONDARY + Pinbox24 Staging

Environment ID: bms-3 Type: production Status: active

Purpose

Dual-role server: MongoDB rs0 SECONDARY (full voting member) + Pinbox24 staging environment. The staging instances serve pre-production testing for Pinbox24 v31–v42.

Hardware

FieldValue
Labelp4-ovh-bms-3-ns3129867
Hostnamens3129867
ProviderOVH / Kimsufi dedicated
IP51.68.155.224
OSUbuntu 22.04.5 LTS
vCPU8
RAM32 GB
Disk410 GB NVMe RAID1 (/dev/md3) — 44% used

Running Containers (11 total)

ContainerPurpose
mongodMongoDB 7.0 rs0 SECONDARY
pinbox24-staging-v31 through pinbox24-staging-v42Pinbox24 staging instances
traccarGPS tracking (staging)
nginx-proxyReverse proxy
mt5MetaTrader 5 related service

MongoDB rs0 Role

FieldValue
Replica setrs0
RoleFull voting member / SECONDARY
Current statusSECONDARY (bms-2 is currently PRIMARY)

Note: bms-3 was PRIMARY from approximately 2025-12-19 until bms-2 took over.

Access

# As ubuntu (human developer)
ssh ubuntu@51.68.155.224 -i C:\Users\konar\.ssh\id_ed25519

Claude Worker Dispatch Status

bms-3 is a live 3rd worker dispatch node for the p24-infra queue system (meta-dispatcher → dev_r_server_capacity table, row server_label = 'bms-3') — provisioned per docs/playbooks/provision-bms-3-dispatch-node.md (#2147). The stack is fully DB-driven: the enabled column on that row is the single switch that gates whether dispatchers spawn Claude Code workers onto this host.

Worker dispatch DISABLED as of 2026-07-31. dev_r_server_capacity.enabled = false, mongo_guard_disabled = false for server_label = 'bms-3' (verified via a follow-up SELECT). This is a deliberate, reversible pause — architect decision — not a teardown. bms-4 and vps-i1 dispatch rows are unaffected.

Why: three open issues, all still unresolved at time of disable:

  • #4717 — bms-3 light jobs stuck in a spawn-fail loop; weight_ram_gb.light was set to 1 GB vs bms-4’s 3 GB, too low for real light jobs to spawn.
  • #4728 — infra-task to raise weight_ram_gb.light to 3 GB and requeue the stuck jobs (4026/4028) — not yet applied.
  • #2369 — bms-3’s root SSH password in secrets/bms-servers.env.sops is stale/rejected; no recovery path except an OVH KVM console session (human-action, unresolved).

The RAM Guard (#3397, closed) protects MongoDB from worker load once dispatch resumes, but the 2-week soak test / rollback drill documented in docs/playbooks/provision-bms-3-dispatch-node.md §Soak test was never completed — bms-3 was never meant to be a full peer of bms-4.

Condition to re-enable: #4717 and #4728 resolved (RAM weight fix applied + stuck jobs cleared/requeued) AND #2369 resolved (root SSH recovered via OVH console). Then flip dev_r_server_capacity.enabled = true for server_label = 'bms-3' per the Activation section of the provisioning playbook — do not skip the smoke test even though the host was previously live. See docs/priorities.md for the tracked item.

Known Constraints

  • MongoDB RAM pressure: MongoDB uses ~21.7 GB RAM. Watch for OOM events — if available memory drops below 4 GB, consider tuning wiredTigerCacheSizeGB.
  • Disk usage 44%: Monitor growth. Logs and old Docker images accumulate.

Operations

# Check MongoDB secondary lag
ssh ubuntu@51.68.155.224 "mongosh --eval 'rs.printSecondaryReplicationInfo()'"
 
# Check disk usage
ssh ubuntu@51.68.155.224 "df -h"
 
# View MongoDB memory usage
ssh ubuntu@51.68.155.224 "free -h"

MongoDB credentials: secrets/bms-servers.env.sops (SOPS+age)

  • docs/servers/p4-ovh-bms-3-ns3129867-operations.md — detailed operational runbook
  • docs/mongodb-rs0-operations.md — replica set management
  • docs/playbooks/provision-bms-3-dispatch-node.md — worker dispatch node provisioning, RAM Guard model, activation/rollback SQL

Graph View