p24-infra Docs

provision-bms-3-dispatch-node

12 min read

Playbook: Provision bms-3 as a 3rd Claude Agent Dispatch Node

Issue: #2147 · Strategy Phase 2 item [11] RAM Guard: #3397 — Mongo primary/secondary hard gate + GB bin-packing scheduler (this doc’s model, superseding the original static-count seed) Scope: Bring bms-3 (51.68.155.224) online as a third worker dispatch node, raising parallel capacity from 6-7 → up to 11-12 agents without new hardware.

⚠️ bms-3 is a production MongoDB rs0 voting member + Pinbox24 staging host. Every step below runs on a live database server. The MongoDB PRIMARY/SECONDARY must never be CPU/RAM starved. Caps start deliberately low and ramp up only after the soak test. This is a human-supervised server operation — annotate each mutating SSH/SCP with # PLAYBOOK: provision-bms-3-dispatch-node.md.

Status (2026-07-08): The two blockers that force-disabled bms-3 (20260702100000_disable_bms3_dispatch_node.sql) are resolved and re-verified live as of this date — SSH works (key-based, BatchMode=yes, no interaction) and MongoDB replication lag is 0 with bms-3 confirmed SECONDARY (see #2382, #2384, both closed with the verification evidence). The RAM Guard (#3397) below is what makes it safe to flip enabled=true again — it was not flipped automatically as part of that work; the final activation is still the human-supervised step in the “Activation” section further down.

Additional prerequisite resolved (#3427, 2026-07-09): spawn-worker.sh previously ignored per-server weight_ram_gb entirely — its case "$WEIGHT" block hardcoded light) MEM_GB=3 regardless of server, so the actual systemd cgroup limit and pre-spawn free-RAM check on bms-3 used 3 GB per light worker even though the RAM Guard’s own bin-packing ledger (get_ram_usage/job_ram in queue-dispatcher-loop.py) assumed 1 GB — a 3x ledger-vs-enforcement mismatch that could have let bms-3 be oversubscribed to real RAM well past the ~17.6 GB budget once enabled. Fixed by queue-dispatcher-loop.py’s spawn_worker() now resolving and passing the per-server value through as CLAUDE_WORKER_MEM_GB, honoured by spawn-worker.sh. Step 2 below (deploy scripts to bms-3) must deploy the fixed spawn-worker.sh + queue-dispatcher-loop.py — a copy dated before #3427 reintroduces this exact mismatch.

RAM Guard model (#3397) — read this before touching enabled

The original bring-up (#2147) used a static worker-count cap (max_workers_prime=1, fixed 4 GB budget). That model is superseded — bms-3 now has:

  1. A live Mongo-role hard gate. Every dispatcher cycle (~2 min), queue-dispatcher-loop.py’s bms3_mongo_guard_cycle() reads bms-3’s own MongoDB replica-set state via the existing mongodb-exporter-bms3 container (runs ON bms-4 itself, port 9217 — see bms-4/docker-compose.yml; no new Mongo credentials, no new reachability path). If bms-3 is not confirmed SECONDARY (i.e. it is PRIMARY or the read fails → UNKNOWN), the guard hard-flips dev_r_server_capacity.enabled = FALSE for bms-3 and skips it for the current cycle too (not just the next one) — no new workers are dispatched. Once bms-3 is confirmed SECONDARY again, the guard automatically re-enables dispatch — but only if the row is currently disabled because the guard itself disabled it (tracked via the mongo_guard_disabled column). A human-initiated disable (e.g. the ships-disabled pre-soak-test state, or a deliberate maintenance disable) is never silently overridden by the guard.
  2. GB bin-packing instead of a worker-count cap. weight_ram_gb for bms-3 is {"light":1,"heavy":8,...} — “small” (light) workers cost 1 GB each, no fixed count cap; max_workers_prime/max_workers_night are set high (30) purely so that column never becomes the binding constraint — live GB headroom is. The worker budget while SECONDARY is os_ram_gb (31) - reserved_ram_gb (9)) * 0.80 ≈ 17.6 GB (kept just under the 18 GB target because those two columns are INT, not NUMERIC). While PRIMARY the budget is effectively 0 — the Mongo-role gate above blocks all dispatch regardless of any RAM math.
  3. Heavy jobs are lowest priority, capped at 1 concurrent, never preempt. The dispatch loop runs a small-first, heavy-last two-pass order on bms-3 — the opposite of bms-4’s heavy-first policy (docs/playbooks/queue-dispatcher-priority.md §“bms-4 heavy-first”). Small (light) jobs bin-pack against the full budget first; at most one heavy job (BMS3_MAX_CONCURRENT_HEAVY = 1) dispatches afterward, and only into whatever headroom remains — nothing in the dispatcher ever kills a running job to make room, so a queued heavy job simply waits until small jobs finish and free enough GB naturally.
  4. A Prometheus guard-failure safety net. Bms3DispatchEnabledWhilePrimary (monitoring/prometheus/rules/servers.yml, group bms3_ram_guard) fires if bms-3 ever reports PRIMARY while dispatch capacity is still enabled — i.e. if the guard above failed to flip in time. Mirrors the VpsI1WorkersUnexpectedlyEnabled pattern in docs/playbooks/worker-capacity-management.md.

Migration: supabase/migrations/20260708130000_bms3_ram_guard.sql. Code: scripts/queue-dispatcher-loop.py (bms3_mongo_role, bms3_mongo_guard_cycle, _bms3_guard_disable, _bms3_guard_reenable, the BMS3_LABEL-gated dispatch_passes branch). Tests: scripts/tests/test_bms3_ram_guard.py.

How dispatch targeting actually works (read first)

The dispatcher is fully DB-driven. There is no static SERVERS config file. scripts/queue-dispatcher.sh builds DISPATCHER_SERVERS at runtime from:

SELECT … FROM dev_r_server_capacity WHERE enabled = true;

So “add bms-3 to the SERVERS list” = insert/enable the bms-3 row in dev_r_server_capacity.

Consequence — order matters: the moment the bms-3 row is enabled = true, both dispatcher leaders (vps-i1 + bms-4) immediately try to ssh root@51.68.155.224 and run spawn-worker.sh. If the host is not yet prepared, every spawn fails (exit 255) → spawn_failures + Discord error spam + backoff. Therefore enabled = true is the LAST step, after install + verification.

The migration supabase/migrations/20260630011055_bms3_dispatch_node_capacity.sql seeds the row with enabled = false and conservative caps. It does not activate anything.

Capacity row (seeded by #2147’s migration, updated by #3397’s RAM Guard migration)

ColumnValueWhy
server_labelbms-3PK
ssh_host51.68.155.224dispatcher SSH target
ssh_userrootdispatcher SSHes as root, then sus to the worker user (migration 037) — not claude-runner
enabledfalseinert until provisioned + soak started — not touched by the #3397 migration; flipping to true remains the human “Activation” step below
mongo_guard_disabledfalsenew in #3397 — true only while the Mongo-role guard itself is the reason enabled=false; lets the guard auto-re-enable without ever overriding a human disable
max_workers_prime / night30 / 30intentionally high (#3397) — no static worker-count cap; GB bin-packing is the real limit
max_weight_prime / nightheavy / heavysmall (light) jobs + at most 1 heavy; still no playwright/orchestrator on a DB host
emergency_max_workers1legacy hard cap column — not read by the current dispatch loop
os_ram_gb / reserved_ram_gb31 / 9confirmed live via free -g 2026-07-08 → (31-9)*0.80 ≈ 17.6 GB worker budget while SECONDARY only (0 while PRIMARY, enforced by the Mongo-role hard gate, not by these columns)
worker_ram_gb1small-worker OOM guard (was 3; #3397 changed “small” workers to 1 GB each)
weight_ram_gb{"light":1,"heavy":8,"subagent":0,"playwright":8,"orchestrator":12}per-server RAM-per-weight override (#3397) — light=1 GB is bms-3-specific, others are fleet defaults
claude_accountsNULLdispatcher falls back to ["claude-runner"]; set the real account list in step 1

The original issue’s aspirational target was “3-4 agents” under a static count cap. #3397 replaced that framing entirely: there is no longer a target agent count — the GB budget (~17.6 GB) and the live Mongo-role gate are what bound bms-3’s load. Ramp-up now means widening the RAM budget (raising reserved_ram_gb down, i.e. lowering it) after a soak test proves headroom, not raising a worker count.

Live provisioning steps (human action)

Run from a session that can SSH to bms-3. Read the secrets-safety rules first — never echo key values.

1. Create the worker user + Claude account

ssh root@51.68.155.224   # PLAYBOOK: provision-bms-3-dispatch-node.md
# Create claude-runner (same as bms-4) if absent
id claude-runner >/dev/null 2>&1 || useradd -m -s /bin/bash claude-runner
# Set CLAUDE_WORKER_ID per the naming convention (see adding-new-worker.md)
echo 'export CLAUDE_WORKER_ID=bms3-cw-1' >> /home/claude-runner/.bashrc

Install Claude Code for claude-runner and authenticate the subscription (OAuth). If bms-3 carries more than one account, update dev_r_server_capacity.claude_accounts for bms-3 to the JSONB list ([{"user":"claude-runner","depleted":false,...}]), mirroring the bms-4 seed in supabase/migrations/20260628102631_claude_accounts_depletion.sql.

2. Copy the dispatch scripts from bms-4

# From a host with access to both (or via the repo deploy path /opt/p24-infra):
scp /opt/p24-infra/scripts/spawn-worker.sh           root@51.68.155.224:/opt/p24-infra/scripts/   # PLAYBOOK: provision-bms-3-dispatch-node.md
scp /opt/p24-infra/scripts/queue-dispatcher-loop.py  root@51.68.155.224:/opt/p24-infra/scripts/   # PLAYBOOK: provision-bms-3-dispatch-node.md
scp /opt/p24-infra/scripts/queue-dispatcher.sh       root@51.68.155.224:/opt/p24-infra/scripts/   # PLAYBOOK: provision-bms-3-dispatch-node.md

Preferred: pull via the existing /opt/p24-infra git deploy on bms-3 rather than scp, so the scripts stay in sync with main.

3. Provide the dispatcher’s env + SSH key

queue-dispatcher.sh loads secrets from an EnvironmentFile and reads the host label from /opt/p24-infra/.server-label. The dispatcher SSHes to workers with /root/.ssh/p24-dispatcher.

ssh root@51.68.155.224   # PLAYBOOK: provision-bms-3-dispatch-node.md
echo 'bms-3' > /opt/p24-infra/.server-label
# Ensure /opt/p24-infra/bms-4/.env equivalent exists for bms-3 (SUPABASE_SERVICE_ROLE_KEY,
#   P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL, GH_TOKEN). Source from SOPS — never paste values:
#   sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops | grep '^KEY=' …
# Install the dispatcher SSH key /root/.ssh/p24-dispatcher (same key bms-4 uses) and ensure
#   bms-3's own sshd authorizes it for root (self-SSH for local spawn).

bms-3 only needs to act as a worker target; it does not need to be a dispatcher leader. The vps-i1/bms-4 leaders already dispatch to it once enabled. Adding the */2 cron on bms-3 (step 4) is only required if you want bms-3 to also compete for the dispatcher lease (HA). For the minimum bring-up, skip the cron and let the existing leaders spawn onto bms-3.

4. (Optional, HA) Add the dispatcher cron/timer on bms-3

Only if bms-3 should also be a dispatcher leader (it will claim the lease via claim_dispatcher_lease() and skip when another leader holds it):

*/2 * * * * /opt/p24-infra/scripts/queue-dispatcher.sh

Or replicate the bms-4 systemd timer (queue-dispatcher.timer + .service).

5. Smoke test BEFORE enabling

With the row still enabled = false, dry-run a spawn as a leader would:

ssh root@51.68.155.224 "su -s /bin/bash claude-runner -c 'cd /opt/p24-infra && ls scripts/spawn-worker.sh && echo OK'"   # PLAYBOOK: provision-bms-3-dispatch-node.md

Confirm: claude-runner exists, scripts present + executable, Claude Code authenticated, dispatcher SSH key works (ssh -i /root/.ssh/p24-dispatcher root@51.68.155.224 true from a leader returns 0).

Activation (flip the switch)

Only after step 5 passes. Enable the row so the leaders start dispatching to bms-3:

UPDATE public.dev_r_server_capacity SET enabled = true WHERE server_label = 'bms-3';
UPDATE public.dev_r_services SET status = 'active' WHERE service_name = 'queue-dispatcher@bms-3';

Within ~2 min the next dispatch cycle will place a light worker on bms-3. Watch the dispatcher log:

ssh root@54.36.123.110 "tail -n 50 /var/log/p24-infra-dispatcher.log"   # PLAYBOOK: provision-bms-3-dispatch-node.md

Soak test (2 weeks)

Goal: confirm the added load never starves MongoDB, AND that the RAM Guard (#3397) actually protects it under a real or simulated failover. Monitor for 14 days:

  • MongoDB PRIMARY/SECONDARY CPU (bms-2 PRIMARY, bms-3 SECONDARY) — Grafana node_exporter panels. Alert if bms-3 sustained CPU > 80% or MongoDB replication lag rises.
  • bms-3 RAM — workers must stay within the ~17.6 GB budget; watch for OOM kills (dmesg -T | grep -i oom).
  • Pinbox24 staging responsiveness on bms-3.
  • Worker success rate on bms-3 (dev_r_worker_queue rows with server_node='bms-3').
  • Guard flip-to-disable within one cycle. Simulate (or wait for a real) rs0 election that promotes bms-3 to PRIMARY. Confirm:
    1. dev_r_server_capacity.enabled flips to false and mongo_guard_disabled flips to true for bms-3 within one dispatcher cycle (~2 min) of the election.
    2. No new dev_r_worker_queue rows get server_node='bms-3' after that point.
    3. A Discord warning is posted (bms3_mongo_guard_cycle → _bms3_guard_disable).
    4. If the flip does not happen, Bms3DispatchEnabledWhilePrimary (Prometheus) should fire within ~2 minutes as the safety net — confirm it does, as part of the drill.
  • Guard flip-to-re-enable once SECONDARY returns. After the simulated election reverses (bms-3 back to SECONDARY), confirm enabled auto-flips back to true (only because mongo_guard_disabled was true) and dispatch resumes within one cycle, without any human step.
  • Heavy-job policy. Queue a mix of light + heavy jobs; confirm small (light) jobs dispatch first and bin-pack against the live GB budget, at most 1 heavy job runs concurrently, and a queued heavy job never causes a running light job to be killed.

If clean for 2 weeks, ramp up cautiously (one step at a time, re-soak each) by lowering reserved_ram_gb (widening the GB budget) — never by raising max_workers_prime/night, which are deliberately uncapped already (#3397) and are not the lever that controls load. Confirm real headroom exists (Mongo CPU/RAM, Pinbox24 staging responsiveness) before each step.

Rollback

If CPU/RAM contention or MongoDB impact is detected at any point:

-- Immediate: stop new dispatch to bms-3 (running workers finish naturally)
UPDATE public.dev_r_server_capacity SET enabled = false, mongo_guard_disabled = false WHERE server_label = 'bms-3';

Explicitly set mongo_guard_disabled = false alongside enabled = false for a manual/human rollback. Otherwise, if the row happened to already be mongo_guard_disabled = true from a prior guard-driven disable, the Mongo-role guard could auto-re-enable it the next time bms-3 reports SECONDARY — overriding this deliberate rollback. Setting the marker to false makes this disable unambiguously human-initiated.

Then (optional, full removal):

DELETE FROM public.dev_r_server_capacity WHERE server_label = 'bms-3';
UPDATE public.dev_r_services SET status = 'retired' WHERE service_name = 'queue-dispatcher@bms-3';

On the host: remove the */2 dispatcher cron/timer if added in step 4. No data is lost — bms-3 keeps running MongoDB + staging exactly as before.

Checklist

  • Migration 20260630011055_bms3_dispatch_node_capacity.sql applied (row present, enabled=false)
  • Migration 20260708130000_bms3_ram_guard.sql applied (RAM Guard model, mongo_guard_disabled column, ~17.6 GB budget, small=1 GB/heavy=8 GB weight_ram_gb, uncapped max_workers_prime/night) — #3397
  • SSH to bms-3 confirmed live (#2382, resolved and closed)
  • MongoDB replication lag confirmed 0, bms-3 confirmed SECONDARY (#2384, resolved and closed)
  • spawn-worker.sh honours per-server weight_ram_gb via CLAUDE_WORKER_MEM_GB instead of a hardcoded global table — merged (#3427); the deployed copy on bms-3 (step 2) must postdate this
  • claude-runner user + CLAUDE_WORKER_ID=bms3-cw-1 set; Claude Code authenticated
  • spawn-worker.sh + queue-dispatcher-loop.py deployed to /opt/p24-infra/scripts (postdating #3427)
  • /opt/p24-infra/.server-label = bms-3; dispatcher env + SSH key installed
  • Smoke test passes (scripts present, SSH key works) while still disabled
  • enabled = true flipped (human-supervised — NOT part of #3397); first worker dispatched + completed on bms-3
  • 2-week soak: MongoDB CPU/lag, bms-3 RAM, staging, worker success rate all nominal
  • Soak also confirms: guard disables within 1 cycle of a PRIMARY reading, re-enables within 1 cycle of SECONDARY returning, Bms3DispatchEnabledWhilePrimary alert fires if the guard ever fails to flip, and heavy jobs never preempt running small jobs
  • Rollback tested (set enabled = false, mongo_guard_disabled = false, confirm no new spawns)

Written 2026-06-30 — Implements #2147. Related: adding-new-worker.md, worker-queue-operations.md, queue-dispatcher-operations.md.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="deploy",
    resource="bms-3-dispatch-node",
    result="success",  # "success" | "failed" | "skipped"
    detail="bms-3 dispatch node provisioned — claude-runner installed and registered",
    env="bms-3",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'deploy', 'bms-3-dispatch-node', 'success', 'bms-3 dispatch node provisioned — claude-runner installed and registered', 'bms-3')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks