queue-dispatcher-operations.md
Purpose
The queue dispatcher system manages RAM-aware job scheduling for Claude worker agents and GitHub Actions jobs on self-hosted runners. It replaces naive slot-count gating with RAM-sum math, giving the dispatcher accurate visibility into actual memory pressure.
Components:
| Component | Location | Role |
|---|---|---|
queue-dispatcher-loop.py | scripts/ on bms-4 + vps-i1 | Main dispatch loop — reads queue, spawns workers |
spawn-worker.sh | scripts/ on bms-4 + vps-i1 | Spawns a single Claude worker via systemd-run |
gha-queue-claim.sh | scripts/ on bms-4 + vps-i1 | Claims RAM slot for GHA job at workflow start |
gha-queue-release.sh | scripts/ on bms-4 + vps-i1 | Releases RAM slot at workflow end (always step) |
available_ram_gb(server_label) | Supabase RPC | Returns bookable RAM headroom (80% cap) |
reserve_subagent_slots(...) | Supabase RPC | Reserves subagent slots with advisory lock |
complete_subagent_slot(...) | Supabase RPC | Marks a subagent slot done/failed/cancelled |
claim_gha_slot(...) | Supabase RPC | Inserts a GHA slot row as status=running |
release_gha_slot(...) | Supabase RPC | Updates GHA slot row to status=done |
Job-type → prompt routing (#1423)
spawn-worker.sh selects the agent prompt from job_type, then the worker self-routes inside
worker-issue.md Step 0-ROUTE before any codebase work:
job_type | Prompt file | What the worker does |
|---|---|---|
dev-issue | worker-issue.md | Full implement-an-issue flow (claim → branch → PR) |
infra-alert | worker-issue.md | Same implementation flow, triggered by an infra alert |
infra-task | infra-task-request-worker.md | Infra ops prompt — SSH into servers, no PR |
grafana-dashboard | worker-issue-scoped.md | Grafana dashboard JSON authoring via grafana-dashboard-builder role; triggered by the grafana-dashboard label on an issue (#7930) |
review-pr | worker-issue.md | Step 0-ROUTE invokes /review-pr <PR_NUMBER> (Haiku), marks queue row done, stops — no claim/branch/PR |
review-plan | worker-issue.md | Step 0-ROUTE invokes /review-plan <ISSUE_NUMBER> (Haiku), marks queue row done, stops |
Review-job parameters travel through the queue row’s metadata JSONB:
queue-dispatcher-loop.py extracts pr_number / model and exports QUEUE_JOB_TYPE,
PR_NUMBER, and optional CLAUDE_MODEL; spawn-worker.sh re-exports them into the
systemd-run worker scope (and applies --model only when CLAUDE_MODEL is set — review
skills normally pin Haiku via their own frontmatter, so it is empty by default).
bms4 proxy (planned — ADR-005 §9, #6112)
A planned, not-yet-deployed centralization component on bms-4 that will sit alongside the
dispatcher: (1) a single static egress IP for worker→Anthropic traffic, (2) centralized
account/task routing that reuses queue-dispatcher-loop.py’s existing _account_order headroom
ranker (no new logic), and (3) OAuth session custody that injects authenticated sessions into
ephemeral workers (windows-dev Docker containers, #6110). Purpose 3 is a deliberately accepted repeat
of the openclaw-gateway credential-custody risk and is human-gated behind refresh-failure
monitoring + a tested circuit-breaker. Full design-of-record and build phases:
docs/bms4-proxy-operations.md. It does not replace the queue spine.
Deployment
bms-4 (54.36.123.110)
Primary dispatch host. Runs queue-dispatcher-loop.py as a cron or systemd service.
# Verify dispatcher is running
ps aux | grep queue-dispatcher-loop
# Restart after deploy
systemctl restart queue-dispatcher || pkill -f queue-dispatcher-loop.py
nohup python3 /opt/p24-infra/scripts/queue-dispatcher-loop.py &
# Check last 50 log lines
tail -50 /var/log/p24-infra-dispatcher.logvps-i1 (217.154.82.162)
Secondary/monitoring host. Scripts deployed here for Alertmanager + lightweight jobs only.
# Check server capacity config
grep -E 'bms-4|vps-i1' /opt/p24-infra/monitoring/.envGH_TOKEN on the dispatch host (#5621)
queue-dispatcher-loop.py gates scan_prs_for_review() (and every GH-API scan
path) on if not GH_TOKEN. The token is present in vps-i1’s dispatcher
EnvironmentFile (/opt/p24-infra/monitoring/.env) but not in bms-4’s
(/opt/p24-infra/bms-4/.env) — on bms-4 the repo-write GH_TOKEN lives only in
the isolated per-service file /opt/p24-infra/bms-4/n8n-bms4-gh.env (deployed by
secrets-sync.yml from secrets/n8n-bms4-gh.env.sops). When bms-4 holds the HA
lease, a missing dispatcher GH_TOKEN makes scan-based dispatch a silent no-op
(masked ~6-day fleet-wide outage, #5621).
queue-dispatcher.sh therefore sources GH_TOKEN from the already-deployed
per-host worker token file when the env lacks it, in priority order:
/opt/p24-infra/bms-4/n8n-bms4-gh.env, then /opt/p24-infra/vps-i1/.env
(override with GH_TOKEN_FALLBACK_FILES for tests). This reuses the existing
token — it never introduces a second one. The canonical durable fix (delivering
GH_TOKEN into the dispatcher EnvironmentFile itself via SOPS) is a
secret-manager follow-up tracked on #5621.
# Confirm the leader can resolve GH_TOKEN (name only — never print the value):
grep -q '^GH_TOKEN=' /opt/p24-infra/bms-4/n8n-bms4-gh.env && echo present || echo MISSING
# After a dispatch cycle, the log records where it was sourced from:
grep 'GH_TOKEN sourced' /var/log/p24-infra-dispatcher.log | tail -1Deployment sequence after PR merge
secrets-sync.ymldeploys updated.envfiles to bms-4 and vps-i1 from SOPSsecrets-sync.ymlcopiesgha-queue-claim.sh+gha-queue-release.shto/opt/p24-infra/scripts/and runschmod +x- Apply Supabase migration
038_ram_budget_dispatch.sqlviasupabase db pushor dashboard SQL editor - Restart queue-dispatcher on bms-4 (see above)
- Verify: check Mezmo for
dispatch_cyclelog lines within 3 minutes of restart
Weight Class RAM Table
| Weight | RAM Budget | Use case |
|---|---|---|
light | 3 GB | Simple fixes, docs, single-file changes |
heavy | 8 GB | Multi-file features, schema migrations, TypeScript |
playwright | 6 GB | Browser automation tests (separate routing label) |
orchestrator | 12 GB | Issues requiring parallel subagents (fan-out) |
subagent | 0 GB | Cloud subagent slot reservation — reserves no host RAM (migration 039) |
80% RAM cap: bms-4 has 32 GB total, 4 GB reserved = 28 GB usable. At 80%: 22.4 GB bookable.
| Scenario | RAM booked | Available |
|---|---|---|
| Empty queue | 0 GB | 22.4 GB |
| 3 light workers | 9 GB | 13.4 GB |
| 1 orchestrator | 12 GB | 10.4 GB |
| 1 orchestrator + GHA heavy | 20 GB | 2.4 GB |
| 1 orchestrator + 1 heavy + GHA heavy | 28 GB | -5.6 GB (gated) |
Exit Code Reference
| Code | Name | Meaning | Dispatcher action |
|---|---|---|---|
| 0 | OK | Worker spawned successfully | Mark row running, record PID |
| 1 | Bad args | Invalid arguments to spawn-worker.sh | Mark failed |
| 2 | OOM | systemd cgroup MemoryMax hit; worker was killed | Bump spawn_failures; escalate weight once (first failure), then hold weight + 5-min backoff; fail after MAX_SPAWN_FAILURES |
| 3 | RAM contention | /proc/meminfo check: actual RAM < required×1.2 at spawn time | Bump spawn_failures, requeue same weight with 5-min next_attempt_at; fail after MAX_SPAWN_FAILURES |
| 4 | Needs heavier | Worker planning phase determined it needs subagents but RAM is insufficient | Escalate weight: light→heavy→playwright→orchestrator→fail |
| 5 | Subscription blocked | spawn-worker.sh subscription check (usage ≥99 %, or ≥95 % with reset >1 d) or a post-launch weekly-limit hit; returned once every account is blocked | Requeue with 1 h backoff, no retry/spawn_failures bump; prefer another valid host next time; Discord ping |
| 6 | Git clone/auth failure | Bad GH_TOKEN / rejected gh-app-token on the host | Bump spawn_failures, re-route to bms-4, backoff; fail after MAX_SPAWN_FAILURES (spawn_git_auth) |
| 7 | OAuth revoked (post-launch) | Worker’s claude died at launch with “OAuth access token has been revoked” on every account; also stamps auth_revoked_at per account (#7667) | Hard-fail immediately (spawn_oauth_expired) + human-action escalation — re-auth is a human browser step, retrying can’t self-heal |
| 8 | No healthy account (pre-spawn) | The pre-spawn health gate (#7667) found every account on the host dead-token / auth-revoked / depleted, so nothing was launched | Requeue with NO_HEALTHY_ACCOUNT_BACKOFF_MIN (15 min) backoff, no retry/spawn_failures bump; prefer another valid host; feeds fleet-stopped detection |
| 255 | SSH connectivity failure | SSH Permission-denied / connection-refused to the host | Bump spawn_failures, re-route to bms-4, backoff; fail after MAX_SPAWN_FAILURES (spawn_ssh_failure) |
Exit 7 vs exit 8 — post-launch vs pre-spawn (#7667). Both are OAuth/credential failures, but
exit 8 is the proactive case: the dispatcher checks each account’s credential health before
SSH-spawning and routes around (or declines to spawn against) a dead-token account, so it never
burns the ~3 s launch + the row’s retries the way the #7575 outage did. Exit 7 remains the
reactive backstop for a token that looks fine pre-spawn but fails at launch. An individual
account’s exit 7 now stamps auth_revoked_at so the pre-spawn gate skips it next cycle (turning a
repeat exit-7 into a cheap exit-8 reroute); a successful spawn clears both depleted and
auth_revoked_at.
Escalation chain for exit 4:
light → heavy → playwright → orchestrator → fail (ceiling)
After orchestrator fails, the job is marked failed with error message Cannot find capacity for orchestrator task.
When the dispatcher marks a dev_r_worker_queue row failed, it also records a
failure_reason (exit 2 → spawn_oom / timeout, exit 3 → spawn_ram, exit 4 ceiling →
spawn_oom, other non-zero → implementation_error). See Failure Reason Taxonomy (#1680) below.
Spawn failures vs implementation retries (#1675)
Two separate counters on dev_r_worker_queue track two different failure classes — do not
conflate them:
| Counter | Incremented by | Drives | Cap |
|---|---|---|---|
spawn_failures | Dispatch/spawn failures (exit 2 = cgroup OOM, exit 3 = RAM contention) | RAM backoff + one bounded weight escalation | MAX_SPAWN_FAILURES (5) → failed + human-action |
retry_count | Implementation timeouts only (reset_stale_workers(), 2 h) | Weight escalation + terminal failed | max_retries (default 2) |
Before #1675 the dispatcher bumped retry_count and escalated weight on every spawn failure.
Under RAM pressure each failed spawn escalated to a heavier (more RAM-hungry) tier and burned a
retry — so a row could hit failed from spawn failures alone, with no implementation ever
attempted (observed: #1428, 9 spawn failures before the first successful spawn). The two-counter
split keeps RAM backoff from consuming the implementation-retry budget, and next_attempt_at
delays re-dispatch by SPAWN_BACKOFF_MIN (5 min) so a contended host gets breathing room instead
of being hammered every cycle.
spawn_failures is exposed to grafana_readonly (table grant) for a dedicated panel.
Pre-spawn account-health gate + fleet-stopped alert (#7667)
Follow-up of the #7575 outage (2026-10-08): claude-runner-2 carried a non-renewable token
(no refreshToken in .credentials.json) while claude-runner was at ≥95 % usage. The dispatcher
spawned claude-runner-2 first regardless, the worker died in ~3 s, and the row’s retries were
spent in seconds — with no alert saying the fleet itself could not run jobs.
Pre-spawn gate (skip dead-token accounts)
Before spawn_worker() SSH-spawns an account it now runs a pure, fail-open health gate
(scripts/lib/account_health.py). An account is treated as blocked when it is positively known
to be:
- depleted — DB
claude_accounts[].depleted(a confirmed exit-5 rate-limit block), or - auth-revoked — DB
claude_accounts[].auth_revoked_atwithinAUTH_REVOKED_COOLDOWN_S(30 min), stamped when an account produces an exit-7 spawn, or - dead-token — credential material cannot authenticate now and cannot renew: not in ADR-007
token mode and no
refreshTokenand no still-valid access token.
The credential facts are read over SSH by _read_credential_health() as booleans only
(token_mode, creds_present, has_refresh_token, access_expired) — the token values never
leave the host — and cached per (host, account) for 5 min, same as the headroom cache. The remote
probe’s stdout is parsed by the pure helper _coerce_credential_health(), which schema-validates
the response: it returns the four booleans only when all four keys are present, otherwise {}.
This keeps the gate fail-open on a partial or malformed probe response too — a missing key coerces
to unknown (account kept), never to a falsy value that could wrongly mark a healthy account dead.
Fail-open is the core safety property: an account whose health is unknown (SSH read failed, no
ssh_host) stays a candidate. The gate only ever removes accounts with positive evidence of
being dead. Routing:
- ≥1 healthy account → spawn only the healthy account(s), in ranked order.
- 0 healthy accounts →
spawn_worker()returns exit 8 (NO_HEALTHY_ACCOUNT_EXIT); the row is requeued with a 15-min backoff and no retry burn, preferring another valid host.
Fleet-stopped alert
Once per cycle main() checks whether any health-gated host is job-capable
(_host_job_capable: ≥1 worker slot in the current window and ≥1 non-blocked account). If none
is, _alert_fleet_stopped() emits one deduped signal — a Discord embed and a bug +
fleet-stopped-labelled GH issue. Dedup is GH-issue-based (not in-process, because main() runs
once per cycle): while an open fleet-stopped issue exists, no new issue/Discord ping is sent; the
check fails closed on a GH-API error so a flaky API never spams duplicates. Close the issue once
the fleet can dispatch again — it re-files if the condition recurs.
HEALTH-GATE staleness review (#7667)
The issue asked whether the HEALTH-GATE staleness rule (HEALTHY_STATUSES = {"healthy", "degraded"}, missing server_status defaulting to "healthy") is too strict. Conclusion: keep
it as-is. Loosening it to admit unknown/down hosts would risk dispatching to a genuinely dead
host (a worse failure than requeuing). Intermittent hosts (windows-dev, dev-laptop, lap2) are
correctly gated out when their heartbeat is stale. Instead of weakening the gate, the
fleet-stopped alert names the stale-gated hosts in its body, so a human can judge whether a host
that is actually healthy is being wrongly excluded (check its heartbeat cron /
dev_r_server_capacity.server_status) — observability over a blunt policy change.
Tests: scripts/tests/test_account_health.py,
scripts/tests/test_dispatcher_loop_no_healthy_account.py,
scripts/tests/test_dispatcher_fleet_stopped.py.
Coordinates with #7662 (treat the Claude 5 h session limit as exit 5) — that work owns the 5 h/weekly-limit → exit-5 path; this change does not touch it.
Pre-dispatch Dedup Guard (scripts/queue_dispatcher.py)
Before a dispatch path INSERTs a new dev_r_worker_queue row, it must confirm the
issue has not already shipped or isn’t already in flight. Without this guard, a
dispatcher crash → row reset → re-dispatch (see #1660) creates a duplicate row
for an already-merged issue; the new worker finds nothing to do, exhausts its
retries, and ends failed — a false red counter in Grafana for shipped work
(observed: issue #1383, row 82 done → duplicate row 126 cancelled, #1670).
scripts/queue_dispatcher.py is the canonical implementation. Skip dispatch when
either:
- the most recent
dev_r_worker_queuerow for the issue is in a blocking state —queued,claimed,running, ordone, or - a merged PR referencing the issue already exists on GitHub (issue timeline
cross-referenced→ PRmerged_atnon-null).
A most-recent row of failed / cancelled / oom_killed with no merged PR
still allows re-dispatch, so the genuine retry path is preserved.
CLI (usable from any dispatcher / shell):
python3 scripts/queue_dispatcher.py should-dispatch <issue_number> <repo>
# exit 0 -> dispatch (no duplicate found)
# exit 10 -> skip (duplicate / already shipped; reason printed)
# exit 2 -> error (network / config)
# env: SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, GH_TOKEN (merged-PR check skipped if absent)The issues-queue and process-issues-remote skills mirror this logic inline in
PowerShell before their REST INSERT. Tests: scripts/tests/test_queue_dispatcher.py.
Failure Reason Taxonomy (#1680)
dev_r_worker_queue.failure_reason (added in migration 042_worker_queue_failure_reason.sql)
classifies why a row reached failed / oom_killed, so Grafana can alert only on
failures that need a human. All four legacy failed scenarios used to look identical
(observed: rows 125/#1428 and 126/#1383 both failed despite their work shipping via PRs
#1405 / #1469).
failure_reason | Set where | Severity | In alert count? |
|---|---|---|---|
spawn_oom | dispatcher: exit 2 cgroup OOM after MAX_SPAWN_FAILURES; exit 4 weight ceiling | High — work never started; check server RAM, re-queue | Yes |
timeout | dispatcher: SSH spawn timed out (worker unreachable) after MAX_SPAWN_FAILURES | Medium — re-queue | Yes |
implementation_error | dispatcher: unexpected non-zero spawn exit (e.g. exit 1 bad args); worker Step 11.5 on a genuine ship failure | High — manual investigation | Yes |
spawn_ram | dispatcher: exit 3 RAM contention after MAX_SPAWN_FAILURES | Low — transient host pressure | No |
mutex_collision | worker Step 0: claim_task() returned NULL (per-issue lock held by a live sibling) | None — duplicate worker | No |
stale_duplicate | dispatcher pre-dispatch guard: GitHub issue already closed (work shipped / won’t-fix) | None — cancel | No |
The Grafana “Failed (actionable)” stat (worker-queue.json, panel id 4) counts only
failure_reason IN ('spawn_oom','timeout','implementation_error'). spawn_ram,
mutex_collision, and stale_duplicate rows still appear in the Recent Jobs table
(colour-coded grey/yellow) but never trip the red alert stat.
complete_worker_queue_item() takes an optional 5th p_failure_reason arg (NULL-safe;
older 2/3/4-arg callers are unchanged). The dispatcher writes failure_reason via direct
PostgREST PATCH on the failed paths. The legacy dev_r_infra_task_requests table has no
failure_reason column, so the dispatcher only sets it for dev_r_worker_queue rows.
Mezmo Log Events
All events logged to Mezmo with app=p24-queue, hostname=<server_short>, tags=p24-queue.
| Event | Level | Source | Meaning |
|---|---|---|---|
spawn_ok | INFO | spawn-worker.sh | Worker started successfully |
spawn_oom | ERROR | spawn-worker.sh | Exit 2: OOM, re-queued heavier |
spawn_ram_contention | WARN | spawn-worker.sh | Exit 3: /proc/meminfo check failed |
dispatch_cycle | INFO | queue-dispatcher-loop.py | Dispatcher heartbeat each loop |
dispatch_scheduled | INFO | queue-dispatcher-loop.py | Job dispatched to server |
dispatch_exit3 | WARN | queue-dispatcher-loop.py | RAM contention backoff applied |
dispatch_exit4 | WARN | queue-dispatcher-loop.py | Weight escalation applied |
dispatch_failed | ERROR | queue-dispatcher-loop.py | Job marked failed (max retries) |
Mezmo alert views:
| View | Query | Trigger |
|---|---|---|
p24-queue: RAM spike | app:p24-queue spawn_ram_contention | 3+ hits / 10 min → Discord |
p24-queue: exit-4 loop | app:p24-queue dispatch_exit4 | 3+ hits / 5 min → Discord |
p24-queue: dispatcher silent | app:p24-queue (absence) | 0 hits / 30 min → Discord critical |
Grafana Alert
# Alert: bms-4 worker RAM headroom below 4 GB for 5 minutes
p24_worker_ram_available_gb{server="bms-4"} < 4Gauge updated every 60s by queue-exporter on vps-i1 (:9200).
RPC Signatures
available_ram_gb(server_label TEXT) → NUMERIC
Returns bookable RAM headroom in GB (negative = overbooked).
SELECT available_ram_gb('bms-4'); -- e.g. 22.4 with empty queuemax_subagents_for_weight(server_label TEXT, weight TEXT) → INT
Returns how many 1 GB subagent slots fit within the 80% cap after reserving the worker’s own RAM.
SELECT max_subagents_for_weight('bms-4', 'orchestrator'); -- 10
SELECT max_subagents_for_weight('bms-4', 'heavy'); -- 16
SELECT max_subagents_for_weight('vps-i1', 'light'); -- 0reserve_subagent_slots(parent_session_id TEXT, weight TEXT, count INT, server_label TEXT, parent_issue INT DEFAULT NULL) → TABLE(slot_id BIGINT)
Reserves count subagent slots under advisory lock. Returns fewer rows if RAM is tight.
SELECT slot_id FROM reserve_subagent_slots('sess-abc', 'subagent', 3, 'bms-4', 1265);
-- Returns 0–3 slot_id rows depending on available RAMcomplete_subagent_slot(slot_id BIGINT, child_session_id TEXT, status TEXT) → VOID
Called by subagent worker at Step Final. status must be done, failed, or cancelled.
SELECT complete_subagent_slot(42, 'sess-xyz', 'done');
-- RAISES EXCEPTION if status is invalidclaim_gha_slot(run_id TEXT, server_label TEXT, weight TEXT) → BIGINT
Claims a RAM slot for a GitHub Actions job. Returns the queue row ID.
SELECT claim_gha_slot('12345678', 'bms-4', 'heavy'); -- returns row idrelease_gha_slot(run_id TEXT, server_label TEXT) → VOID
Releases the GHA slot. Must be called with if: always() in GHA workflow.
SELECT release_gha_slot('12345678', 'bms-4');GHA Workflow Integration
Add these two steps to any self-hosted GHA workflow:
jobs:
build:
runs-on: [self-hosted, bms-4]
steps:
- name: Claim queue slot
run: /opt/p24-infra/scripts/gha-queue-claim.sh "${{ github.run_id }}" "heavy"
# ... actual job steps ...
- name: Release queue slot
if: always() # MUST be always() -- releases slot even on failure/cancel
run: /opt/p24-infra/scripts/gha-queue-release.sh "${{ github.run_id }}"Weight selection:
| Job type | Weight |
|---|---|
| Lint, type-check only | light |
| npm install + full test suite | heavy |
| Docker build + push | heavy |
| Simple shell script | light |
Liveness re-dispatch guard (#1679)
Before claiming a queued row, the dispatcher builds a set of issue_numbers that
already have an alive worker and excludes them from claim_job() (PostgREST
issue_number=not.in.(...)). This prevents a duplicate worker being spawned when a
second queued row exists for an issue whose original worker is still running.
A running row is considered alive when (mirrors reset_stale_workers()):
heartbeat_atis within the last 3 minutes (HEARTBEAT_FRESH_MINUTES), ORheartbeat_atis stillNULLbutstarted_at/claimed_at/queued_atis within the last 3 minutes (startup grace — the worker hasn’t sent its first 60s heartbeat yet)
A worker stale >3 min is NOT alive: its duplicate becomes claimable again and
reset_stale_workers() re-queues the original row. The set is seeded once per cycle
and extended in-cycle as new workers are dispatched, so a duplicate row for an issue
started earlier in the same cycle is also skipped.
-- Issues currently protected from re-dispatch (alive workers)
SELECT issue_number, status, heartbeat_at, started_at
FROM dev_r_worker_queue
WHERE status = 'running' AND heartbeat_at > NOW() - INTERVAL '3 minutes';spawn-worker.sh PATCHes heartbeat_at every 60s while the worker lives.
pg_cron TTL Reaper
Single combined cron job reap-orphan-queue-slots runs every 30 minutes:
- Subagent/worker rows (
job_type != 'github-actions'): deleted after 3 hours ofstatus=running - GHA rows (
job_type = 'github-actions'): deleted after 8 hours (GitHub hard limit is 6h; extra 2h for cleanup grace)
-- Check scheduled jobs
SELECT jobname, schedule, command FROM cron.job WHERE jobname = 'reap-orphan-queue-slots';Troubleshooting
available_ram_gb returns unexpected value:
Check for ghost running rows in dev_r_worker_queue. These accumulate if complete_subagent_slot was not called or a GHA release_gha_slot step was skipped.
SELECT id, job_type, weight, server_node, status, started_at
FROM dev_r_worker_queue
WHERE status = 'running' AND server_node = 'bms-4'
ORDER BY started_at;Dispatcher not picking up jobs:
- Check
MEZMO_SERVICE_KEYis set in/opt/p24-infra/bms-4/.env - Verify
available_ram_gb('bms-4')is positive - Check emergency_max_workers:
SELECT emergency_max_workers FROM dev_r_server_capacity WHERE server_label = 'bms-4'
Exit-4 loop (same issue escalating repeatedly):
Check retry_count and weight in dev_r_worker_queue. If weight has reached orchestrator and still exit-4, the issue requires manual intervention — the worker could not reserve enough subagent slots even at maximum weight.