queue-dispatcher-operations.md
Purpose
The queue dispatcher system manages RAM-aware job scheduling for Claude worker agents and GitHub Actions jobs on self-hosted runners. It replaces naive slot-count gating with RAM-sum math, giving the dispatcher accurate visibility into actual memory pressure.
Components:
| Component | Location | Role |
|---|---|---|
queue-dispatcher-loop.py | scripts/ on bms-4 + vps-i1 | Main dispatch loop — reads queue, spawns workers |
spawn-worker.sh | scripts/ on bms-4 + vps-i1 | Spawns a single Claude worker via systemd-run |
gha-queue-claim.sh | scripts/ on bms-4 + vps-i1 | Claims RAM slot for GHA job at workflow start |
gha-queue-release.sh | scripts/ on bms-4 + vps-i1 | Releases RAM slot at workflow end (always step) |
available_ram_gb(server_label) | Supabase RPC | Returns bookable RAM headroom (80% cap) |
reserve_subagent_slots(...) | Supabase RPC | Reserves subagent slots with advisory lock |
complete_subagent_slot(...) | Supabase RPC | Marks a subagent slot done/failed/cancelled |
claim_gha_slot(...) | Supabase RPC | Inserts a GHA slot row as status=running |
release_gha_slot(...) | Supabase RPC | Updates GHA slot row to status=done |
Job-type → prompt routing (#1423)
spawn-worker.sh selects the agent prompt from job_type, then the worker self-routes inside
worker-issue.md Step 0-ROUTE before any codebase work:
job_type | Prompt file | What the worker does |
|---|---|---|
dev-issue | worker-issue.md | Full implement-an-issue flow (claim → branch → PR) |
infra-alert | worker-issue.md | Same implementation flow, triggered by an infra alert |
infra-task | infra-task-request-worker.md | Infra ops prompt — SSH into servers, no PR |
review-pr | worker-issue.md | Step 0-ROUTE invokes /review-pr <PR_NUMBER> (Haiku), marks queue row done, stops — no claim/branch/PR |
review-plan | worker-issue.md | Step 0-ROUTE invokes /review-plan <ISSUE_NUMBER> (Haiku), marks queue row done, stops |
Review-job parameters travel through the queue row’s metadata JSONB:
queue-dispatcher-loop.py extracts pr_number / model and exports QUEUE_JOB_TYPE,
PR_NUMBER, and optional CLAUDE_MODEL; spawn-worker.sh re-exports them into the
systemd-run worker scope (and applies --model only when CLAUDE_MODEL is set — review
skills normally pin Haiku via their own frontmatter, so it is empty by default).
bms4 proxy (planned — ADR-005 §9, #6112)
A planned, not-yet-deployed centralization component on bms-4 that will sit alongside the
dispatcher: (1) a single static egress IP for worker→Anthropic traffic, (2) centralized
account/task routing that reuses queue-dispatcher-loop.py’s existing _account_order headroom
ranker (no new logic), and (3) OAuth session custody that injects authenticated sessions into
ephemeral workers (windows-dev Docker containers, #6110). Purpose 3 is a deliberately accepted repeat
of the openclaw-gateway credential-custody risk and is human-gated behind refresh-failure
monitoring + a tested circuit-breaker. Full design-of-record and build phases:
docs/bms4-proxy-operations.md. It does not replace the queue spine.
Deployment
bms-4 (54.36.123.110)
Primary dispatch host. Runs queue-dispatcher-loop.py as a cron or systemd service.
# Verify dispatcher is running
ps aux | grep queue-dispatcher-loop
# Restart after deploy
systemctl restart queue-dispatcher || pkill -f queue-dispatcher-loop.py
nohup python3 /opt/p24-infra/scripts/queue-dispatcher-loop.py &
# Check last 50 log lines
tail -50 /var/log/p24-infra-dispatcher.logvps-i1 (217.154.82.162)
Secondary/monitoring host. Scripts deployed here for Alertmanager + lightweight jobs only.
# Check server capacity config
grep -E 'bms-4|vps-i1' /opt/p24-infra/monitoring/.envGH_TOKEN on the dispatch host (#5621)
queue-dispatcher-loop.py gates scan_prs_for_review() (and every GH-API scan
path) on if not GH_TOKEN. The token is present in vps-i1’s dispatcher
EnvironmentFile (/opt/p24-infra/monitoring/.env) but not in bms-4’s
(/opt/p24-infra/bms-4/.env) — on bms-4 the repo-write GH_TOKEN lives only in
the isolated per-service file /opt/p24-infra/bms-4/n8n-bms4-gh.env (deployed by
secrets-sync.yml from secrets/n8n-bms4-gh.env.sops). When bms-4 holds the HA
lease, a missing dispatcher GH_TOKEN makes scan-based dispatch a silent no-op
(masked ~6-day fleet-wide outage, #5621).
queue-dispatcher.sh therefore sources GH_TOKEN from the already-deployed
per-host worker token file when the env lacks it, in priority order:
/opt/p24-infra/bms-4/n8n-bms4-gh.env, then /opt/p24-infra/vps-i1/.env
(override with GH_TOKEN_FALLBACK_FILES for tests). This reuses the existing
token — it never introduces a second one. The canonical durable fix (delivering
GH_TOKEN into the dispatcher EnvironmentFile itself via SOPS) is a
secret-manager follow-up tracked on #5621.
# Confirm the leader can resolve GH_TOKEN (name only — never print the value):
grep -q '^GH_TOKEN=' /opt/p24-infra/bms-4/n8n-bms4-gh.env && echo present || echo MISSING
# After a dispatch cycle, the log records where it was sourced from:
grep 'GH_TOKEN sourced' /var/log/p24-infra-dispatcher.log | tail -1Deployment sequence after PR merge
secrets-sync.ymldeploys updated.envfiles to bms-4 and vps-i1 from SOPSsecrets-sync.ymlcopiesgha-queue-claim.sh+gha-queue-release.shto/opt/p24-infra/scripts/and runschmod +x- Apply Supabase migration
038_ram_budget_dispatch.sqlviasupabase db pushor dashboard SQL editor - Restart queue-dispatcher on bms-4 (see above)
- Verify: check Mezmo for
dispatch_cyclelog lines within 3 minutes of restart
Weight Class RAM Table
| Weight | RAM Budget | Use case |
|---|---|---|
light | 3 GB | Simple fixes, docs, single-file changes |
heavy | 8 GB | Multi-file features, schema migrations, TypeScript |
playwright | 6 GB | Browser automation tests (separate routing label) |
orchestrator | 12 GB | Issues requiring parallel subagents (fan-out) |
subagent | 0 GB | Cloud subagent slot reservation — reserves no host RAM (migration 039) |
80% RAM cap: bms-4 has 32 GB total, 4 GB reserved = 28 GB usable. At 80%: 22.4 GB bookable.
| Scenario | RAM booked | Available |
|---|---|---|
| Empty queue | 0 GB | 22.4 GB |
| 3 light workers | 9 GB | 13.4 GB |
| 1 orchestrator | 12 GB | 10.4 GB |
| 1 orchestrator + GHA heavy | 20 GB | 2.4 GB |
| 1 orchestrator + 1 heavy + GHA heavy | 28 GB | -5.6 GB (gated) |
Exit Code Reference
| Code | Name | Meaning | Dispatcher action |
|---|---|---|---|
| 0 | OK | Worker spawned successfully | Mark row running, record PID |
| 1 | Bad args | Invalid arguments to spawn-worker.sh | Mark failed |
| 2 | OOM | systemd cgroup MemoryMax hit; worker was killed | Bump spawn_failures; escalate weight once (first failure), then hold weight + 5-min backoff; fail after MAX_SPAWN_FAILURES |
| 3 | RAM contention | /proc/meminfo check: actual RAM < required×1.2 at spawn time | Bump spawn_failures, requeue same weight with 5-min next_attempt_at; fail after MAX_SPAWN_FAILURES |
| 4 | Needs heavier | Worker planning phase determined it needs subagents but RAM is insufficient | Escalate weight: light→heavy→playwright→orchestrator→fail |
Escalation chain for exit 4:
light → heavy → playwright → orchestrator → fail (ceiling)
After orchestrator fails, the job is marked failed with error message Cannot find capacity for orchestrator task.
When the dispatcher marks a dev_r_worker_queue row failed, it also records a
failure_reason (exit 2 → spawn_oom / timeout, exit 3 → spawn_ram, exit 4 ceiling →
spawn_oom, other non-zero → implementation_error). See Failure Reason Taxonomy (#1680) below.
Spawn failures vs implementation retries (#1675)
Two separate counters on dev_r_worker_queue track two different failure classes — do not
conflate them:
| Counter | Incremented by | Drives | Cap |
|---|---|---|---|
spawn_failures | Dispatch/spawn failures (exit 2 = cgroup OOM, exit 3 = RAM contention) | RAM backoff + one bounded weight escalation | MAX_SPAWN_FAILURES (5) → failed + human-action |
retry_count | Implementation timeouts only (reset_stale_workers(), 2 h) | Weight escalation + terminal failed | max_retries (default 2) |
Before #1675 the dispatcher bumped retry_count and escalated weight on every spawn failure.
Under RAM pressure each failed spawn escalated to a heavier (more RAM-hungry) tier and burned a
retry — so a row could hit failed from spawn failures alone, with no implementation ever
attempted (observed: #1428, 9 spawn failures before the first successful spawn). The two-counter
split keeps RAM backoff from consuming the implementation-retry budget, and next_attempt_at
delays re-dispatch by SPAWN_BACKOFF_MIN (5 min) so a contended host gets breathing room instead
of being hammered every cycle.
spawn_failures is exposed to grafana_readonly (table grant) for a dedicated panel.
Pre-dispatch Dedup Guard (scripts/queue_dispatcher.py)
Before a dispatch path INSERTs a new dev_r_worker_queue row, it must confirm the
issue has not already shipped or isn’t already in flight. Without this guard, a
dispatcher crash → row reset → re-dispatch (see #1660) creates a duplicate row
for an already-merged issue; the new worker finds nothing to do, exhausts its
retries, and ends failed — a false red counter in Grafana for shipped work
(observed: issue #1383, row 82 done → duplicate row 126 cancelled, #1670).
scripts/queue_dispatcher.py is the canonical implementation. Skip dispatch when
either:
- the most recent
dev_r_worker_queuerow for the issue is in a blocking state —queued,claimed,running, ordone, or - a merged PR referencing the issue already exists on GitHub (issue timeline
cross-referenced→ PRmerged_atnon-null).
A most-recent row of failed / cancelled / oom_killed with no merged PR
still allows re-dispatch, so the genuine retry path is preserved.
CLI (usable from any dispatcher / shell):
python3 scripts/queue_dispatcher.py should-dispatch <issue_number> <repo>
# exit 0 -> dispatch (no duplicate found)
# exit 10 -> skip (duplicate / already shipped; reason printed)
# exit 2 -> error (network / config)
# env: SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, GH_TOKEN (merged-PR check skipped if absent)The issues-queue and process-issues-remote skills mirror this logic inline in
PowerShell before their REST INSERT. Tests: scripts/tests/test_queue_dispatcher.py.
Failure Reason Taxonomy (#1680)
dev_r_worker_queue.failure_reason (added in migration 042_worker_queue_failure_reason.sql)
classifies why a row reached failed / oom_killed, so Grafana can alert only on
failures that need a human. All four legacy failed scenarios used to look identical
(observed: rows 125/#1428 and 126/#1383 both failed despite their work shipping via PRs
#1405 / #1469).
failure_reason | Set where | Severity | In alert count? |
|---|---|---|---|
spawn_oom | dispatcher: exit 2 cgroup OOM after MAX_SPAWN_FAILURES; exit 4 weight ceiling | High — work never started; check server RAM, re-queue | Yes |
timeout | dispatcher: SSH spawn timed out (worker unreachable) after MAX_SPAWN_FAILURES | Medium — re-queue | Yes |
implementation_error | dispatcher: unexpected non-zero spawn exit (e.g. exit 1 bad args); worker Step 11.5 on a genuine ship failure | High — manual investigation | Yes |
spawn_ram | dispatcher: exit 3 RAM contention after MAX_SPAWN_FAILURES | Low — transient host pressure | No |
mutex_collision | worker Step 0: claim_task() returned NULL (per-issue lock held by a live sibling) | None — duplicate worker | No |
stale_duplicate | dispatcher pre-dispatch guard: GitHub issue already closed (work shipped / won’t-fix) | None — cancel | No |
The Grafana “Failed (actionable)” stat (worker-queue.json, panel id 4) counts only
failure_reason IN ('spawn_oom','timeout','implementation_error'). spawn_ram,
mutex_collision, and stale_duplicate rows still appear in the Recent Jobs table
(colour-coded grey/yellow) but never trip the red alert stat.
complete_worker_queue_item() takes an optional 5th p_failure_reason arg (NULL-safe;
older 2/3/4-arg callers are unchanged). The dispatcher writes failure_reason via direct
PostgREST PATCH on the failed paths. The legacy dev_r_infra_task_requests table has no
failure_reason column, so the dispatcher only sets it for dev_r_worker_queue rows.
Mezmo Log Events
All events logged to Mezmo with app=p24-queue, hostname=<server_short>, tags=p24-queue.
| Event | Level | Source | Meaning |
|---|---|---|---|
spawn_ok | INFO | spawn-worker.sh | Worker started successfully |
spawn_oom | ERROR | spawn-worker.sh | Exit 2: OOM, re-queued heavier |
spawn_ram_contention | WARN | spawn-worker.sh | Exit 3: /proc/meminfo check failed |
dispatch_cycle | INFO | queue-dispatcher-loop.py | Dispatcher heartbeat each loop |
dispatch_scheduled | INFO | queue-dispatcher-loop.py | Job dispatched to server |
dispatch_exit3 | WARN | queue-dispatcher-loop.py | RAM contention backoff applied |
dispatch_exit4 | WARN | queue-dispatcher-loop.py | Weight escalation applied |
dispatch_failed | ERROR | queue-dispatcher-loop.py | Job marked failed (max retries) |
Mezmo alert views:
| View | Query | Trigger |
|---|---|---|
p24-queue: RAM spike | app:p24-queue spawn_ram_contention | 3+ hits / 10 min → Discord |
p24-queue: exit-4 loop | app:p24-queue dispatch_exit4 | 3+ hits / 5 min → Discord |
p24-queue: dispatcher silent | app:p24-queue (absence) | 0 hits / 30 min → Discord critical |
Grafana Alert
# Alert: bms-4 worker RAM headroom below 4 GB for 5 minutes
p24_worker_ram_available_gb{server="bms-4"} < 4Gauge updated every 60s by queue-exporter on vps-i1 (:9200).
RPC Signatures
available_ram_gb(server_label TEXT) → NUMERIC
Returns bookable RAM headroom in GB (negative = overbooked).
SELECT available_ram_gb('bms-4'); -- e.g. 22.4 with empty queuemax_subagents_for_weight(server_label TEXT, weight TEXT) → INT
Returns how many 1 GB subagent slots fit within the 80% cap after reserving the worker’s own RAM.
SELECT max_subagents_for_weight('bms-4', 'orchestrator'); -- 10
SELECT max_subagents_for_weight('bms-4', 'heavy'); -- 16
SELECT max_subagents_for_weight('vps-i1', 'light'); -- 0reserve_subagent_slots(parent_session_id TEXT, weight TEXT, count INT, server_label TEXT, parent_issue INT DEFAULT NULL) → TABLE(slot_id BIGINT)
Reserves count subagent slots under advisory lock. Returns fewer rows if RAM is tight.
SELECT slot_id FROM reserve_subagent_slots('sess-abc', 'subagent', 3, 'bms-4', 1265);
-- Returns 0–3 slot_id rows depending on available RAMcomplete_subagent_slot(slot_id BIGINT, child_session_id TEXT, status TEXT) → VOID
Called by subagent worker at Step Final. status must be done, failed, or cancelled.
SELECT complete_subagent_slot(42, 'sess-xyz', 'done');
-- RAISES EXCEPTION if status is invalidclaim_gha_slot(run_id TEXT, server_label TEXT, weight TEXT) → BIGINT
Claims a RAM slot for a GitHub Actions job. Returns the queue row ID.
SELECT claim_gha_slot('12345678', 'bms-4', 'heavy'); -- returns row idrelease_gha_slot(run_id TEXT, server_label TEXT) → VOID
Releases the GHA slot. Must be called with if: always() in GHA workflow.
SELECT release_gha_slot('12345678', 'bms-4');GHA Workflow Integration
Add these two steps to any self-hosted GHA workflow:
jobs:
build:
runs-on: [self-hosted, bms-4]
steps:
- name: Claim queue slot
run: /opt/p24-infra/scripts/gha-queue-claim.sh "${{ github.run_id }}" "heavy"
# ... actual job steps ...
- name: Release queue slot
if: always() # MUST be always() -- releases slot even on failure/cancel
run: /opt/p24-infra/scripts/gha-queue-release.sh "${{ github.run_id }}"Weight selection:
| Job type | Weight |
|---|---|
| Lint, type-check only | light |
| npm install + full test suite | heavy |
| Docker build + push | heavy |
| Simple shell script | light |
Liveness re-dispatch guard (#1679)
Before claiming a queued row, the dispatcher builds a set of issue_numbers that
already have an alive worker and excludes them from claim_job() (PostgREST
issue_number=not.in.(...)). This prevents a duplicate worker being spawned when a
second queued row exists for an issue whose original worker is still running.
A running row is considered alive when (mirrors reset_stale_workers()):
heartbeat_atis within the last 3 minutes (HEARTBEAT_FRESH_MINUTES), ORheartbeat_atis stillNULLbutstarted_at/claimed_at/queued_atis within the last 3 minutes (startup grace — the worker hasn’t sent its first 60s heartbeat yet)
A worker stale >3 min is NOT alive: its duplicate becomes claimable again and
reset_stale_workers() re-queues the original row. The set is seeded once per cycle
and extended in-cycle as new workers are dispatched, so a duplicate row for an issue
started earlier in the same cycle is also skipped.
-- Issues currently protected from re-dispatch (alive workers)
SELECT issue_number, status, heartbeat_at, started_at
FROM dev_r_worker_queue
WHERE status = 'running' AND heartbeat_at > NOW() - INTERVAL '3 minutes';spawn-worker.sh PATCHes heartbeat_at every 60s while the worker lives.
pg_cron TTL Reaper
Single combined cron job reap-orphan-queue-slots runs every 30 minutes:
- Subagent/worker rows (
job_type != 'github-actions'): deleted after 3 hours ofstatus=running - GHA rows (
job_type = 'github-actions'): deleted after 8 hours (GitHub hard limit is 6h; extra 2h for cleanup grace)
-- Check scheduled jobs
SELECT jobname, schedule, command FROM cron.job WHERE jobname = 'reap-orphan-queue-slots';Troubleshooting
available_ram_gb returns unexpected value:
Check for ghost running rows in dev_r_worker_queue. These accumulate if complete_subagent_slot was not called or a GHA release_gha_slot step was skipped.
SELECT id, job_type, weight, server_node, status, started_at
FROM dev_r_worker_queue
WHERE status = 'running' AND server_node = 'bms-4'
ORDER BY started_at;Dispatcher not picking up jobs:
- Check
MEZMO_SERVICE_KEYis set in/opt/p24-infra/bms-4/.env - Verify
available_ram_gb('bms-4')is positive - Check emergency_max_workers:
SELECT emergency_max_workers FROM dev_r_server_capacity WHERE server_label = 'bms-4'
Exit-4 loop (same issue escalating repeatedly):
Check retry_count and weight in dev_r_worker_queue. If weight has reached orchestrator and still exit-4, the issue requires manual intervention — the worker could not reserve enough subagent slots even at maximum weight.