p24-infra Docs

queue-dispatcher-ram-gate-exhaustion

5 min read

Playbook: Queue Dispatcher RAM-Gate Exhaustion

Diagnose and clear the failure mode where the worker-queue dispatcher stops spawning workers (dispatched 0 jobs) even though the queue is full, because the RAM budget has drained to zero from phantom stale rows.

Target: < 5 min to resolve. This is the runbook for the exact incident fixed in PR #1340 + migration 037_fix_reset_stale_workers_server_label.sql.

What triggers this

The 2-min dispatcher (scripts/queue-dispatcher.sh → queue-dispatcher-loop.py, systemd timer on vps-i1 primary / bms-4 backup) gates every spawn on a RAM budget:

avail_ram = available_ram_gb(server)        # total_ram − Σ weight_ram_gb of claimed/running rows
if avail_ram < weight_ram_gb[job_weight]:   # block — "RAM gate blocked"

available_ram_gb() subtracts the RAM weight of every claimed/running row from the server budget. If those rows are never cleaned up, the budget only ever goes down. reset_stale_workers() is the function that is supposed to release rows older than 2 hours back to queued (or failed) — so a broken reset_stale_workers() lets stale rows accumulate forever until the budget hits 0 and all dispatches are blocked.

Meanwhile the hourly triage keeps inserting fresh queued rows, so the queue grows while nothing is dispatched.

Symptoms:

  • Dispatcher log: Server <name>: dispatched 0 jobs this cycle despite a non-empty queue
  • Dispatcher log: repeated RAM gate blocked: avail=0.0GB needed=3GB ...
  • Discord #infra-scripts-errors: recurring reset_stale_workers() RPC failed embeds from queue-dispatcher error on <server>
  • New ai-dev-queued issues sit untouched; SLA watchdog starts flagging them

How to confirm

Run against Supabase (mcp__claude_ai_Supabase__execute_sql, project mwkqmgadqnkkihjdeqsi, or psql/REST with the service-role key).

-- 1. RAM budget drained? 0 or negative on either server confirms the gate is stuck.
SELECT available_ram_gb('bms-4') AS bms4, available_ram_gb('vps-i1') AS vpsi1;
 
-- 2. Phantom rows? > 0 stale claimed/running rows older than 2h is the smoking gun.
SELECT count(*) AS stale_rows
FROM dev_r_worker_queue
WHERE status IN ('claimed','running')
  AND COALESCE(started_at, claimed_at, queued_at) < NOW() - INTERVAL '2 hours';
 
-- 3. Queue is actually backed up (work is waiting)
SELECT status, count(*) FROM dev_r_worker_queue GROUP BY status ORDER BY status;
 
-- 4. Inspect the stale rows
SELECT id, github_issue_number, job_type, weight, status, server_node,
       COALESCE(started_at, claimed_at, queued_at) AS age_ref, retry_count, max_retries
FROM dev_r_worker_queue
WHERE status IN ('claimed','running')
  AND COALESCE(started_at, claimed_at, queued_at) < NOW() - INTERVAL '2 hours'
ORDER BY age_ref;

Confirm the RPC itself is the cause (it should error, not silently no-op):

SELECT reset_stale_workers();   -- if it raises "column ... does not exist", the function is broken

On the leader server, confirm the dispatcher is the one logging zeros:

ssh root@<leader> 'tail -40 /var/log/p24-infra-dispatcher.log'
# look for: "RAM gate blocked: avail=0.0GB" and "dispatched 0 jobs this cycle"

Step-by-step fix

1. Try the function first

SELECT reset_stale_workers();
  • Returns void without error → the function is healthy; jump to step 3 (verify). The phantom rows were probably from a one-off crash, not a code bug.
  • Raises an error (e.g. column "server_label" does not exist) → the function body is broken. Go to step 2 to repair it, then re-run cleanup.

2. Repair the function + cleanup phantom rows

The known root cause (migration 036) was reset_stale_workers() referencing the non-existent column server_label on dev_r_worker_queue (the real column is server_node). The canonical fix is migration monitoring/supabase/migrations/037_fix_reset_stale_workers_server_label.sql — apply it if not already applied:

SELECT name FROM supabase_migrations.schema_migrations
WHERE name LIKE '%reset_stale_workers%';   -- empty → not applied yet

If not applied, run migration 037 via mcp__claude_ai_Supabase__apply_migration (or paste its body into execute_sql). It does two things: recreates reset_stale_workers() with the correct server_node column, and runs a one-time cleanup of accumulated phantom rows.

If you cannot apply the migration immediately, do the manual cleanup directly (this is exactly what migration 037 part 2 does). It releases retryable rows and fails out exhausted ones, leaving slot-tracking rows (subagent, github-actions) alone:

-- Retryable: send back to the queue
UPDATE dev_r_worker_queue
SET status='queued', claimed_at=NULL, started_at=NULL, server_node=NULL,
    error_message='manual reset — RAM-gate exhaustion playbook'
WHERE status IN ('claimed','running')
  AND COALESCE(started_at, claimed_at, queued_at) < NOW() - INTERVAL '2 hours'
  AND retry_count < max_retries
  AND job_type NOT IN ('subagent','github-actions');
 
-- Exhausted: mark failed (terminal)
UPDATE dev_r_worker_queue
SET status='failed',
    error_message='max retries exceeded — RAM-gate exhaustion playbook'
WHERE status IN ('claimed','running')
  AND COALESCE(started_at, claimed_at, queued_at) < NOW() - INTERVAL '2 hours'
  AND retry_count >= max_retries
  AND job_type NOT IN ('subagent','github-actions');

3. Verify RAM recovers

SELECT available_ram_gb('bms-4') AS bms4, available_ram_gb('vps-i1') AS vpsi1;
-- expect positive values: bms-4 ~22 GB budget, vps-i1 lower (8 GB host)

4. Watch the next dispatcher cycle (≤ 2 min)

ssh root@<leader> 'tail -f /var/log/p24-infra-dispatcher.log'
# expect: "Server <name>: dispatched N jobs this cycle"  (N > 0)

If it’s still blocked after one cycle, re-check available_ram_gb() — a worker that is genuinely running and consuming RAM will (correctly) hold budget; only stale rows (> 2h) are phantom.

Prevention

  • Root cause fixed: migration 037 corrected the server_label → server_node column bug, so reset_stale_workers() now succeeds on every dispatcher cycle and phantom rows can no longer accumulate.
  • Monitor Discord: the dispatcher posts reset_stale_workers() RPC failed to P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL whenever the RPC errors (scripts/queue-dispatcher.sh step 2). Treat any such alert as an early warning of this exact failure mode — investigate before the budget drains.
  • When editing reset_stale_workers() or any function touching dev_r_worker_queue: the table uses server_node, not server_label (which only exists on dev_r_server_capacity). Always test the function with SELECT reset_stale_workers(); after a migration — a column typo fails silently at runtime, not at deploy time.
  • Periodic spot-check: SELECT available_ram_gb('bms-4'), available_ram_gb('vps-i1'); should always be > 0 during normal operation.

Escalation

If RAM stays at 0 after cleanup and the function runs clean:

  1. The dev_r_server_capacity config may be wrong — verify os_ram_gb, reserved_ram_gb, weight_ram_gb, and emergency_max_workers for the affected server (SELECT * FROM dev_r_server_capacity WHERE enabled;). The budget formula is (os_ram_gb − reserved_ram_gb) × 0.80.
  2. Check for genuinely runaway workers (pid still alive on the host) holding real RAM — those are not phantom and should not be force-reset.
  3. Send a P1 Discord alert and open a GitHub issue per the Error Notification Standard (CLAUDE.md).

Graph View

Backlinks