p24-infra Docs

dispatcher-oom-loop

4 min read

Dispatcher OOM Loop — Playbook

What triggers this

A queue worker gets OOM-killed by its cgroup (MemoryMax enforced by systemd-run --scope) after spawn-worker.sh already returned exit 0 (worker was alive at the 1-second check). The dispatcher marks the row running. When the worker dies silently, the row stays running.

How to confirm

-- Find stuck running rows older than 2 hours
SELECT id, issue_number, status, weight, retry_count, max_retries,
       started_at, server_node, error_message
FROM dev_r_worker_queue
WHERE status = 'running'
  AND started_at < NOW() - INTERVAL '2 hours';

If retry_count is 0 and you see the same issue looping repeatedly, it’s this bug.

Also check for re-dispatch cycles:

SELECT issue_number, COUNT(*) AS dispatch_count, MAX(completed_at) AS last_seen
FROM dev_r_worker_queue
WHERE issue_number = <N>
GROUP BY issue_number;

Root cause (fixed in migration 20260625_fix_reset_stale_workers_retry_count.sql)

reset_stale_workers() was resetting stale running rows to queued WITHOUT incrementing retry_count. The termination condition retry_count < max_retries was always 0 < 2, creating an infinite loop every ~2 hours.

Secondary bug: queue-dispatcher.sh fetched max_weight (old column) instead of max_weight_prime/max_weight_night (migration 040 columns), causing weight_filter(None) to accept all weights during prime time.

Step-by-step fix

Immediate (manual — if loop is in progress)

  1. Find the looping row:

    SELECT id, retry_count, max_retries FROM dev_r_worker_queue
WHERE issue_number = <N> AND status IN ('running','queued');

  2. Force-fail it to break the loop:

    UPDATE dev_r_worker_queue
SET status = 'failed', error_message = 'manual: OOM loop — break'
WHERE id = <row_id>;

  3. Optionally re-queue with higher weight if the task is legitimate:

    INSERT INTO dev_r_worker_queue (job_type, weight, issue_number, repo, priority)
VALUES ('dev-issue', 'heavy', <N>, 'radieu/p24-infra', <original_priority>);

Permanent fix (already applied)

Migration 20260625_fix_reset_stale_workers_retry_count.sql:

  • reset_stale_workers() now increments retry_count on each stale-reset
  • After max_retries resets, row is set to failed (terminal)

queue-dispatcher.sh:

  • Now fetches max_weight_prime,max_weight_night (migration 040 columns) instead of max_weight

Escalation

If rows keep looping after the fix:

  1. Check spawn-worker.sh exit code handling — confirm exit 2 is being returned on cgroup OOM
  2. Check systemd cgroup memory events: journalctl -u user@<uid>.service | grep -i oom
  3. Increase MemoryMax for heavy workers in spawn-worker.sh if 6GB is insufficient
  4. Create issue with label human-action if OOM is persistent at max weight

Small-host OOM anti-affinity reroute (#6093)

Since #6093, the dispatcher’s exit=2 (cgroup OOM, worker dead) handler in queue-dispatcher-loop.py no longer loops forever on a repeatedly-OOM’ing small host. Small hosts (_SMALL_HOSTS = vps-i1 / vps-h1 / dev-laptop) are sorted first every cycle (#2421), so without intervention a small host re-claims its own just-OOM’d job before bms-4’s light-fill pass ever sees it — dead-ending in human-action after MAX_SPAWN_FAILURES (the #6085 incident: 6x OOM on dev-laptop). Now, after OOM_REROUTE_AFTER (2) spawn failures on a small host, the retry is pinned with server_preference='bms-4' (more RAM headroom), mirroring the exit=255 SSH-failure reroute. bms-4’s own OOMs are left alone — there is no larger host to route to, so backoff + eventual MAX_SPAWN_FAILURES escalation remains the only path there. This is host rerouting, not weight escalation (exit=4) — bumping weight would demand even more RAM on an already-exhausted host (#1675). The server_preference is soft for dev-issue rows, so _release_stale_server_preferences() still frees it after SOFT_PREF_MINUTES if bms-4 is itself saturated.

Prevention

  • Monitor retry_count approaching max_retries: alert if any row has retry_count >= 2
  • Add a Grafana panel: count of status='failed' rows with error_message ILIKE '%oom%'
  • Heavy MemoryMax was already raised from 6GB to 8GB on bms-4 (32GB total, headroom available) — applied 2026-06-25 (#1737) and live in queue-dispatcher-loop.py’s current WEIGHT_RAM_GB (heavy=8). Do not re-propose this; it is shipped, not pending.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="meta-dispatcher",
    result="success",  # "success" | "failed" | "skipped"
    detail="Dispatcher OOM loop resolved — memory limit applied, service restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'meta-dispatcher', 'success', 'Dispatcher OOM loop resolved — memory limit applied, service restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks