p24-infra Docs

dispatcher-keyerror-stall

4 min read

Playbook — worker dispatch stalled (queue grows, nothing runs)

Symptom: dev_r_worker_queue accumulates rows with status=queued and none move to running. No claude -p worker processes spawn on bms-4. GitHub issues stay ai-dev-queued with no PRs.

Trigger / confirm

  1. Count queued jobs:

    SELECT status, count(*) FROM dev_r_worker_queue GROUP BY status;

    Many queued, 0 running → dispatch is stalled.

  2. Check the leader’s dispatcher log on vps-i1 (primary leader):

    ssh -i ~/.ssh/id_ed25519 root@217.154.82.162 \
  "grep -E 'Leader|\[loop\]|WARN|AUTO-HEAL' /var/log/p24-infra-dispatcher.log | tail -20"
    • Leader: dispatching (server=vps-i1) with no following [loop] lines → the Python loop is crashing before its first log() call (likely a KeyError on a missing DB column — see root cause below).
    • WARN: bms-4 idle but N job(s) queued (weights=[...], server_max_weight=...) → weight mismatch; dispatcher will auto-heal within 2 minutes.
    • AUTO-HEAL: bms-4.max_weight_prime bumped light → heavy → already self-healed.
    • Not leader — skipping dispatch. on both vps-i1 and bms-4 → lease problem (check dev_r_dispatcher_lease); bms-4’s service-role key 401s (#1329) so bms-4 can never win the lease — that is expected, vps-i1 is primary.

  3. Reproduce the crash with the real env on the leader:

    ssh -i ~/.ssh/id_ed25519 root@217.154.82.162 'bash -lc "
  cd /opt/p24-infra; set -a; source monitoring/.env; set +a
  export SUPABASE_SERVICE_ROLE_KEY=\${SUPABASE_SERVICE_ROLE_KEY:-\$SUPABASE_SERVICE_KEY}
  export DISPATCHER_SUPA_URL=https://mwkqmgadqnkkihjdeqsi.supabase.co
  export DISPATCHER_SERVER_NAME=vps-i1 DISPATCHER_IS_NIGHT=false DISPATCHER_LOG=/tmp/d.log
  export DISPATCHER_SERVERS=\$(curl -sf \"\$DISPATCHER_SUPA_URL/rest/v1/dev_r_server_capacity?enabled=eq.true&select=*\" -H \"apikey: \$SUPABASE_SERVICE_ROLE_KEY\" -H \"Authorization: Bearer \$SUPABASE_SERVICE_ROLE_KEY\")
  python3 scripts/queue-dispatcher-loop.py"'

    A KeyError here names the column the bash SELECT failed to fetch.

Root causes

1. Missing column in dispatcher SELECT (most common)

scripts/queue-dispatcher-loop.py reads a dev_r_server_capacity column that scripts/queue-dispatcher.sh’s SELECT (section “3. Load server capacity”) does not fetch. The missing key never reaches SERVERS, so server["<col>"] throws KeyError before the first log line, the bash wrapper swallows it via || discord_error, and every 2-min cycle silently does nothing.

Seen 2026-06-25 (#1398): migration 040 added max_weight_prime/max_weight_night columns but did not add them to the bash SELECT.

Prevention: whenever queue-dispatcher-loop.py starts reading a new dev_r_server_capacity column, add it to the SELECT in queue-dispatcher.sh in the same PR. Keep the loop’s first log() line at the very top of main() (before any server[...] access) so a crash is visible in the log, not silent.

Fix:

# Confirm current SELECT has all needed columns
grep 'max_weight_prime' /opt/p24-infra/scripts/queue-dispatcher.sh
# If missing, pull latest dev which includes the fix
cd /opt/p24-infra && git pull --ff-only

2. Weight mismatch (server max_weight too low for queued jobs)

All queued jobs are heavy but dev_r_server_capacity.max_weight_prime = 'light'. The dispatcher auto-heals this since 2026-06-25 — it bumps max_weight_prime to match the heaviest queued weight and posts a Discord warning. No human action needed unless auto-heal itself fails.

Confirm auto-heal ran:

grep 'AUTO-HEAL' /var/log/p24-infra-dispatcher.log | tail -5

Manual fix if auto-heal failed:

UPDATE dev_r_server_capacity
SET max_weight_prime = 'heavy'
WHERE server_label = 'bms-4';

Escalation

If the lease is held but the loop still won’t dispatch after column + weight fixes:

  • Check spawn-worker.sh exit codes in [loop] lines (spawn failed exit=N)
  • Check bms-4 RAM: ssh root@54.36.123.110 "free -g"
  • MongoDB/quorum/credential items remain human-gated

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="meta-dispatcher",
    result="success",  # "success" | "failed" | "skipped"
    detail="Dispatcher KeyError stall cleared — stale job fixed and dispatcher restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'meta-dispatcher', 'success', 'Dispatcher KeyError stall cleared — stale job fixed and dispatcher restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks