queue-dispatcher-priority

Queue Dispatcher — Job Priority and Routing

Last updated: 2026-07-09

Job priority reference (lower = claimed first)

Fixed 2026-07-09 (#3526) after a live starvation incident: unlabeled dev-issue jobs defaulted to priority=100 in .github/workflows/dispatch-to-queue.yml, while review-pr jobs were continuously regenerated every ~2min dispatcher cycle at priority=15–20 — review-pr always won a free dispatch slot, and 51+ dev-issue jobs sat queued for 22+ hours. review-pr itself had two insertion paths that had drifted apart (15 in scan_prs_for_review() vs 20 in the pull_request handler); both are now aligned to 15.

Job type	Priority	Set in
Credential-rotation reminder issues	1	.github/workflows/rotate-schedule.yml
dev-issue — P1 label	10	dispatch-to-queue.yml (issues.milestoned, Triage)
Auto-filed error/bug issues (agent-push-error.py)	10	scripts/agent-push-error.py
dev-issue — unlabeled default	12	dispatch-to-queue.yml (issues.milestoned, Triage)
review-pr	15	dispatch-to-queue.yml (pull_request handler) and scan_prs_for_review() in scripts/queue-dispatcher-loop.py — keep these two in sync
review-plan	30	dispatch-to-queue.yml (Review milestone, Code-change-design body)
continue-issue	40	dispatch-to-queue.yml (Review milestone re-queue)
CF Worker default (meta-dispatcher/src/index.ts, no explicit priority)	10	infra-src/meta-dispatcher/src/index.ts
CF Worker wave-based dev-issue queueing (/issues-review batches)	waveNumber*10 ± label/size/billing-tier adjustments (roughly -16..40)	computePriority() in infra-src/meta-dispatcher/src/wave-builder.ts
dev-issue — P2 label	50	dispatch-to-queue.yml (issues.milestoned, Triage)
DSA audit issues (create-audit-issues.py)	100	scripts/dsa/create-audit-issues.py

Resulting relative order for the single-issue Triage path: P1 (10) < default (12) < review-pr (15) < review-plan (30) < continue-issue (40) < P2 (50).

When adding a new job-type insertion point: check this table first so the new priority doesn’t accidentally starve or get starved by an existing job type, and update the table when you add or change one.

How jobs reach workers

The queue dispatcher (scripts/queue-dispatcher-loop.py) is a centralized push model, not a pull model. Workers do NOT poll for jobs autonomously.

Systemd timer (every 2 min, bms-4)
  → queue-dispatcher.sh  (claims lease, loads servers from Supabase)
    → queue-dispatcher-loop.py  (claims jobs, SSHes to targets, runs spawn-worker.sh)

1. queue-dispatcher.sh runs every 2 minutes via systemd timer on bms-4.
2. It claims the dispatcher lease via claim_dispatcher_lease() RPC — only one server is leader at a time. The lease is sticky with no failback: bms-4 (the intended active dispatcher) holds it indefinitely while healthy; vps-i1 only takes over if bms-4’s lease expires (host down >5 min), and leadership never returns to bms-4 automatically afterward. See docs/worker-queue-operations.md → Dispatcher lease election (sticky leader — no failback).
3. Loads enabled servers from dev_r_server_capacity?enabled=eq.true.
4. Exports them as DISPATCHER_SERVERS env var.
5. Calls queue-dispatcher-loop.py which does the actual dispatch.

Server ordering (dispatch priority)

Since #2421, servers are sorted so small hosts come first in the dispatch loop:

vps-i1 → vps-h1 → dev-laptop → bms-4

_SMALL_HOSTS = {"vps-i1", "vps-h1", "dev-laptop"} — processed before bms-4.

Small hosts have max_weight_prime=light in Supabase, so they only claim light jobs. bms-4 (max_weight_prime=orchestrator) gets whatever remains after small hosts have taken their light slots.

Job weight routing

Server	max_weight (prime)	max_workers	Notes
vps-i1	light	1	Monitoring stack — conservative
vps-h1	light	1	WAHA gateway — 2 cores only
dev-laptop	light	2	Dev workstation
bms-4	orchestrator	4	Heavy compute, all weights. Per-weight concurrency caps: light=10, heavy=1 (BMS4_MAX_CONCURRENT_LIGHT / BMS4_MAX_CONCURRENT_HEAVY, #3435)

weight_filter(max_weight) converts the DB setting to a PostgREST query fragment:

• light → weight=eq.light
• heavy → weight=in.(light,heavy)
• orchestrator → weight=in.(light,heavy,playwright,orchestrator)

Job claim ordering

Within a server’s dispatch pass, jobs are claimed in order: priority ASC, queued_at ASC.

Lower priority number = claimed first. Same priority = older jobs first (FIFO).

bms-4 heavy-first two-pass dispatch (#2422, #3435)

bms-4 runs a two-pass dispatch so heavy jobs claim their slots before light jobs grab them:

1. heavy-first pass — claim heavy/playwright/orchestrator jobs, capped at BMS4_MAX_CONCURRENT_HEAVY = 1 concurrent (#3435).
2. light-fill pass — fill remaining RAM with light jobs, capped at BMS4_MAX_CONCURRENT_LIGHT = 10 concurrent (#3435).

Each pass’s count cap is re-checked every iteration (via get_active_heavy_count() / get_active_light_count(), which read claimed/running rows), so a job claimed earlier in the same pass is counted before the next is considered. These per-weight caps replaced the single BMS4_MAX_TOTAL_WORKERS = 7 total-worker ceiling (#3023). Booked RAM (ram_available > 0) and the per-server slot cap (#3340) remain as underlying gates.

Host affinity

Some job types are pinned to specific servers via server_preference column:

• nc-alert-batch is affined to vps-i1 (reads localhost:9090 Prometheus)

When vps-i1 is disabled, stranded nc-alert-batch rows are auto-cancelled by cancel_unroutable_nc_alert_batch() after NC_ALERT_STRAND_MINUTES of waiting.

Night mode

Between 20:00–02:59 UTC, IS_NIGHT=true:

• Uses max_workers_night (smaller) instead of max_workers_prime
• Uses max_weight_night for weight filtering

Subscription balancing

When a server has multiple claude_accounts (JSONB array in dev_r_server_capacity):

• Non-depleted accounts are tried first
• If all accounts return exit code 5 (subscription at limit), job is requeued with 1h backoff
• _auto_unlock_reset_accounts() checks reset_at per cycle and clears depleted=True when reset time passes

Troubleshooting dispatch gaps

1. Check dev_r_server_capacity?enabled=eq.true — any servers enabled?
2. Check server_status — unknown status is health-gated out of dispatch
3. Check dispatcher lease: dev_r_dispatcher_leases — which server holds it?
4. Check logs: /var/log/p24-infra-dispatcher.log on bms-4
5. Check dev_r_worker_queue?status=eq.queued — jobs waiting?