Worker-queue budget guard (#7674)
When the combined Claude-subscription headroom runs low, the dispatcher protects important work by
shedding low-priority jobs first and reserving headroom for P0/P1, instead of letting every job
type compete until both accounts hit their limits. (Background: 2026-10-08 — claude-runner went
9% → 100% of its 7-day window in ~2 days and claude-runner-2 hit its 5h limit the same evening;
~32 jobs failed and ~20 issues/PRs were marked human-action. review-pr alone put 26 + 25 rows
on PRs 7586 in 24h.)
This is the policy layer on top of #7662 (limit detection + failover to the other account) and
#7663 (review-pr burn). The fixed constraint (owner, 2026-10-09): there are only two subscriptions
(claude-runner, claude-runner-2) and they must be enough — so when headroom is tight, spend it
on what matters.
Implementation lives entirely in scripts/queue-dispatcher-loop.py + the tracked config
scripts/dispatcher-budget-config.json. The protected infra/agent-prompts/review-pr-worker.md
is not touched.
How it works
Fleet headroom
Each dispatcher cycle computes fleet headroom = max over all non-depleted accounts of
each account’s remaining headroom (100 − most-constrained-window-utilization). It uses the same
per-account source (_account_headroom, SSH read of ~/.claude/sub-status.json, 5-min cache) as
the dispatch ranker, so the guard can never disagree with routing. Both subscriptions live on
bms-4, so one host covers both accounts; max is idempotent across duplicate reads from other
hosts. A depleted account contributes no headroom (it is already at its limit).
If no non-depleted account anywhere reports a usage signal, fleet headroom is unknown and the guard fails open — dispatch proceeds exactly as before.
Zones (thresholds on fleet headroom)
| Zone | Condition | Effect |
|---|---|---|
normal | headroom ≥ BUDGET_SOFT_PCT (30) | everything dispatches |
soft | BUDGET_HARD_PCT (12) ≤ headroom < soft | low-value classes held |
hard | headroom < BUDGET_HARD_PCT (12) | only P0/P1 + alert-response dispatch |
off | guard disabled, or headroom unknown | fail-open — dispatch as today |
Job classes
Each dev_r_worker_queue row is classified from its priority int + job_type + weight
(priority is INT, lower = higher priority):
| Class | Matches | Held? |
|---|---|---|
alert | alert-response job types (alert-response, infra-alert, nc-alert-batch, alert-triage-batch) | never |
p1 | priority ≤ BUDGET_P1_MAX_PRIORITY (12) — incl. the default dev-issue @12; a NULL priority defaults here (protected) | never |
low | priority ≥ BUDGET_LOW_MIN_PRIORITY (50, i.e. P2/P3), a low-value job type (night-task/nightly-cron), a heavy/playwright/orchestrator weight, or a docs-only / unchanged-sha review-pr | soft and hard |
mid | everything else, e.g. a normal code review-pr @15 | soft: no · hard: yes |
The priority cut-offs match the current scheme: P1 (10) < default dev-issue (12) < review-pr (15) < P2 (50).
What “held” means
A held row is left queued — the dispatcher briefly claims it then releases it back with
claimed_at=NULL, with no spawn_failures/retry_count bump and no human-action label.
It is logged budget_hold class=… headroom=… zone=… and skipped for the rest of that cycle (an
in-cycle id skip-set, so no busy-loop). It auto-resumes on a later cycle the moment the zone
widens (headroom recovers or a subscription window resets).
For review-pr specifically, the gate also runs at the enqueue source
(scan_prs_for_review()): in the hard zone no review-pr rows are created at all, and in the
soft zone docs/CHANGELOG/.md-only PRs are skipped — so the review burn never even materialises
as queue rows. (Unchanged-head-sha re-reviews are already deduped by the ai-review-queued label.)
No false stalled/SLA
Held rows are plain long-queued unique rows. Verified against the cancellation paths:
queue-retry-worker.py Part A only cancels duplicate queued rows (an issue that already has a
running/claimed/done row), Part B only retries failed rows, Part C only touches stale_duplicate
and stuck-claimed rows (held rows have claimed_at=NULL). The queue-exporter’s
QUEUE_POSTPONED/QUEUE_RUNNING_STALE key off next_attempt_at/running-progress, neither of which
a held row sets. So nothing ages a held row into a stalled/SLA false alarm — no exemption code is
required.
Configuration
scripts/dispatcher-budget-config.json is the tracked, non-secret source of truth (deployed to
/opt/p24-infra by deploy-vps-scripts.yml via git reset --hard, so it survives any .env
regeneration by secrets-sync.yml — the same durable-config pattern as
dispatcher-review-config.json, #6216). A matching env var in the dispatcher .env overrides
the file value (host-specific tweaks + emergency changes):
| Key | Default | Meaning |
|---|---|---|
BUDGET_GUARD_ENABLED | true | master switch — anything non-truthy fully disables (fail-open) |
BUDGET_SOFT_PCT | 30 | enter soft zone below this headroom % |
BUDGET_HARD_PCT | 12 | enter hard zone below this headroom % |
BUDGET_P1_MAX_PRIORITY | 12 | priority ≤ this is P0/P1 (never held) |
BUDGET_LOW_MIN_PRIORITY | 50 | priority ≥ this is low value (held first) |
BUDGET_STALE_HOLD_HOURS | 6 | hours a row must be continuously held before a stale-hold Discord + GH issue fires (#7843) |
Disable / tune (no deploy): set the env var in the dispatcher .env on bms-4, e.g.
BUDGET_GUARD_ENABLED=false to turn the guard off, or BUDGET_SOFT_PCT=20 to shed later. Env wins
over the file. Tune durably: edit dispatcher-budget-config.json and merge to main.
Observability
The dispatcher writes a Prometheus textfile (/var/lib/prometheus/node-exporter/p24_queue_budget.prom,
override via DISPATCHER_BUDGET_METRICS_FILE) every cycle, scraped by node_exporter on bms-4:
p24_queue_budget_headroom_pct— fleet headroom (NaN when unknown/fail-open).p24_queue_budget_held_jobs{class}— rows held this cycle, per class (always emits all four classes incl. zeros, so a drop-to-zero is visible rather than an absent series).p24_queue_budget_zone— 0 = off/normal, 1 = soft, 2 = hard.
Grafana: three panels on the Worker Queue dashboard (headroom gauge, zone stat, held-jobs-by- class bars).
Alert: WorkerQueueBudgetHardZone (monitoring/prometheus/rules/ai-workers.yml) — warning,
for: 10m, fires on p24_queue_budget_zone >= 2.
Discord (zone entry): the dispatcher posts a once-per-episode notice (via the standard
P24_DISCORD_INFRA_SCRIPTS_WEBHOOK_URL — the shared infra-scripts channel) only on the
transition into the hard zone — this is normal throttling, so it does not open a GH issue.
Discord + GH issue (stale hold, #7843): if a row has been continuously held by the budget guard
for more than BUDGET_STALE_HOLD_HOURS (default 6h), the dispatcher fires one deduplicated alert
per row per hold episode: a Discord warning AND a bug GH issue
([Infra] Budget guard: queue row N held Xh). If the row is released (zone recovers, row
dispatches, or it is cancelled) and then put on hold again later, the timer resets and a fresh
alert fires after the next 6h window. Episode state (first-seen timestamps + alerted row ids) is
stored alongside the zone-episode state in DISPATCHER_BUDGET_STATE_FILE
(/var/lib/p24-infra/dispatcher-budget-state.json).
infra-task P1 chain protection (#7800/#7801/#7843)
A dev-issue → infra-task reroute (Step 0b-ROLE in infra/agent-prompts/worker-issue.md) carries
the originating dev-issue priority into the new infra-task row (PR #7801). This means an
infra-task row created as the continuation of a P1 dev-issue (priority ≤ 12) inherits that priority
and is classified as p1 — it is never held by the budget guard, even in the hard zone.
A routine infra-task (priority 100, created directly without a P1 reroute context) remains
low and is held in both the soft and hard zones — this is intentional: routine server operations
are deferrable during headroom pressure.
Other infra-task paths:
| Source | Priority | Budget class |
|---|---|---|
dev-issue → infra-task reroute (Step 0b-ROLE) | inherited from dev-issue (≤ 12 = p1) | p1 — never held |
/queue-issue explicit POST with job_type=infra-task | caller-supplied; defaults to 10 | p1 — never held |
n8n infra-task-queue-processor (dev_r_infra_task_requests) | N/A — legacy table, budget guard only applies to dev_r_worker_queue | always dispatches (guard skips the legacy table) |
| Manual queue insert with priority 100 | 100 (low class) | held in soft + hard zones |
Fail-open guarantees
Every failure mode dispatches as before: headroom unknown (no signal / read error), a crash in the
headroom computation, or BUDGET_GUARD_ENABLED=false. The metrics/state/Discord writes are all
best-effort and never crash the dispatch loop.