Worker Queue — Operations Manual

The unified worker queue routes all Claude Code worker spawns through Supabase and enforces per-server concurrency limits via a centralized HA dispatcher.

Architecture

┌──────────────────────────────────────────────────────────────────┐
│  JOB SUBMITTERS                                                  │
│  hourly-devops-triage (Phase 3) · /new-issue · manual INSERT     │
└──────────────────────────┬───────────────────────────────────────┘
                           │ INSERT  priority int (10=urgent 50=normal 100=low)
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│  dev_r_worker_queue  (Supabase PostgreSQL)                       │
│  status: queued → running → done                                 │
│                         └──→ failed / oom_killed / cancelled     │
└──────────────────────────┬───────────────────────────────────────┘
                           │ every 2 min (systemd timer)
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│  queue-dispatcher.sh  (bms-4)                                    │
│  ① claim_dispatcher_lease('bms-4')  → is leader?                │
│  ② reset_stale_workers()            → re-queue running > 2h     │
│  ③ load dev_r_server_capacity       → enabled servers + limits  │
│  ④ queue-dispatcher-loop.py         → fill slots by priority    │
└──────────────────────────┬───────────────────────────────────────┘
                           │ SSH + sudo spawn-worker.sh
                           ▼
┌──────────────────────────────────────────────────────────────────┐
│  bms-4  (sole active worker host)                                │
│  spawn-worker.sh: systemd-run --scope --MemoryMax=<weight>       │
│  → claude --dangerously-skip-permissions -p "<worker-prompt>"    │
│  → PID returned → dispatcher PATCH status=running, pid=PID       │
│  → worker completes Step 11 → PATCH status=done, completed_at   │
└──────────────────────────────────────────────────────────────────┘

Role-Based Routing (#1706)

The meta-dispatcher classifies every task at enqueue time and writes three columns to dev_r_worker_queue. The worker then reads role to select its agent prompt — no inference at spawn time — and each server claims only the rows it should run.

Column	Values	Meaning
`role`	`infra` \| `dev` \| `orchestrator` (default `dev`)	Selects the agent prompt in `spawn-worker.sh`: `infra`→`infra-task-request-worker.md`, `dev`→`worker-issue.md`, `orchestrator`→`worker-issue.md` (Step 5-ORCH fan-out lives there)
`repo_context`	repo slug or NULL	For `role=dev`: repo to clone (e.g. `radieu/et-operational-platform`). NULL = default `radieu/p24-infra`
`server_preference`	`vps-i1` \| `bms-4` \| NULL	Server affinity. A server claims rows where `server_preference = <its label> OR server_preference IS NULL`

Classification rules (infra-src/meta-dispatcher/src/classify.ts, applied in singleDispatch() + runMetaDispatcher()):

Signal	role	repo_context	server_preference
`job_type` `review-pr`/`review-plan` (light)	`dev`	repo if dev repo, else NULL	`vps-i1`
Title prefix `[alert]` or `[secret-rotation]`	`infra`	NULL	`bms-4`
Label `infra-task-request`	`infra`	NULL	`bms-4`
Repo `radieu/p24-infra` + label `bug`	`infra`	NULL	`bms-4`
Dev repo (et-operational-platform, Art-Agency, brandpilot, radekkonarski-personal-brand)	`dev`	the repo slug	`bms-4`
Default (p24-infra dev-issue)	`dev`	NULL	NULL (any server)

Rollout safety: all three columns are additive with safe defaults — existing rows backfill to role='dev', NULL repo_context/server_preference. spawn-worker.sh falls back to job_type-based prompt selection when the ROLE arg is empty, so an old dispatcher and the new worker (or vice versa) interoperate during deploy.

Timing Reference

Event	Interval / Threshold
Dispatcher cycle	every 2 min (systemd timer on bms-4)
`reset_stale_workers()`	every 2 min (called inside each dispatcher run)
Job stale threshold	2h in `running` → re-queue (max 3 retries, then `failed`)
Dispatcher lease TTL	5 min (any enabled server claims after expiry)
SLA escalation — infra jobs	4h with no PR → P1 alert (triage Phase 1C watchdog)
Prometheus stuck-job alert	`running` > 2h → `WorkerQueueJobStuck` warning
`NightlyTriageNotRunning`	no heartbeat for >25h (nightly triage, daily 20:00 UTC) → warning

Job Lifecycle

queued ──(dispatcher claims)──→ running ──(worker Step 11)──→ done
                                   │
                    ┌──────────────┼───────────────────┐
                    ▼              ▼                    ▼
                 failed        oom_killed           cancelled
                    │
             retry_count < 3?
               yes → queued  (re-queued by reset_stale_workers after 2h)
               no  → failed  (terminal)

Self-Healing

Failure	Detection	Auto-recovery
Worker process crashes silently	`reset_stale_workers()` every 2 min	Re-queues after 2h running (max 3 retries)
Worker OOM (exit 2)	`spawn-worker.sh` exit code check	Re-queues as `weight=heavy`; terminal if heavy also OOM
Dispatcher process dies	systemd timer auto-restarts the service	Next 2-min tick starts a fresh dispatcher run
Dispatcher server (bms-4) reboots	Lease expires in 5 min	Timer auto-starts post-boot; lease claimed on first run
Lease stuck on dead server	`expires_at < NOW()` condition	Any enabled server claims leadership on its next cycle
bms-4 fully offline (extended)	No dispatching, no heartbeat	No auto-failover — vps-i1 disabled. Enable manually (see below). `NightlyTriageNotRunning` fires after >25h.
WIP label orphaned after crash	Phase 1A WIP watchdog (hourly triage)	Clears stale `WIP` labels when no active session row exists in Supabase

Dispatcher lease election (sticky leader — no failback)

The active dispatcher is chosen by the claim_dispatcher_lease(p_holder) RPC (monitoring/supabase/migrations/036_worker_queue_operational_extensions.sql), which queue-dispatcher.sh calls once per 2-min cycle. The election is a sticky lease with no failback and no preemption — it does not prefer any “nominal primary”. Two branches:

Branch	Condition	Effect
Renew	`holder = p_holder AND expires_at > NOW()`	Push `expires_at` to `NOW()+5min`, return `TRUE` (stay leader)
Acquire	`holder IS NULL OR expires_at < NOW()`	Take the lease, `acquired_at=NOW()`, `generation+1`, return `TRUE`

With a 5-min TTL and a 2-min renew cycle, whoever holds the lease keeps it indefinitely while it stays healthy. A follower (e.g. vps-i1) calls the RPC every cycle, matches neither branch (it is not the holder, and the holder’s lease has not expired), gets FALSE, logs Not leader, and exits. A follower only becomes leader once the current holder misses renewals for >5 min (host down) and the lease expires — and even then the lease does not return to the previous holder on its own; it is claimed by whichever enabled server’s timer fires first during the expired window.

This means a leader never fails back. Once bms-4 acquired the lease it holds it forever, by design. That is why bms-4 has been the leader continuously since 2026-07-30 even though vps-i1 is healthy — expected steady state, not a stuck failover (#5643). This is also preferable: bms-4 is the intended active dispatcher (see Active Worker Hosts — bms-4 enabled, 3 workers / playwright weight; vps-i1 disabled, 1 worker / light).

How to tell renewal from re-acquisition in dev_r_dispatcher_lease:

SELECT holder, acquired_at, expires_at, generation FROM dev_r_dispatcher_lease;

acquired_at frozen while expires_at keeps advancing → the same holder is renewing continuously (sticky, healthy). This is the normal picture.
acquired_at moving / generation climbing → the lease actually changed hands (a real failover happened because the prior holder went unhealthy).

To deliberately hand leadership back to a specific server, expire the lease so the next cycle re-elects (see Force dispatcher lease release); there is no automatic mechanism for it.

Active Worker Hosts

Server	Status	Max workers prime/night	Max weight
`bms-4` (`54.36.123.110`)	enabled	3 / 3	playwright
`vps-i1` (`217.154.82.162`)	disabled	—	—

To enable vps-i1 as emergency fallback:

UPDATE dev_r_server_capacity SET enabled = true WHERE server_label = 'vps-i1';

Priority System

Integer value	Priority	Used by
`10`	urgent	Infra alerts, P1 issues
`50`	normal	Routine dev issues
`100`	low	Background / deferred tasks

Lower number = higher priority. Column type is INTEGER — never pass a string.

Submitting a Job

-- Via Supabase MCP (recommended from local workstation — sb_secret_ key blocks REST)
INSERT INTO dev_r_worker_queue
  (job_type, weight, issue_number, github_issue_number, repo, priority, metadata)
VALUES
  ('dev-issue', 'light', 1234, 1234, 'radieu/p24-infra', 50, '{"source":"manual"}');

# Via hourly-devops-triage function (inside triage script)
Invoke-QueueWorker -issueNum 1234 -jobType "infra-alert" -weight "light" -priority 10
# priority must be [int] 10 / 50 / 100 — not a string

# Via bash on a server
curl -sf -X POST "$SUPABASE_URL/rest/v1/dev_r_worker_queue" \
  -H "apikey: $SUPABASE_SERVICE_KEY" \
  -H "Authorization: Bearer $SUPABASE_SERVICE_KEY" \
  -H "Content-Type: application/json" \
  -d '{"job_type":"infra-alert","weight":"light","issue_number":999,"repo":"radieu/p24-infra","priority":10}'

Key Tables

Table	Purpose
`dev_r_worker_queue`	All worker jobs — status, priority, weight, pid, retry_count
`dev_r_server_capacity`	Per-server slot limits and weight class support
`dev_r_dispatcher_lease`	HA singleton — which server is active dispatcher
`dev_r_agent_sessions`	Per-session tracking (INSERT at Step 0, PATCH at Step 11)

Monitoring

Grafana dashboard: grafana.vps-i1.infra.zintegrowana.online/d/worker-queue-v1

Panels: queued/running/done/failed stat counters · queue depth timeseries · active workers table with running-time heat-map · recent 25 jobs table with status colour coding.

Prometheus metrics (queue-exporter :9200 on vps-i1):

supabase_queue_depth{queue="dev_r_worker_queue", status="queued"}
supabase_queue_depth{queue="dev_r_worker_queue", status="running"}
supabase_queue_depth{queue="dev_r_worker_queue", status="done"}
supabase_queue_depth{queue="dev_r_worker_queue", status="failed"}
supabase_queue_depth{queue="dev_r_worker_queue", status="oom_killed"}
supabase_queue_depth{queue="dev_r_worker_queue", status="cancelled"}

Alertmanager rules (monitoring/prometheus/rules/ai-workers.yml):

Alert	Condition	Severity
`WorkerQueueJobStuck`	any job `running` > 2h	warning
`NightlyTriageNotRunning`	no triage heartbeat for >25h (nightly, daily 20:00 UTC)	warning
`NightlyTriageFailing`	last triage run ended `failed`	warning

Quick-check SQL:

-- Queue depth by status
SELECT status, COUNT(*) FROM dev_r_worker_queue GROUP BY status ORDER BY COUNT(*) DESC;
 
-- Who holds the dispatcher lease?
SELECT holder, expires_at, NOW() > expires_at AS expired FROM dev_r_dispatcher_lease;
 
-- Active workers
SELECT id, job_type, server_node, weight, started_at, pid
FROM dev_r_worker_queue WHERE status = 'running';
 
-- Stuck jobs (>2h — alert territory)
SELECT id, job_type, server_node, started_at, retry_count
FROM dev_r_worker_queue
WHERE status = 'running' AND started_at < NOW() - INTERVAL '2 hours';

# Check dispatcher timer on bms-4
ssh root@54.36.123.110 "systemctl status p24-queue-dispatcher.timer"
ssh root@54.36.123.110 "journalctl -u p24-queue-dispatcher.service --since '10 min ago'"

Manual Operations

Re-queue a stuck job

UPDATE dev_r_worker_queue
SET status = 'queued', retry_count = retry_count + 1
WHERE id = <N>;

Fail a job permanently

UPDATE dev_r_worker_queue
SET status = 'failed', error_message = 'manually failed — <reason>'
WHERE id = <N>;

Kill a running worker

# List active worker scopes on bms-4
ssh root@54.36.123.110 "systemctl list-units 'p24-worker-*' --state=active"
 
# Stop a specific worker
ssh root@54.36.123.110 "systemctl stop p24-worker-dev-issue-1234.scope"

Then manually update the queue row or let reset_stale_workers() catch it after 2h.

Force dispatcher lease release

-- Forces any enabled server to claim leadership on its next 2-min cycle
UPDATE dev_r_dispatcher_lease SET expires_at = NOW() - INTERVAL '1 second';

Trigger dispatcher manually

ssh root@54.36.123.110 "systemctl start p24-queue-dispatcher.service"

Force re-queue all stale running jobs now

SELECT public.reset_stale_workers();

Dispatcher Setup (one-time per new server)

# 1. Server identity file
echo "bms-4" > /opt/p24-infra/.server-label
 
# 2. Linger for claude-runner (allows systemd-run from SSH)
loginctl enable-linger claude-runner
 
# 3. Sudoers entry
echo "root ALL=(claude-runner) NOPASSWD: /opt/p24-infra/scripts/spawn-worker.sh" \
  > /etc/sudoers.d/p24-spawn-worker
chmod 440 /etc/sudoers.d/p24-spawn-worker
 
# 4. Enable timer
systemctl enable --now p24-queue-dispatcher.timer

Weight Classes

Weight	Memory limit	Eligible servers
`light`	3 GB	bms-4 (and vps-i1 if enabled)
`heavy`	8 GB	bms-4
`playwright`	6 GB	bms-4 only

p24-infra Docs

Explorer

worker-queue-operations