p24-infra Docs

worker-queue-operations

3 min read

Playbook: Worker Queue — Stuck Jobs, OOM, Failover

Trigger

  • Worker stays running for >2 hours without completing
  • WorkerQueueJobStuck Prometheus alert fires
  • Jobs stay queued for >10 min with no dispatcher activity
  • Discord alert: “queue-dispatcher error” or “worker re-dispatch”

Confirm

-- Queue depth by status
SELECT status, COUNT(*) FROM dev_r_worker_queue GROUP BY status ORDER BY COUNT(*) DESC;
 
-- Who holds the dispatcher lease?
SELECT holder, expires_at, NOW() > expires_at AS expired FROM dev_r_dispatcher_lease;
 
-- Active workers (running)
SELECT id, job_type, server_node, weight, started_at, pid
FROM dev_r_worker_queue WHERE status = 'running';
 
-- Stuck jobs (>2h — should have been re-queued already)
SELECT id, job_type, server_node, started_at, retry_count
FROM dev_r_worker_queue
WHERE status = 'running' AND started_at < NOW() - INTERVAL '2 hours';
# Check dispatcher timer — primary host is bms-4
ssh root@54.36.123.110 "systemctl status p24-queue-dispatcher.timer"
ssh root@54.36.123.110 "journalctl -u p24-queue-dispatcher.service --since '10 min ago'"

Fix — Stuck job

Automatic: reset_stale_workers() runs every 2 min, re-queues running rows after 2h (max 3 retries, then failed).

If reset_stale_workers() isn’t running (dispatcher is dead), manually re-queue:

UPDATE dev_r_worker_queue SET status = 'queued', retry_count = retry_count + 1 WHERE id = <N>;
-- or force stale cleanup:
SELECT public.reset_stale_workers();

Fix — Queue not draining (no dispatcher)

  1. Check timer on bms-4: ssh root@54.36.123.110 "systemctl status p24-queue-dispatcher.timer"
  2. Check lease: SELECT holder, expires_at FROM dev_r_dispatcher_lease;
  3. If timer stopped: ssh root@54.36.123.110 "systemctl enable --now p24-queue-dispatcher.timer"
  4. If lease stuck on dead server: UPDATE dev_r_dispatcher_lease SET expires_at = NOW() - INTERVAL '1 second'; → forces failover on next cycle
  5. Trigger dispatcher manually: ssh root@54.36.123.110 "systemctl start p24-queue-dispatcher.service"

Fix — OOM worker (exit 2)

Dispatcher auto-re-dispatches as weight=heavy. If it OOM-kills again at heavy:

UPDATE dev_r_worker_queue
SET status = 'failed', error_message = 'manually failed — OOM at heavy'
WHERE id = <N>;

Split the issue into smaller pieces or implement it manually.

Fix — bms-4 fully down (worst case)

  1. Verify bms-4 is genuinely down: ping 54.36.123.110
  2. Enable vps-i1 as emergency fallback: 
    UPDATE dev_r_server_capacity SET enabled = true WHERE server_label = 'vps-i1';
  3. vps-i1 will claim the dispatcher lease on its next 2-min tick and start dispatching
  4. Create a human-action GH issue: “bms-4 down — restore and re-disable vps-i1 when back”

Fix — All slots full

SELECT server_node, COUNT(*)
FROM dev_r_worker_queue WHERE status = 'running'
GROUP BY server_node;

If legitimately full: wait. If ghost rows (workers crashed but not re-queued):

SELECT public.reset_stale_workers();

Escalation

If queue stays stuck for >30 min with no dispatcher activity:

  1. Create GH issue: label human-action — “Queue dispatcher down — manual intervention required”
  2. SSH to bms-4, check disk (df -h) and memory (free -h), restart timer
  3. If bms-4 unreachable: enable vps-i1 (see above) and notify

Prevention

  • Grafana dashboard worker-queue-v1 shows queue depth, running count, stuck-job heat-map
  • WorkerQueueJobStuck alert fires after 2h — respond before it escalates
  • Review dev_r_server_capacity monthly — enable bms-3 if bms-4 is routinely at capacity

Graph View

Backlinks