Playbook: Queue Failure Handling

Related issues: #1811, #2081, #2091 Timer: p24-queue-retry.timer on vps-i1 (runs every 10 min) Script: /opt/p24-infra/scripts/queue-retry-worker.py Last updated: 2026-07-01 (Part E: trivial auto-requeue; Part F: worker OOM health monitor; FailedJobsAccumulating alert — #2379)

1. Status taxonomy — failed vs cancelled

dev_r_worker_queue has three terminal non-success states:

Status	Colour	Meaning
`cancelled`	🟢 green	Intentional early exit — work done elsewhere or not needed
`failed`	🔴 red	Actionable error — being retried, or awaiting human review
`failed_permanently`	⚫ black	Retry-exhausted and human-notified — retired terminal state

Rule: only genuine errors land in failed. Smart exits land in cancelled.

failed_permanently (#2500): once a failed row exhausts its retries (retry_count >= max_retries) and a human has been notified (queue-analyst escalation → human-action label, or the Path A permanent-failure webhook), the retry-worker (Part B) retires it to failed_permanently. This clears it from the active failed set so the FailedJobsAccumulating alert stops firing on terminally-dead jobs, while failure_reason / error_message / analyst_notes are preserved for audit. A failed_permanently row is inert — never re-selected by the dispatcher, retry-worker, or queue-analyst. The failure is still tracked on its GitHub issue (labelled human-action).

2. failure_reason taxonomy

`failure_reason`	`status`	Written by	Meaning	Action
`stale_duplicate`	`cancelled`	dispatcher loop	Issue/PR already closed before dispatch	None — correct behaviour
`mutex_collision`	`cancelled`	worker (claim_task NULL)	Another worker holds the per-issue lock	None — other worker is doing the work
`implementation_error`	`failed`	dispatcher loop	spawn-worker.sh exited non-zero (bad args, allowlist miss)	Check spawn log, fix root cause, re-queue
`timeout`	`failed`	dispatcher loop	SSH spawn timed out or max spawn retries hit	Check server load / RAM, re-queue if issue still open
`spawn_oom`	`failed`	dispatcher loop	Worker OOM-killed or weight ceiling reached	Re-dispatched automatically at heavier weight tier
`spawn_ram`	`failed`	dispatcher loop	Not enough free RAM at spawn time	Re-dispatched automatically when RAM frees
`other`	`cancelled`	retry-worker (Part A)	Duplicate queued row — issue already running/done	None

Key distinction (2026-06-29): stale_duplicate and mutex_collision were previously written as status=failed. This was wrong — they are correct behaviour, not errors. Fixed in PR #2091: both now write status=cancelled from the dispatcher and worker prompts.

3. Job types

Valid values for job_type in dev_r_worker_queue:

`job_type`	What it does	spawn-worker.sh supported
`dev-issue`	Implement a GitHub issue end-to-end	✅
`continue-issue`	Re-process an issue in Review milestone (address PR comments, CI fixes)	✅ (added PR #2081)
`review-pr`	Run `/review-pr` skill on an open PR	✅
`review-plan`	Run `/review-plan` skill on a design comment	✅
`infra-task`	Server-side infra operation (SSH, Docker, etc.)	✅
`infra-alert`	Respond to a Prometheus/Grafana alert	✅

Note: continue-issue was missing from spawn-worker.sh validation until 2026-06-29 (PR #2081), causing every continue-issue job to fail immediately with implementation_error / spawn failed exit=1. If you see a batch of implementation_error for continue-issue rows, check that spawn-worker.sh on bms-4 contains continue-issue in the allowlist (line ~51).

4. Quick analysis via SSH

# All failed jobs — breakdown by type and reason
SB_URL="https://mwkqmgadqnkkihjdeqsi.supabase.co"
ssh root@54.36.123.110 "
  set -a; source /opt/p24-infra/bms-4/.env; set +a
  curl -s '$SB_URL/rest/v1/dev_r_worker_queue?status=eq.failed&select=id,github_issue_number,job_type,failure_reason,error_message,retry_count,max_retries,queued_at&order=id.desc&limit=50' \
    -H \"apikey: \$SUPABASE_SERVICE_ROLE_KEY\" -H \"Authorization: Bearer \$SUPABASE_SERVICE_ROLE_KEY\"
" | python3 -c "
import sys, json
data = json.load(sys.stdin)
print(f'Total failed: {len(data)}')
for r in data:
    err = (r.get('error_message') or '')[:80].replace(chr(10),' ')
    print(f'  id={r[\"id\"]} #{r[\"github_issue_number\"]} [{r[\"job_type\"]}] reason={r[\"failure_reason\"]} retries={r[\"retry_count\"]}/{r[\"max_retries\"]}')
    if err: print(f'     {err}')
"

5. Decision tree: what to do with a failed row

failed row?
├── failure_reason = stale_duplicate / mutex_collision?
│   → Should already be cancelled (PR #2091). If status=failed, bulk-cancel (§ 9).
│
├── job_type = continue-issue AND failure_reason = implementation_error AND error contains "Invalid job_type"?
│   → spawn-worker.sh allowlist bug. Deploy fix (PR #2081), re-queue rows.
│
├── job_type = review-pr AND failure_reason = timeout?
│   → Check if PR is already merged via gh pr view <PR#>
│   ├── merged → cancel (review pointless)
│   └── open   → re-queue (§ 8)
│
├── failure_reason = timeout AND retry_count >= max_retries?
│   → Check if issue/PR still open. If open: reset retry_count and re-queue (§ 8).
│   → If closed: cancel (§ 9).
│
├── failure_reason = implementation_error?
│   → Check spawn log: /var/log/p24-infra-workers/<job_type>-<issue>.log
│   → Fix root cause, then re-queue (§ 8).
│
└── failure_reason = spawn_oom / spawn_ram?
    → Dispatcher handles automatically. Only intervene if stuck > 30 min.

6. Overview: retry-worker

queue-retry-worker.py runs as a systemd oneshot service every 10 minutes on vps-i1. It performs several housekeeping jobs on dev_r_worker_queue (Parts A–F):

Part A — Cancel duplicate queued rows When an issue already has a running, claimed, or done row, any additional queued rows for the same issue_number are cancelled with failure_reason='other' and a descriptive error_message. This prevents double-dispatch after network hiccups or manual re-queues.

Part B — Retry failed jobs Rows with status='failed' and failure_reason IN ('implementation_error', 'timeout') are re-queued up to max_retries times (default 3) with exponential backoff:

New retry_count	Backoff before next attempt
1	5 minutes
2	15 minutes
≥ 3	45 minutes (also sends a red “LAST retry” Discord alert)

Once a row reaches retry_count >= max_retries it is no longer retried. On the first such cycle Part B notifies a human (Path A permanent-failure webhook → GitHub comment + human-action label, falling back to github_issue_number when issue_number is NULL) or recognises an existing queue-analyst escalation; it then retires the row to status='failed_permanently' (see §1). Rows it cannot notify (no issue number and the webhook is down) stay in failed so the failure is never silently hidden — the FailedUnreviewedOld alert continues to surface those.

Part C — Delete old stale_duplicate rows Rows with failure_reason='stale_duplicate' older than 24 hours are permanently deleted (they accumulate if the dispatcher’s per-cycle prune misses them).

Part D — Reset stuck claimed rows Rows with status='claimed', started_at IS NULL, heartbeat_at IS NULL, and claimed_at older than 15 minutes are released back to status='queued'. This covers the case where the dispatcher claimed a job but the SSH spawn failed silently: spawn-worker.sh never ran, so neither started_at nor heartbeat_at was written. Without Part D these rows stay claimed forever, blocking re-dispatch.

Detection window: 15 minutes (generous enough to not fire during slow SSH, tight enough to recover before the next dispatcher cycle assigns the slot to something else).
Recovery: row reset to queued with a descriptive error_message; yellow Discord embed.
No retry_count bump — the dispatcher will re-claim the row on the next cycle. If the spawn continues to fail, retry_count increments naturally and the queue-analyst reroutes or escalates.

Part E — Auto-requeue trivial failures (#2379) Rows with status='failed' and failure_reason IN ('mutex_collision', 'spawn_ram') are re-queued directly, with no analyst gate — these are transient (a competing lock or momentary RAM contention), not code problems, so they need no diagnosis:

failure_reason	Backoff	Rationale
`mutex_collision`	immediate (0 min)	Another worker held the per-issue lock — just try again.
`spawn_ram`	5 minutes	RAM contention was transient; give the server a moment.

max_retries is still honoured, so a row that keeps colliding/starving cannot loop forever. Part B deliberately excludes these reasons — Part E is their only retry path.

Part F — Worker OOM health monitor (#2379) When one server_node accrues ≥3 spawn_oom failures within 30 minutes, the server is treated as unavailable:

Opens a GitHub issue [Infra] Worker {server} unavailable — spawn_oom xN (labels bug, infra-task), deduped against any existing open issue so the 10-min timer does not spam duplicates.
Sends a red Discord alert naming the server and affected issues.
Re-queues the affected jobs with server_node=NULL and server_preference=NULL so the dispatcher routes them to a different, healthy server. max_retries is honoured.

Thresholds: OOM_THRESHOLD=3, OOM_WINDOW_MINUTES=30 (module constants). Because re-routed rows leave status='failed', they drop out of the 30-min window immediately — providing natural idempotency for the re-route.

Prometheus alert — FailedJobsAccumulating (#2379) The queue-exporter exposes p24_worker_queue_failed_actionable (count of status='failed' rows with failure_reason IN (spawn_oom, timeout, implementation_error) — trivial reasons excluded). The FailedJobsAccumulating rule (rules/ai-workers.yml) fires when this gauge is > 3 for 15 min, i.e. genuine failures are piling up faster than Parts B/E/F clear them.

Relationship to the main dispatcher: The main dispatcher (queue-dispatcher.sh / queue-dispatcher-loop.py) claims and spawns workers. The retry-worker is a separate, lighter process that only manages queue state — it does not SSH to workers or spawn processes.

7. Trigger — when to read this playbook

You see a Discord embed with title containing retry-worker
Failed jobs are accumulating in Grafana’s Worker Queue panel
You see a LAST retry red Discord embed for a specific issue
Duplicate queued rows are visible in the database
systemctl status p24-queue-retry.timer is not active (waiting)
You see “released stuck claimed rows” in the retry-worker journal (Part D fired)
A job has been in claimed status for >15 min with no worker log appearing
A [Infra] Worker {server} unavailable — spawn_oom xN issue was auto-opened (Part F fired)
The FailedJobsAccumulating Grafana alert is firing

8. Confirm commands — Supabase SQL

Check failure_reason breakdown in the queue

-- Run in Supabase SQL Editor or psql
SELECT failure_reason, COUNT(*) AS count
FROM dev_r_worker_queue
WHERE status = 'failed'
GROUP BY failure_reason
ORDER BY count DESC;

Check for retryable failed rows

SELECT id, issue_number, retry_count, max_retries, failure_reason, error_message, queued_at
FROM dev_r_worker_queue
WHERE status = 'failed'
  AND failure_reason IN ('implementation_error', 'timeout')
ORDER BY queued_at ASC
LIMIT 20;

Check for duplicate queued rows

-- Issue numbers with more than one queued row
SELECT issue_number, COUNT(*) AS queued_count, array_agg(id) AS ids
FROM dev_r_worker_queue
WHERE status = 'queued'
  AND issue_number IS NOT NULL
GROUP BY issue_number
HAVING COUNT(*) > 1
ORDER BY queued_count DESC;

Check for stuck claimed rows (Part D)

-- claimed rows with no started_at / heartbeat_at older than 15 min
SELECT id, issue_number, server_node, claimed_at,
       NOW() - claimed_at AS stuck_duration
FROM dev_r_worker_queue
WHERE status = 'claimed'
  AND started_at IS NULL
  AND heartbeat_at IS NULL
  AND claimed_at < NOW() - INTERVAL '15 minutes'
ORDER BY claimed_at ASC;

Check timer status on vps-i1

ssh root@217.154.82.162 "systemctl status p24-queue-retry.timer"

View recent journal output

ssh root@217.154.82.162 "journalctl -u p24-queue-retry.service -n 50 --no-pager"

Manually trigger one run immediately

ssh root@217.154.82.162 "systemctl start p24-queue-retry.service"

9. Bulk-cancel safe rows (stale_duplicate / mutex_collision stuck as failed)

Run on bms-4 if rows with these reasons ended up in status=failed before PR #2091:

SB_URL="https://mwkqmgadqnkkihjdeqsi.supabase.co"
ssh root@54.36.123.110 "
  set -a; source /opt/p24-infra/bms-4/.env; set +a
  for REASON in stale_duplicate mutex_collision; do
    curl -s -X PATCH \"\$SB_URL/rest/v1/dev_r_worker_queue?status=eq.failed&failure_reason=eq.\${REASON}\" \
      -H \"apikey: \$SUPABASE_SERVICE_ROLE_KEY\" -H \"Authorization: Bearer \$SUPABASE_SERVICE_ROLE_KEY\" \
      -H 'Content-Type: application/json' -H 'Prefer: return=representation' \
      -d '{\"status\":\"cancelled\"}' | python3 -c \"
import sys,json; d=json.load(sys.stdin)
print(f'\${REASON}: cancelled {len(d)} rows')
\"
  done
"

10. Re-queue a specific failed row

-- Supabase SQL Editor — replace <JOB_ID>
UPDATE dev_r_worker_queue
SET
    status          = 'queued',
    retry_count     = 0,
    failure_reason  = NULL,
    error_message   = NULL,
    next_attempt_at = NULL,
    server_node     = NULL,
    claimed_at      = NULL,
    started_at      = NULL
WHERE id = <JOB_ID>
  AND status = 'failed';

Via SSH (REST):

SB_URL="https://mwkqmgadqnkkihjdeqsi.supabase.co"
JOB_ID=<ID>
ssh root@54.36.123.110 "
  set -a; source /opt/p24-infra/bms-4/.env; set +a
  curl -s -X PATCH \"\$SB_URL/rest/v1/dev_r_worker_queue?id=eq.\${JOB_ID}&status=eq.failed\" \
    -H \"apikey: \$SUPABASE_SERVICE_ROLE_KEY\" -H \"Authorization: Bearer \$SUPABASE_SERVICE_ROLE_KEY\" \
    -H 'Content-Type: application/json' \
    -d '{\"status\":\"queued\",\"failure_reason\":null,\"error_message\":null,\"claimed_at\":null,\"server_node\":null,\"retry_count\":0}'
"

11. Fix: timer not running

Symptom: systemctl status p24-queue-retry.timer shows inactive or failed.

Steps:

ssh root@217.154.82.162
# Check why it stopped
systemctl status p24-queue-retry.timer
journalctl -u p24-queue-retry.service -n 30 --no-pager
 
# Restart the timer
systemctl restart p24-queue-retry.timer
systemctl status p24-queue-retry.timer  # should show "active (waiting)"
 
# Verify the service runs successfully
systemctl start p24-queue-retry.service
journalctl -u p24-queue-retry.service -n 20 --no-pager

If the service itself fails: Check the environment file is present and readable:

ls -la /opt/p24-infra/monitoring/.env
grep -c "DISPATCHER_SUPA_URL" /opt/p24-infra/monitoring/.env
grep -c "SUPABASE_SERVICE_ROLE_KEY" /opt/p24-infra/monitoring/.env

If the .env is missing, re-sync secrets from SOPS (see docs/playbooks/secrets-sync-failure.md).

12. Fix: give a permanently-failed job another chance

Symptom: A specific job has retry_count >= max_retries. After human notification the retry-worker retires it to status='failed_permanently' (#2500); before notification it is still status='failed'. Either way it will not retry on its own. You want to re-run it.

Steps — reset retry counter in Supabase SQL Editor:

-- Replace <JOB_ID> with the actual id from dev_r_worker_queue.
-- Matches either terminal state ('failed' pre-notification, 'failed_permanently' after).
UPDATE dev_r_worker_queue
SET
    status         = 'queued',
    retry_count    = 0,
    failure_reason = NULL,
    error_message  = NULL,
    next_attempt_at = NULL,
    server_node    = NULL,
    claimed_at     = NULL,
    started_at     = NULL
WHERE id = <JOB_ID>
  AND status IN ('failed', 'failed_permanently');

The retry-worker will then pick it up on its next cycle and retry it with fresh backoff.

13. Fix: duplicates not being cancelled

Symptom: Multiple queued rows exist for the same issue_number, but the retry-worker is not cancelling them.

Root causes:

The retry-worker timer is not running (see § 4).
The duplicate rows have issue_number IS NULL — the retry-worker only deduplicates rows with a non-null issue_number.
The “active” row (the one that should block) is in status='queued' itself, not running/claimed/done — in this case both are legitimately queued and the dispatcher will claim one; the other will be blocked by the exclude_issues guard in the dispatcher.

Manual cancel for a specific issue_number:

-- Replace <ISSUE_NUMBER> with the actual GitHub issue number
-- Check first what you're about to cancel
SELECT id, status, queued_at, error_message
FROM dev_r_worker_queue
WHERE issue_number = <ISSUE_NUMBER>
ORDER BY queued_at ASC;
 
-- Cancel all but the most-recently-queued row (keeps the latest)
UPDATE dev_r_worker_queue
SET
    status         = 'cancelled',
    failure_reason = 'other',
    error_message  = 'manually cancelled: duplicate queued row'
WHERE issue_number = <ISSUE_NUMBER>
  AND status = 'queued'
  AND id NOT IN (
      SELECT id FROM dev_r_worker_queue
      WHERE issue_number = <ISSUE_NUMBER>
        AND status = 'queued'
      ORDER BY queued_at DESC
      LIMIT 1
  );

14. First-time bootstrap on vps-i1

Run once after the feat/1811-queue-retry-worker PR is merged to dev and /opt/p24-infra has been updated via git pull:

ssh root@217.154.82.162
 
# Copy systemd units
cp /opt/p24-infra/deploy/p24-queue-retry.service /etc/systemd/system/
cp /opt/p24-infra/deploy/p24-queue-retry.timer   /etc/systemd/system/
 
# Enable and start
systemctl daemon-reload
systemctl enable --now p24-queue-retry.timer
 
# Verify
systemctl status p24-queue-retry.timer

Expected output for the timer:

● p24-queue-retry.timer - p24 Queue Retry Worker - re-queues failed jobs and cancels duplicates
     Loaded: loaded (/etc/systemd/system/p24-queue-retry.timer; enabled; ...)
     Active: active (waiting) since ...
    Trigger: ...

Run a manual test to confirm the script connects to Supabase:

systemctl start p24-queue-retry.service
journalctl -u p24-queue-retry.service -n 30 --no-pager

Expected log lines:

[...] [retry-worker] === queue-retry-worker starting ===
[...] [retry-worker] Part A: checking for duplicate queued rows ...
[...] [retry-worker] Part B: fetching failed rows eligible for retry ...
[...] [retry-worker] === queue-retry-worker done: cancelled=0 requeued=0 ===

15. Prevention notes

Do not delete dev_r_worker_queue rows — marking status='cancelled' is the correct way to retire a row; deletion breaks audit trails and Grafana panels.
failure_reason is a Postgres enum — only use values defined in the schema: implementation_error, timeout, stale_duplicate, spawn_oom, spawn_ram, mutex_collision, other. The retry-worker uses 'other' for duplicate cancellation.
stale_duplicate and mutex_collision write status=cancelled (not failed) since PR #2091 (2026-06-29). Rows with these reasons in status=failed are pre-fix orphans — bulk-cancel them with the command in § 9.
New job types need allowlist sync — when adding a new job_type to the dispatcher, also add it to the case "$JOB_TYPE" in allowlist in scripts/spawn-worker.sh (line ~51) and deploy to bms-4. Missing this caused 30 failures on 2026-06-29 (PR #2081).
If max_retries is too low for a class of jobs, increase it on the queue row at insertion time (via queue-push.ps1 or the dispatcher API), not by manually resetting retry_count in production.
Monitor via Grafana — the Worker Queue panel shows failed job counts. A rising failed count with retry-worker running normally means jobs are hitting max_retries and need human review; once notified they move to failed_permanently (a separate series) so the FailedJobsAccumulating alert reflects only un-retired failures.
failed_permanently is terminal (#2500) — the retry-worker (Part B) writes it when a retry-exhausted row has been human-notified. Never re-use it for anything else, and do not cancel/delete these rows — they are the audit record of a job that died after full retry.

p24-infra Docs

Explorer

queue-failure-handling