Queue Analyst — Operations Runbook

Service: queue-analyst (scripts/queue-analyst.py) Host: bms-4 Cadence: every 30 min (p24-queue-analyst.timer, OnCalendar=*:0/30) Introduced: issue #2061

What it does

queue-analyst is an autonomous failure investigator for the developer worker queue (dev_r_worker_queue). The mechanical retry-worker (queue-retry-worker.py) used to blindly re-queue implementation_error / timeout failures with exponential backoff — repeat failures hit the same wall 2-3 times, burned Claude subscription slots, and ended on human-action with no explanation.

The analyst runs before the retry cycle for repeat failures:

Selection — status='failed' AND failure_reason IN ('implementation_error','timeout') AND retry_count >= 1 AND (analyst_reviewed_at IS NULL OR analyst_reviewed_at < NOW()-4h), LIMIT 10 per cycle.
Context collection — queue row (all columns), GitHub issue body + last 10 comments, dev_r_agent_sessions.summary, worker log tail (/var/log/p24-infra-workers/{job_type}-{issue_number}.log, 200 lines), and PR state + CI rollup.
Diagnosis — spawns a Claude analyst agent (claude -p, prompt in scripts/queue-analyst-prompt.md) that returns a structured 8-field JSON decision.
Action — writes analyst_notes / analyst_reviewed_at / analyst_action to the row; optionally requeues (status='queued', clears failure_reason / error_message, may switch job_type to continue-issue); posts a human-readable comment on the issue; applies labels; sends a Discord embed.

The retry-worker gate

queue-retry-worker.py Part B skips any row with retry_count >= 1 and analyst_reviewed_at IS NULL, unless it has been failed for more than 4 hours (starvation bypass — covers the analyst service being down). Once the analyst sets analyst_reviewed_at (and optionally requeues), the retry-worker proceeds normally.

Ownership boundary

failure_reason	Owner	Action
`spawn_oom` / `spawn_ram` / `mutex_collision` / `stale_duplicate`	queue-dispatcher-loop.py	unchanged
`timeout` / `implementation_error`, `retry_count=0`	queue-retry-worker.py	fast requeue (unchanged)
`timeout` / `implementation_error`, `retry_count>=1`	queue-analyst → queue-retry-worker	analyst diagnoses first
`failure_reason IS NULL` (pre-042 orphans)	meta-dispatcher `requeueFailed()`	safety-net requeue only

Fast-path rule: spawn-fail server rerouting

When the analyst sees a spawn failed exit=255 with no session and no log (the worker never started), it applies a server rerouting rule in the fast-path classifier (0 tokens):

`retry_count` when analyst runs	Action
`1` (first repeat — initial + 1 retry)	Requeue with `server_preference` set to the other known node. Map: `bms-4 → dev-laptop`, `dev-laptop → bms-4`. Unknown server falls back to `bms-4`.
`≥ 2` (reroute was already tried)	Escalate with `human-action` label — persistent SSH/spawn issue on both nodes.

The server_preference column nudges the dispatcher toward that node on the next cycle. If that node is busy or disabled the dispatcher falls back to any available node.

Why rerouting instead of immediate escalation? bms-3 was disabled 2026-06-30 after all its spawn jobs failed with exit=255. Jobs stacked up as human-action requiring manual re-queue. With rerouting the first spawn fail auto-recovers to the healthy node; only persistent failures (retry_count≥2) reach human review.

Analyst output schema

{
  "diagnosis": "string (max 200 chars)",
  "category": "transient_infra | spec_too_large | spec_ambiguous | ci_failure | pr_conflict | dependency_blocked | repeated_crash | stale",
  "action": "requeued | fixed_and_requeued | escalated | needs_clarification | closed_stale | skipped",
  "fix_applied": "string | null",
  "requeue": true,
  "requeue_job_type": "dev-issue | continue-issue | infra-task",
  "comment": "markdown posted to GH issue | null",
  "escalate": false
}

Stored verbatim in dev_r_worker_queue.analyst_notes (JSONB).

Labels applied

Label	When
`ai-analysis-done`	every reviewed row (informational)
`needs-clarification`	`action = needs_clarification`
`ai-blocked`	`category = dependency_blocked`
`human-action`	`escalate = true` or any fallback escalation

Checking status

# Timer scheduling + last/next run
ssh bms-4 systemctl status p24-queue-analyst.timer
ssh bms-4 systemctl list-timers p24-queue-analyst.timer
 
# Last run logs
ssh bms-4 journalctl -u p24-queue-analyst.service -n 100 --no-pager
 
# Follow live
ssh bms-4 journalctl -u p24-queue-analyst.service -f

Force a re-analysis cycle now

ssh bms-4 systemctl start p24-queue-analyst.service
ssh bms-4 journalctl -u p24-queue-analyst.service -n 50 --no-pager

Re-trigger analysis on a specific row

Clear analyst_reviewed_at so the next cycle re-selects it (PostgREST, service role key — extract silently, never echo):

SUPA_KEY=$(grep '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"')
curl -s -X PATCH \
  "$SUPA_URL/rest/v1/dev_r_worker_queue?id=eq.<ROW_ID>" \
  -H "apikey: $SUPA_KEY" -H "Authorization: Bearer $SUPA_KEY" \
  -H "Content-Type: application/json" \
  -d '{"analyst_reviewed_at": null, "analyst_action": null}'
unset SUPA_KEY

($SUPA_URL = the DISPATCHER_SUPA_URL value in /opt/p24-infra/bms-4/.env.)

Troubleshooting

Symptom	Likely cause	Fix
`No unreviewed repeat-failure rows this cycle` every run	No repeat failures (healthy), or all already reviewed	Normal — nothing to do
Discord: skipped cycle (Claude limit/auth)	claude-runner hit monthly spend limit or OAuth expired on bms-4	Raise the limit at claude.ai/settings/usage, or re-auth (see #981 / `reauth-bms4.py`). Rows are NOT marked reviewed; they retry next cycle
Log tail always `no log available`	Worker log dir not readable by `claude-runner`, or logs rotated	`ssh bms-4 'ls -la /var/log/p24-infra-workers && sudo chgrp -R claude-runner /var/log/p24-infra-workers'`. Analyst still works without log context
`failed_unreviewed_old` Grafana alert firing	Analyst service down > 4h; retry-worker bypass now active	Check the timer/service status above; restart with `systemctl start p24-queue-analyst.timer`
`analyst_action_escalated_24h` Grafana alert firing	A human has not acted on an analyst escalation for 24h	Review the issue tagged `human-action` + `ai-analysis-done`; act on the analyst’s `comment`. The gauge counts only escalations whose GitHub issue is still open + `human-action` (#2458) — closing the issue or removing the label clears the count on the next 60s scrape (no queue-row edit needed)
Analyst returns invalid JSON repeatedly	Prompt drift / model change	Inspect `analyst_notes.raw_response` on the escalated row; iterate `scripts/queue-analyst-prompt.md` (no Python redeploy needed)
A new `[alert] AnalystActionEscalated24h` issue appears most days, each closed as “not a code defect”	The gauge was flapping to a false 0 (#4330): a partial GitHub page or a failed Supabase fetch published an under-count, resolving the alert → the auto-closer closed the issue → the gauge recovered → the alert re-fired → `find_open_issue()` (which searches `is:open` only) filed a brand-new issue. Produced 20 duplicates in 10 days	Fixed in #4330 — `_gh_open_human_action_issues()` now raises on an incomplete walk and the escalated-rows fetch uses `_get_rows_strict()`, so the gauge holds its last good value; the alert expr is smoothed with `max_over_time(...[1h])`. If duplicates reappear, check `queue_exporter_gh_scrape_errors_total{operation=~"escalated.*"}` — a rising counter means the gauge is degraded and holding, not healthy

Reading the gauge when it looks wrong. p24_worker_queue_analyst_escalated_stale is intentionally held, never zeroed, when its inputs cannot be verified. A flat 0 alongside a rising queue_exporter_gh_scrape_errors_total means “could not determine”, not “nothing pending”. To confirm the true count independently, intersect the escalated queue rows with the open human-action set:
gh issue list --repo radieu/p24-infra --label human-action --state open --limit 200 \
  --json number | jq -r '.[].number' | sort -n > /tmp/open_ha.txt
# then compare against dev_r_worker_queue rows with
# analyst_action='escalated' AND analyst_reviewed_at < NOW()-24h

Escalation path if the analyst is stuck

Confirm the service is enabled: ssh bms-4 systemctl is-enabled p24-queue-analyst.timer.
Confirm claude -p auth on bms-4: ssh bms-4 'sudo -u claude-runner HOME=/home/claude-runner claude -p "say ok" --print'.
The retry-worker 4-hour bypass guarantees the queue never permanently stalls — repeat failures resume blind retry after 4h even with the analyst down.
If the analyst itself is the problem, disable it without affecting the queue: ssh bms-4 systemctl disable --now p24-queue-analyst.timer. Re-enable once fixed.

Deployment

Deployed via Ansible (ansible/roles/bms4-workers) — see the role for the unit install + systemctl enable --now tasks. Manual deploy:

scp deploy/p24-queue-analyst.service deploy/p24-queue-analyst.timer bms-4:/etc/systemd/system/
ssh bms-4 'systemctl daemon-reload && systemctl enable --now p24-queue-analyst.timer'

Compliance

Registered in dev_r_ai_systems (limited risk — internal queue telemetry, no personal data, human override via human-action) and documented in docs/eu-ai-act-compliance.md §4.
Error notification standard: every crash sends a red Discord embed via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL; persistent failures surface via the failed_unreviewed_old Grafana alert.
Registered in dev_r_services (queue infrastructure).

p24-infra Docs

Explorer

queue-analyst-operations