p24-infra Docs

stranded-done-row-redispatch

4 min read

Stranded done queue row — diagnose & unblock re-dispatch

What this is

A GitHub issue that shows as “still queued” (sits in the Queue / In Progress milestone) but on which no further work ever happens, because a prior dev_r_worker_queue row with status='done' blocks every re-dispatch — even though the worker that produced that done row shipped nothing (no merged PR, no artifact).

Before #6137 (#6019 option 2) this was a permanent dead-end: re-milestoning the issue fired issues.milestoned, but the meta-dispatcher dedup guard silently HTTP-200-skipped the insert because done was an unconditional blocking state. The only escape was manual UPDATE dev_r_worker_queue SET status=... WHERE id=... DB surgery (done ad hoc for #2709).

The fix (post-#6137)

done is no longer a blanket blocker. A done row now blocks re-dispatch only when:

  1. a merged closing PR references the issue (has_merged_pr() — the #1383 backstop; the check recognises Closes/Fixes/Resolves/Implements #N), or
  2. the issue carries the human-action label (a human decision is pending).

Otherwise the re-dispatch is allowed and an audit-trail comment (<!-- meta-dispatcher:dedup-redispatch-done -->) is posted so the re-dispatch is not a silent black box. The refined logic lives in three mirrored places (keep them in sync):

  • infra-src/meta-dispatcher/src/dedup.ts → shouldSkipDispatchPure / BLOCKING_STATES
  • scripts/queue_dispatcher.py → should_skip_dispatch / BLOCKING_STATES
  • .claude/commands/issues-queue.md → inline PowerShell dedup guard (steps 1 + 2b)

This is a distinct, complementary fix to the dependency-based auto-requeue (.github/workflows/requeue-blocked-on-dependency.yml, shipped for #6019 option 1) — that handles issues waiting on sibling issues; this handles done-but-unshipped rows.

Symptoms

  • An open issue sits in Queue / In Progress for days with no new worker activity.
  • It has a done dev_r_worker_queue row from days ago, no merged closing PR, and no human-action label.
  • Re-milestoning it to Triage does nothing (pre-#6137: silent 200; the milestone never advances).

Diagnose

Find candidate stranded rows — done rows whose issue is still open:

SELECT id, github_issue_number, job_type, status, completed_at, result_summary
FROM dev_r_worker_queue
WHERE status = 'done'
  AND repo = 'radieu/p24-infra'
ORDER BY completed_at DESC
LIMIT 50;

For a specific issue #N, confirm the three conditions that mean “shipped nothing”:

# 1. Latest queue row for the issue (is it `done`?)
#    (via the queue_dispatcher CLI — same decision the dispatcher uses)
python3 scripts/queue_dispatcher.py should-dispatch N radieu/p24-infra
#    exit 0  -> dispatch allowed (re-dispatch will proceed)
#    exit 10 -> skip (reason printed: merged PR OR human-action OR in-flight)
 
# 2. Any merged closing PR?  (empty output = none)
gh pr list --repo radieu/p24-infra --state merged --search "N in:body" \
  --json number,title,body --jq '.[] | select(.body | test("(?i)(closes|fixes|resolves|implements)\\s*:?\\s*#'"N"'\\b")) | .number'
 
# 3. Does it carry human-action?
gh issue view N --repo radieu/p24-infra --json labels --jq '[.labels[].name] | index("human-action") != null'

If the CLI prints DISPATCH (exit 0), the fix will re-dispatch on the next trigger — no manual action needed. If it prints SKIP, the reason tells you which real condition holds (genuinely shipped, or awaiting a human) — do not force a re-dispatch; resolve the real condition instead.

Unblock (only if the automated path is unavailable)

Preferred: re-milestone the issue to Triage so issues.milestoned re-triggers the meta-dispatcher. Post-#6137 the dedup guard allows the re-dispatch and posts the dedup-redispatch-done breadcrumb automatically.

gh issue edit N --repo radieu/p24-infra --milestone Triage

Manual re-queue (if the milestone trigger is not firing — e.g. the CF Worker is down): POST to /queue-issue, which runs the same dedup guard and posts the audit comment:

# QUEUE_API_URL / QUEUE_API_KEY live in secrets/worker-queue.env.sops (and Vercel/CI envs).
# Extract the key silently — never echo it.
KEY=$(sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops \
  | grep '^QUEUE_API_KEY=' | cut -d= -f2-)
curl -sf -X POST "https://p24-meta-dispatcher.radieu.workers.dev/queue-issue" \
  -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" \
  -d '{"issue_number": N, "trigger_event": "issues.milestoned"}'
unset KEY

Last-resort DB surgery (only when the dispatcher genuinely cannot be used) — mark the stranded done row so the newest-row dedup no longer matches it, then re-milestone:

-- Inspect first; never blind-UPDATE. Confirm this row shipped nothing before touching it.
UPDATE dev_r_worker_queue
SET status = 'cancelled', failure_reason = 'stale_duplicate',
    error_message = 'manually unblocked — done row shipped nothing (see #6137)'
WHERE id = <ROW_ID> AND status = 'done';

Do NOT

  • Force a re-dispatch when the CLI/guard reports human-action — that label exists because a human decision is pending; re-dispatching just loops. Resolve the human-action item.
  • Force a re-dispatch when a merged closing PR exists — the work shipped; the done block is correct (the #1383 backstop). Close the issue instead.
  • Edit the deployed skill copy at ~/.claude/commands/issues-queue.md directly — change the in-repo .claude/commands/issues-queue.md and let it sync, keeping all three mirrors aligned.

Graph View