p24-infra Docs

queue-stale-duplicate-cleanup

2 min read

Queue stale_duplicate — cleanup playbook

What triggers this

The queue dispatcher detects a stale_duplicate when a queued job’s GitHub issue was already shipped — either the issue is closed, or a merged PR references it. The queue row is marked status=failed, failure_reason=stale_duplicate and skipped.

Gap (pre-fix): before PR #1909, the GitHub issue kept the ai-dev-queued label and In Progress milestone even though the work had shipped. The issue appeared stuck.

Fixed (post-fix): gh_cleanup_stale_duplicate() removes ai-dev-queued automatically when the dispatcher marks a row as stale_duplicate.

How to confirm

Check for issues still carrying ai-dev-queued after their work shipped:

# Issues with ai-dev-queued that have merged PRs
gh issue list --repo radieu/p24-infra --state open --label "ai-dev-queued" `
  --json number,title,milestone --limit 50 | ConvertFrom-Json | ForEach-Object {
    $n = $_.number
    $prs = gh pr list --repo radieu/p24-infra --state merged `
      --search "closes #$n OR fixes #$n" --json number --limit 1 | ConvertFrom-Json
    if ($prs) { Write-Host "#$n  ← has merged PR, but still has ai-dev-queued" }
}

Or check the DB for stale_duplicate rows where the GH issue still has the label (this should be 0 after the fix is deployed):

SELECT id, github_issue_number, error_message, created_at
FROM dev_r_worker_queue
WHERE failure_reason = 'stale_duplicate'
ORDER BY id DESC
LIMIT 20;

Manual cleanup (if needed after gap period)

$REPO = "radieu/p24-infra"
$issues = gh issue list --repo $REPO --state open --label "ai-dev-queued" `
  --json number --limit 100 | ConvertFrom-Json | ForEach-Object { $_.number }
 
foreach ($n in $issues) {
    $prs = gh pr list --repo $REPO --state merged `
      --search "closes #$n" --json number --limit 1 | ConvertFrom-Json
    if ($prs) {
        gh issue edit $n --remove-label "ai-dev-queued" --repo $REPO | Out-Null
        gh issue edit $n --milestone "Review" --repo $REPO | Out-Null
        Write-Host "Cleaned #$n (merged PR #$($prs[0].number) exists)"
    }
}

Prevention

The gh_cleanup_stale_duplicate(issue_num) call in queue-dispatcher-loop.py removes ai-dev-queued automatically at detection time. No manual cleanup needed for new occurrences after the fix is deployed.

If new ai-dev-queued orphans appear, check:

  1. SUPABASE_SERVICE_ROLE_KEY and GH_TOKEN are valid in the dispatcher env
  2. Dispatcher log: grep 'cleanup stale_duplicate' /var/log/p24-infra-dispatcher.log
  3. The function _gh_remove_label exits silently on 404 (label already removed — OK)
  • Issue #1749 — stale_duplicate guard (pre-dispatch merged-PR check)
  • Issue #1680 — spawn-failure backoff
  • scripts/queue-dispatcher-loop.py → gh_cleanup_stale_duplicate()
  • scripts/tests/test_dispatcher_loop_dedup.py

Graph View

Backlinks