p24-infra Docs

queue-failed-weekly-limit

3 min read

Playbook: Queue failures — Claude weekly usage limit

Trigger

Multiple queue rows with failure_reason=timeout and retry_count=max_retries (permanently failed), worker logs showing:

[spawn] PID=XXXXXX issue=#N ...
You've hit your weekly limit · resets 3am (UTC)

Occurs when a claude-runner-N subscription hits its Claude Code weekly token limit mid-dispatch-wave. The limit resets every Monday at 03:00 UTC.

Confirm

# On bms-4 — check spawn logs for the weekly limit message
grep -r "weekly limit" /var/log/p24-infra-workers/ | tail -20
 
# Check queue rows
# (from Windows dev machine, via Supabase REST or psql on bms-4)
# status=failed + failure_reason=timeout + error_message contains "timed out"

What happens

  1. Claude Code starts (spawn-worker.sh spawned OK, PID assigned)
  2. Claude Code immediately outputs You've hit your weekly limit · resets 3am (UTC) and exits
  3. Before fix (pre-PR #2185): spawn-worker.sh didn’t detect this → returned exit 0 → dispatcher marked job as running → reset_stale_workers marked it timeout after heartbeat gap → retried 2 more times → permanently failed (retry_count=2/2)
  4. After fix (PR #2185): spawn-worker.sh detects the message within 5s post-spawn → exits 5 (subscription blocked) → dispatcher tries next subscription user immediately

Fix (after PR #2185 deployed)

The weekly limit is self-healing — it resets at 03:00 UTC Monday. No manual action needed for the queue; the meta-dispatcher will automatically re-queue permanently-failed issues on the next dispatch cycle (failed rows don’t block re-dispatch).

If you need to act immediately (e.g. mid-week limit hit on both users):

  1. Check which claude-runner-N users have limits hit: 
    # On bms-4
for u in claude-runner claude-runner-2 claude-runner-3; do
  echo "=== $u ==="
  su - $u -s /bin/bash -c "cat ~/.claude/sub-status.json 2>/dev/null | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d.get(\"usage_pct\"), d.get(\"reset_at\"))'" 2>/dev/null || echo "no status"
done
  2. If all users exhausted: wait for reset, no further action (jobs re-dispatch automatically)
  3. Permanently-failed rows (retry=2/2) are NOT re-queued by retry-worker — but the meta-dispatcher re-queues the underlying GitHub issues on its next run (if still In Progress milestone)

Prevention / detection

  • spawn-worker.sh post-spawn check (PR #2185) catches the message and exits 5 → next-user retry
  • check-sub-usage.py runs as cron and writes ~/.claude/sub-status.json — pre-spawn check reads this; may be stale by ≤cron-interval
  • Grafana panel “Worker Queue — Failed by reason” shows spike in timeout failures
  • Mezmo logs: app:spawn-worker level:WARN "weekly_limit_post_spawn"

Trigger: queue rows with job_type=nc-alert-batch, failure_reason=implementation_error, error_message="spawn failed exit=1".

Root cause: spawn-worker.sh job_type validation list was missing nc-alert-batch → exit 1 on every dispatch attempt.

Fix: PR #2185 adds nc-alert-batch to the allowlist. If stale rows remain after deploy:

# Cancel duplicate older rows, reset newest to queued
# Use Supabase PATCH on dev_r_worker_queue — set only the newest to status=queued,retry_count=0
# Cancel duplicates with status=cancelled,failure_reason=stale_duplicate

Graph View

Backlinks