worker-spawn-eperm-vs-oauth-expiry

Playbook: dev_r_worker_queue spawn failures — check OAuth before chasing filesystem permissions

Category: Agent Operations Related issues: #6713 (incident), #6717 (chronic root cause), #6778 (leftover-directory collision), #6780 (30s-timeout cross-account race) Last verified: 2026-09-02

---

What triggers this problem

A dev-issue (or any) job in dev_r_worker_queue fails with status=failed and an error_message referencing a filesystem operation on /tmp/worker-{issue}-{repo}, most commonly:

spawn failed exit=1: rm: cannot remove '/tmp/worker-{N}-radieu-p24-infra': Operation not permitted

This text is a red herring in the common case. scripts/spawn-worker.sh’s pre-clone rm -rf "$WORKDIR" (no || true guard, unlike the two post-run cleanups later in the script) kills the whole script on any non-zero exit — but the actual failure upstream is very often an OAuth auth failure that puts the script into a clone/kill retry loop, not a genuine directory permission problem. Live investigation on 2026-09-01 (#6713) reproduced the manual rm -rf as claude-runner successfully, twice, while the automated queue kept failing identically — proving ownership/sticky-bit was never the actual blocker.

Three distinct spawn-failure signatures — tell them apart before acting

A /tmp/worker-{issue}-{repo}... error text has three different root causes. They look similar in the truncated error_message but have distinct log signatures; misdiagnosing one for another wastes hours (see #6713). Read the full worker log and match:

#	Root cause	Distinct log signature	Fix
1	OAuth expiry / retry loop (the common red herring)	401 OAuth access token has expired, [AUTH] OAuth token revoked detected post-spawn, or No refreshToken in credentials.json in the token-refresh log — repeating	OAuth section below (claude-oauth-reauth.md) — not a filesystem fix
2	#6778 leftover directory from a previous, already-finished attempt	Single-account rm: cannot remove '/tmp/worker-{N}-...': Operation not permitted at the pre-clone cleanup, with no concurrent second-account spawn in the same window	Pre-clone rm -rf "$WORKDIR" || true guard (PR #6779) — already merged
3	#6780 live cross-account race — two accounts attempting the same job in the same dispatch cycle	Spawn for claude-runner-2 ... timed out (>30s) — trying next account immediately followed by Spawn for claude-runner ... failed (exit 6) — ... rm: cannot remove '/tmp/worker-{N}-...': Permission denied, i.e. a timeout on account A then an exit-6 clone failure on account B against the same path, both within one cycle	Account-specific WORKDIR suffix -$(id -un) in spawn-worker.sh (removes the collision class) + SPAWN_SSH_TIMEOUT_S raised 30s→90s in queue-dispatcher-loop.py (stops pre-empting a legitimately-slow-but-healthy clone) — both merged for #6780

The tell that separates #6778 from #6780: #6778 is one account tripping over a stale leftover from a prior run (no second account involved); #6780 is two accounts in the same cycle, the first timing out mid-clone (>30s) and the second colliding on the shared path it couldn’t remove. If you see a timed out (>30s/>90s) line and an exit 6 on a different account in the same ~30s window, it is #6780, not #6778. With the account-specific WORKDIR fix live, this race can no longer occur — a recurrence means the -$(id -un) suffix was reverted or the two accounts resolve to the same Linux user; verify grep 'WORKDIR=' scripts/spawn-worker.sh still carries the suffix.

How to confirm which one it actually is

Do this before touching /tmp at all. Read the full per-issue worker log, not just the truncated dev_r_worker_queue.error_message:

# On the affected host (bms-4 / vps-i1)
cat /var/log/p24-infra-workers/dev-issue-{N}.log

• If you see Failed to authenticate. API Error: 401 OAuth access token has expired and/or [AUTH] OAuth token revoked detected post-spawn ... killing PID=... (#6447) repeating — this is OAuth, not filesystem. Skip straight to the OAuth section below; do not investigate /tmp ownership/sticky-bit/immutable-attribute theories, they will not reproduce.
• If the log shows a clean clone, a real claude session start, and the failure happens later in a way that doesn’t mention auth — then it may genuinely be a filesystem issue (leftover directory owned by a different account after a cross-account retry, /tmp’s sticky bit blocking a same-directory unlink by a different UID — see docs/playbooks/claude-oauth-reauth.md’s multi-account notes for why bms-4 has more than one claude-runner* Linux account). Confirm live with ls -lan, stat, lsattr -d on the actual path before acting — don’t guess-fix.

Second, independent confirmation — always check this regardless:

tail -50 /var/log/claude-token-refresh.log

Look for No refreshToken in credentials.json / HTTP 403 ... treating refresh_token as stale for the account the job actually ran under. If present, the queue outage’s root cause is OAuth, full stop — any filesystem-looking error text downstream of it is noise from the retry loop, not a separate bug to fix.

Step-by-step fix

1. Confirm via the two checks above that this is OAuth, not filesystem.
2. Follow docs/playbooks/claude-oauth-reauth.md in full — check whether a refresh token is present first (Option 0, passive, no browser); if No refreshToken in credentials.json appears, Option 0 cannot fix it, go straight to Option A/B (interactive browser re-auth via tmux — needs a human to open the OAuth URL, authorize, and paste the code back).
3. Verify: runuser -l "$account" -c "claude -p 'say: <host> online' --print" returns model text.
4. Push a fresh usage metric (python3 /opt/p24-infra/scripts/check-sub-usage.py as the affected account) so the ClaudeRunnerTokenExpired alert clears without waiting for the next cron tick. Do not rely on restarting claude-proxy as part of this fix — per root CLAUDE.md §Do NOT it was removed from bms-4 on 2026-06-24; if a unit by that name is still present on a given host, treat that as drift from the documented state, not a dependency this playbook should lean on.
5. Re-dispatch stranded queue rows. Rows that already exhausted retry_count >= max_retries will not auto-retry once the account is healthy again. Check dev_r_worker_queue for the affected github_issue_numbers; if stuck at failed with retries exhausted, force a fresh dispatch row by re-milestoning the GitHub issue (Triage → Queue), which re-fires dispatch-to-queue.yml’s issues.milestoned trigger.

Escalation path if the fix doesn’t work

• If Option 0 and Option A/B from claude-oauth-reauth.md both fail — follow that playbook’s own escalation section (Discord alert + GH issue, Option C is forbidden).
• If the queue rows still won’t re-dispatch after a clean re-milestone — check docs/playbooks/stranded-done-row-redispatch.md for the dedup-guard class of bug (#6137/#6255).

Prevention

• This class of misdiagnosis (chasing a filesystem permission theory when the real cause is upstream OAuth failure corrupting the script’s control flow) wastes significant investigation time. Always read the full worker log before forming a filesystem hypothesis — the dev_r_worker_queue.error_message column only captures the last line of a much longer, more informative log.
• A “missing refreshToken” condition can persist for a long time (13+ days observed, #6717) behind a throttled Discord alert that’s easy to miss. If a spawn failure investigation turns up a stale claude-token-refresh.log WARNING/ERROR older than a few hours, that’s itself worth escalating as its own incident (see #6717), separate from whatever queue rows it happened to break today.