claude-oauth-reauth

Playbook: Claude OAuth Token Re-Authentication on VPS Agents

Category: Agent Operations
Severity when triggered: P1 (auth expired, agents dark) / P2 (expiry imminent)
Affected hosts: vps-i1 (217.154.82.162), bms-4 (54.36.123.110)
Related issue: #981
Last verified: 2026-06-23

---

What Triggers This Problem

Claude Code on VPS agents (vps-i1, bms-4) authenticates via OAuth subscription (Claude Max), not via ANTHROPIC_API_KEY. The OAuth access token has a TTL of approximately 8–12 hours. The refresh token lasts months.

Expiry happens when:

• The claude-runner user has not run any Claude Code session for >12 hours (no auto-refresh)
• The refresh token is revoked (very rare; requires manual deauthorization on claude.ai)
• Credentials file is deleted or corrupted

Observed impact (2026-06-21): bms-4 token expired at 01:00 UTC. The hourly triage agent was dark for ~30 hours with 401 failures. No alert fired because no health check existed for claude-runner auth state.

Observed impact (2026-10-08, #7575): claude-runner-2 had a non-renewable token (no refreshToken in .credentials.json) while claude-runner was at ≥95 % usage. The dispatcher spawned the dead account first anyway, the worker died in ~3 s (err_1 runtime=3s), and the row’s retries were spent in seconds — with no fleet-level alert.

How the dispatcher now routes around a dead/revoked account (#7667)

The bms-4 dispatcher (scripts/queue-dispatcher-loop.py) runs a pre-spawn account-health gate before launching a worker — it no longer learns an account is dead only after a ~3 s failed launch:

• Dead-token accounts are skipped up front. An account with no ADR-007 token mode and no refreshToken and no still-valid access token is not spawned against. If every account on the host is blocked (dead / recently auth-revoked / depleted), the job is requeued with a 15-min backoff and no retry burn (spawn_worker exit 8) instead of failing.
• auth_revoked_at tracking. When an account produces an exit-7 (OAuth-revoked) spawn, the dispatcher stamps auth_revoked_at in dev_r_server_capacity.claude_accounts and skips that account for 30 min — so a revoked token is not re-spawned against every 2-min cycle. A successful spawn clears both auth_revoked_at and depleted, so a passive refresh or a human re-auth is picked up automatically on the next cycle (no manual DB edit needed).
• Fleet-stopped alert. When no host can run any job (all accounts blocked and the HEALTH-GATE leaves no eligible host), the dispatcher files one deduped bug + fleet-stopped GH issue and a Discord ping. The issue body names the stale-gated hosts. This is your signal that the whole fleet is down — start with Option 0 below on the affected host. Close the issue once dispatch recovers.

Full mechanics: docs/queue-dispatcher-operations.md §Pre-spawn account-health gate.

---

How to Confirm the Problem

Quick check (from local Windows machine)

# vps-i1
ssh root@217.154.82.162 "runuser -l claude-runner -c 'claude -p \"say-ok\" --print 2>&1 | head -3'"
 
# bms-4
ssh root@54.36.123.110 "runuser -l claude-runner -c 'claude -p \"say-ok\" --print 2>&1 | head -3'"

Expected (auth valid): model response text
Expected (auth expired): 401 Unauthorized or Authentication required or empty output

Read token expiry from credentials file

# On the VPS (run as root, pipe to avoid printing the token value)
python3 -c "
import json, datetime
c = json.load(open('/home/claude-runner/.claude/.credentials.json'))
exp = c.get('expiresAt', 0) / 1000
dt = datetime.datetime.utcfromtimestamp(exp)
now = datetime.datetime.utcnow()
diff = (dt - now).total_seconds()
print(f'Expires: {dt.isoformat()}Z')
print(f'Seconds until expiry: {int(diff)}')
print(f'Status: {\"VALID\" if diff > 0 else \"EXPIRED\"}')"

---

Step-by-Step Fix

Option 0: Passive refresh — TRY THIS FIRST (no browser, no human)

When the credentials file still has a valid refreshToken, this is the whole fix. The OAuth access token is short-lived (~8–12h) and lapses whenever the account runs no Claude Code session for >12h, but the refresh token lasts months. Claude Code silently exchanges the refresh token for a new access token on the next invocation — so a single claude -p as the affected account restores auth with no browser, no Playwright, no 2FA.

Confirm a refresh token is present (read-only, prints no secret) and that only the access token lapsed, then trigger the refresh. Substitute the affected account’s Linux user (claude-runner → radieu, claude-runner-2 → ecotrans):

# On the affected host as root (bms-4 / vps-i1). ACCT = claude-runner | claude-runner-2
ACCT=claude-runner-2
python3 - "$ACCT" <<'PY'
import json, datetime, sys
c = json.load(open(f"/home/{sys.argv[1]}/.claude/.credentials.json"))
o = c.get("claudeAiOauth", c)
diff = o.get("expiresAt", 0)/1000 - datetime.datetime.utcnow().timestamp()
print("access token:", "VALID" if diff > 0 else "EXPIRED", "| has refresh token:", bool(o.get("refreshToken")))
PY
 
# If "has refresh token: True", trigger the auto-refresh — this alone fixes it:
timeout 90 runuser -l "$ACCT" -c 'claude -p "say-ok" --print'   # expect: OK / model text
 
# Push the fresh metric so ClaudeRunnerTokenExpired clears without waiting for the 10-min cron:
runuser -l "$ACCT" -c 'python3 /opt/p24-infra/scripts/check-sub-usage.py'

If claude -p returns OK, re-check expiresAt — it now sits ~8h in the future and you are done. Only if claude -p still returns 401 / Authentication required (the refresh token itself was revoked) do you need the full browser flow below.

Verified 2026-07-14 (#4157): ecotrans/claude-runner-2 on bms-4 was 6 days dark; the passive refresh restored it in one command — no browser needed.

Option 0.5: Telegram-driven re-auth — FAST PATH when a browser is available but a root terminal is not (#6650)

When Option 0 fails (the refresh token itself was revoked, so a real browser re-auth is needed) but you are not at a workstation with an SSH key, the Telegram bridge can drive the whole manual Option A/B procedure for you — everything except the one irreducibly human step (opening the URL and clicking Authorize). This is the fast path for the recurring re-auth latency behind #6613 / #6071 / #5290.

From the authorized Telegram chat (the claude-session-manager bridge on bms-4):

1. /auth-worker claude-runner-2 — the bridge SSHes to root@localhost, starts claude auth login in a tmux session auth_<account>, scrapes the OAuth URL, and replies with only that URL. (Allowed accounts: claude-runner, claude-runner-2.)
2. Open the URL in any browser, click Authorize, copy the code. (This is the OAuth human anchor — the same browser step as Option A/B; it is never automated.)
3. /auth-code <code> — the bridge pastes the code, verifies with claude -p "say ok", replies OK / FAIL only, and cleans up the tmux session. If two re-auths are in progress, disambiguate with /auth-code <account> <code>.

This wraps the exact tmux + URL-scrape + code-inject steps of Options A/B below (it does not replace them — the manual steps remain the reference and the fallback when the bridge is down). Scope and security model: docs/telegram-claude-bot-operations.md §Worker re-auth — account allowlist, shell-injection guard, single audited root chokepoint, URL-only / OK-FAIL secret-safety. bms-4 only in the first cut; vps-i1 uses the /proc-based flow in Option B and is a tracked follow-up. This does not fix the refresh-token rotation race (#5290) — it only shortens time-to-reauth.

Option A: Playwright Autonomous Re-Auth (browser re-auth — only if the refresh token is revoked)

This is the intended path for normal token expiry. The Playwright approach navigates the OAuth flow headlessly using a pre-authenticated browser profile.

Prerequisites:

• Playwright MCP tools available in the current Claude Code session (local Windows machine)
• Browser profile at C:\Users\konar\AppData\Local\ms-playwright\ with claude.ai session cookies still valid
• SSH access to the affected VPS

Steps:

1. Start claude auth login on the VPS inside a tmux session

   For bms-4:

   ssh root@54.36.123.110
   tmux kill-session -t claude_auth 2>/dev/null || true
   tmux new-session -d -s claude_auth -x 220 -y 50
   tmux send-keys -t claude_auth "runuser -l claude-runner -c 'claude auth login'" Enter

   For vps-i1:

   ssh root@217.154.82.162
   # vps-i1 uses a process-based approach (no tmux required, see reauth-ionos.py)

2. Capture the OAuth URL from tmux output

   # Poll until URL appears (up to 30s)
   for i in $(seq 30); do
     pane=$(tmux capture-pane -t claude_auth -p 2>/dev/null)
     url=$(echo "$pane" | grep -o 'https://[^ ]*oauth[^ ]*')
     [ -n "$url" ] && { echo "URL: $url"; break; }
     sleep 1
   done

3. Navigate Playwright to the OAuth URL

   Using Playwright MCP tool from the local session:

   • Navigate to the captured URL
   • The browser opens with the user already logged into claude.ai
   • Click the “Authorize” button
   • Capture the redirect URL — it contains ?code=<value>

4. Inject the authorization code into the tmux session

   # Extract code from redirect URL (displayed in Playwright browser address bar)
   CODE="<paste-code-from-redirect>"
    
   # bms-4:
   ssh root@54.36.123.110 "tmux send-keys -t claude_auth '$CODE' Enter"
    
   # Wait 8s for claude auth login to complete
   sleep 8
    
   # Verify
   ssh root@54.36.123.110 "runuser -l claude-runner -c 'claude -p \"say-ok\" --print 2>&1 | head -3'"

5. Restart dependent services

   # On bms-4 — restart n8n workers that call claude via n8n nodes (if applicable)
   ssh root@54.36.123.110 "cd /opt/p24-infra/bms-4 && docker compose restart n8n-worker 2>/dev/null || true"
    
   # On vps-i1 — nothing extra needed (GitHub Actions runner picks up fresh credentials per run)

6. Clean up tmux session

   ssh root@54.36.123.110 "tmux kill-session -t claude_auth 2>/dev/null || true"

---

Option B: Manual Re-Auth via Local Python Scripts

Use this when Playwright is unavailable or Option A fails twice.

Prerequisites:

• Python with paramiko installed locally (pip install paramiko)
• Local SSH key at C:\Users\konar\.ssh\id_ed25519
• A browser window logged into claude.ai

For bms-4 (54.36.123.110):

python d:\tmp\reauth-bms4.py

What the script does:

1. SSH into bms-4 as root
2. Creates a fresh tmux session named claude_auth
3. Runs claude auth login as claude-runner inside tmux
4. Polls tmux pane output until the OAuth URL appears (up to 30s)
5. Prints the URL and prompts you to paste the authorization code
6. Injects the code via tmux send-keys
7. Waits 8 seconds and verifies auth with claude -p 'say: bms4 online'
8. Restarts claude-proxy systemd service (if still running)
9. Cleans up the tmux session

What you do:

1. Run the script
2. When the OAuth URL appears, open it in your browser
3. Click “Authorize” on the claude.ai OAuth page
4. Copy the code from the redirect URL (format: code=XXXX)
5. Paste it at the script prompt

For vps-i1 (217.154.82.162):

python d:\tmp\reauth-ionos.py

What the script does:

1. SSH into vps-i1 as root
2. Writes a Python helper to /tmp/auth_helper.py
3. Runs claude auth login as claude-runner via subprocess (no tmux — uses /proc/<pid>/fd/4)
4. Polls /tmp/auth_out until the OAuth URL appears
5. Prints the URL and prompts for the code
6. Writes the code to /tmp/the_code.txt and pipes it to the helper process stdin
7. Verifies auth with claude -p hello

What you do: same as bms-4 — open URL, authorize, paste code.

---

Option C: Copy Credentials from Local — FORBIDDEN, do not use

Looking for a way to avoid per-host re-auth? Use the per-account setup-token instead: claude-shared-oauth-token.md (ADR 007). It is a non-refreshing token, so it does not have the race below.

This option is banned, not “emergency only.” Copying one host’s .credentials.json to a second host is the documented mechanism of the recurring multi-host OAuth refresh-token race analyzed in docs/adr/002-claude-account-topology.md (see §“The mechanism — what actually races”, and the copy-vector citation of this exact section). Distributing one refresh token across hosts is what turns an isolated, per-grant token rotation into a fleet-wide race: whichever host refreshes next presents a token the server has already rotated away, and the sibling host gets a 403 with a locally-expired access token — the ”🔴 refresh_token stale” failure mode (#4178, #4394, #4401, #4326, #3803).

# DO NOT RUN — kept here only to show what is now forbidden:
# scp C:\Users\konar\.claude\.credentials.json root@<host>:/tmp/creds.json
# ssh root@<host> "cp /tmp/creds.json /home/claude-runner/.claude/.credentials.json ..."

If you find yourself reaching for this because Option A/B are failing or feel slow: stop. Use Option 0 (passive refresh) first — it requires no browser and fixes the vast majority of cases in one command. If Option 0 doesn’t clear it, use Option A (per-host browser re-auth) — every host gets its own independent OAuth grant via its own claude auth login, which is what keeps per-host rotation from colliding in the first place. Never bridge a gap by importing another host’s token.

---

Escalation Path When Playwright Fails

If Playwright-based auto-reauth fails twice (two consecutive attempts within 10 minutes), escalate immediately:

1. Fire Discord alert

curl -s -X POST "$P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -d '{
    "embeds": [{
      "title": "CRITICAL: claude-runner auth expired on <host> — manual re-auth required",
      "color": 15158332,
      "description": "Auto-reauth failed twice.\n\nOAuth URL: <URL>\nHost: <host>\ntmux session: claude_auth\n\nRun: python d:\\\\tmp\\\\reauth-<host>.py",
      "url": "https://github.com/radieu/p24-infra/issues"
    }]
  }'

2. Create GitHub issue

gh issue create --repo radieu/p24-infra \
  --title "CRITICAL: claude-runner auth expired on <host> — manual re-auth required" \
  --label "human-action" \
  --label "bug" \
  --milestone "Triage" \
  --body "## Claude OAuth token expired — auto-reauth failed
 
**Host:** <host>
**Time detected:** $(date -u +%Y-%m-%dT%H:%M:%SZ)
**Auto-reauth attempts:** 2 (both failed)
 
### OAuth URL (open in browser)
\`\`\`
<URL from claude auth login>
\`\`\`
 
### Re-auth steps
1. \`python d:\\tmp\\reauth-<host>.py\` on local Windows machine
2. Open the URL in browser, authorize, paste code
3. Verify: \`ssh root@<ip> \"runuser -l claude-runner -c 'claude -p say-ok --print'\"\`
4. Close this issue
 
### Tmux session state
\`\`\`
<paste tmux capture-pane output>
\`\`\`"

3. Check Telegram notification

If the Telegram Claude bot is configured (see docs/telegram-claude-bot-operations.md), the hourly triage agent will also send a P1 Telegram notification when it detects claude -p say-ok returns 401.

---

Monitoring: Detecting Expiry Before It Happens

Manual expiry check (copy-paste ready)

# Run on each VPS agent host to see token status
python3 -c "
import json, datetime, sys
try:
    c = json.load(open('/home/claude-runner/.claude/.credentials.json'))
    exp = c.get('expiresAt', 0) / 1000
    dt = datetime.datetime.utcfromtimestamp(exp)
    now = datetime.datetime.utcnow()
    diff = (dt - now).total_seconds()
    print(f'Expires: {dt.isoformat()}Z ({int(diff/3600)}h {int((diff%3600)/60)}m from now)')
    sys.exit(0 if diff > 0 else 1)
except Exception as e:
    print(f'ERROR reading credentials: {e}')
    sys.exit(2)
"

Prometheus metric + alerts (IMPLEMENTED — #3420)

Shipped 2026-07-09. The metric is pushed by scripts/check-sub-usage.py (not credential-exporter) — that cron already runs per Linux account (claude-runner → radieu, claude-runner-2 → ecotrans) every 10 min and already reads ~/.claude/.credentials.json. It reads claudeAiOauth.expiresAt (epoch ms), computes seconds-to-expiry, and pushes it to the vps-i1 Pushgateway (job claude-sub-usage-<account>) alongside the existing sub-usage gauges:

# HELP claude_runner_token_expires_seconds Seconds until claude-runner OAuth token expires (negative = expired)
# TYPE claude_runner_token_expires_seconds gauge
claude_runner_token_expires_seconds{account="radieu",machine="bms-4"} 28800
claude_runner_token_expires_seconds{account="ecotrans",machine="bms-4"} 14400

The machine label defaults to bms-4; a vps-i1 deployment sets CLAUDE_SUB_MACHINE=vps-i1 in the cron env. The expiry is read straight from the credentials file, so an already-expired token still reports (as a negative value) even when the API returns HTTP 401 — which is exactly the silent failure mode this closes.

Alert rules live in monitoring/prometheus/rules/claude-token-expiry.yml (loaded by Prometheus via rule_files: /etc/prometheus/rules/*.yml). The warning is bounded > 0 so it never overlaps the expired critical:

- alert: ClaudeRunnerTokenExpiringSoon
  expr: |
    (claude_runner_token_expires_seconds > 0 and claude_runner_token_expires_seconds < 7200)
      and on (job) (time() - push_time_seconds < 1500)
  for: 5m
  labels: { severity: warning }
  # -> warning-digest route (email + n8n/Discord)
 
- alert: ClaudeRunnerTokenExpired
  expr: |
    claude_runner_token_expires_seconds <= 0
      and on (job) (time() - push_time_seconds < 1500)
  for: 15m          # #4202 — was 1m; must outlast one 10-min push cycle
  labels: { severity: critical }
  # -> critical route (email + n8n/Discord)
 
- alert: ClaudeRunnerTokenGaugeStale
  expr: time() - max by (job) (push_time_seconds{job=~"claude-sub-usage-.*"}) > 1800
  for: 15m
  labels: { severity: warning }
  # -> warning-digest route

Why the push_time_seconds join (#4323). The Pushgateway retains the last pushed value forever, and check-sub-usage.py pushes best-effort over SSH with failures swallowed. If pushing stops, the last sample — possibly a negative one caught mid-refresh — is scraped as fresh indefinitely, so for: at any duration eventually elapses and the alert fires against a perfectly valid token (#4321 fired 122 min while expiresAt was ~68 min in the future). The freshness join makes both rules ignore samples older than 1500s (2.5 push cycles).

Consequence: a stale gauge suppresses token-expiry alerting for that account. ClaudeRunnerTokenGaugeStale exists to surface that, and ClaudeAuthSyntheticFailing — an independent real-auth ping on a separate push job — remains the authoritative broken-auth backstop. If you see the stale warning, fix the push path; do not assume the token is fine.

Routing is automatic — no Alertmanager change was needed; the existing severity=warning / severity=critical routes carry these to email + the n8n/Discord bridge.

Missing-refreshToken + “host cannot run jobs” alerts (IMPLEMENTED — #7666)

The ClaudeRunnerTokenExpired family above reads the access-token lifetime. It does not catch the distinct, more dangerous condition where credentials.json has no refreshToken at all — headless refresh is then impossible and the account is on a countdown to a hard outage. The only pre-#7666 signal for that was refresh-claude-token.py’s check_missing_refresh_token() escalation (#6717), which is rate-limited to once per 6h (TERMINAL_ALERT_INTERVAL_H). During the 2026-10-08 worker-queue outage (#7575) claude-runner-2 had no refreshToken for ~169h and that rate-limited page was missed. #7666 adds two independent Prometheus/Discord signals:

1. p4_claude_rt_present gauge → ClaudeRefreshTokenMissing / ClaudeRefreshTokenGaugeStale. refresh-claude-token.py now pushes p4_claude_rt_present on every 15-min tick (un-throttled, job claude-rt-present, grouping key /machine/<m>/account/<user>): 1 when a refreshToken is present, 0 when absent.

p4_claude_rt_present{machine="bms-4",account="claude-runner-2"} 0   # refreshToken MISSING

- alert: ClaudeRefreshTokenMissing      # severity: critical -> critical route (Discord+email, repeat 1h)
  expr: |
    p4_claude_rt_present == 0
      and on (machine, account) (time() - push_time_seconds{job="claude-rt-present"} < 1500)
  for: 15m                               # fires within ~15 min of onset; self-resolves a transient
 
- alert: ClaudeRefreshTokenGaugeStale   # severity: warning — the push-freshness gate's safety net
  expr: time() - max by (machine, account) (push_time_seconds{job="claude-rt-present"}) > 2700
  for: 15m

Because severity: critical routes to the critical receiver (repeat_interval: 1h), this re-notifies hourly until cleared — the short cadence the 6h script escalation lacked. The push_time_seconds join is the same anti-stale guard as #4323 (a dead refresh cron would otherwise pin the gauge at 0 and page forever); ClaudeRefreshTokenGaugeStale makes that suppression visible.

Response: this is a genuine missing-token condition — go straight to an interactive re-auth (Option A/B above). Option 0 (passive refresh) cannot help: there is no refresh token to use.

2. p24_spawn_oauth_expired_failures_30m gauge → ClaudeWorkerOAuthSpawnFailures. A complementary signal that reads real spawn outcomes, not the credential file. queue-exporter counts dev_r_worker_queue rows hard-failed with failure_reason=spawn_oauth_expired in the last 30 min, by server_node. A row reaches that reason only when spawn-worker.sh exits 7 — OAuth revoked/expired on every account on the host (#6447) — so any nonzero value means that host dropped a job and can start no Claude worker.

- alert: ClaudeWorkerOAuthSpawnFailures  # severity: critical
  expr: p24_spawn_oauth_expired_failures_30m > 0
  for: 0m

The two are intentionally redundant from different angles: #1 warns before the outage (token cannot be refreshed), #2 confirms the outage is happening (spawns are failing). Both point here. Routing is automatic — no alertmanager.yml change was needed.

Account-down (unified_status=unknown) + low-headroom alerts (IMPLEMENTED — #7675)

The families above read the credential file (access-token lifetime, refreshToken presence) or spawn outcomes. They missed one more shape: check-sub-usage.py reaching the API and getting a response without an anthropic-ratelimit-unified-status header (it then defaults to "unknown") while the token is expired. That is exactly what claude-runner-2 did for ~3 days (2026-10-05..08): its sub-status.json carried unified_status=unknown, usage_pct=0.0, token_expires_seconds<0. Two things went wrong and are both fixed:

1. The dispatcher ranked a down account as the freest one. sub_headroom.headroom_from_sub_status() turned usage_pct=0.0 into headroom 100.0. It now returns None (unavailable → sorts last among non-depleted) when unified_status=="unknown". Keyed on unified_status only — a negative token_expires_seconds with unified_status=="allowed" is the #5884 self-healing steady state and must stay fully available, so the expiry sign is deliberately not used here. The same guard is mirrored in queue-dispatcher-loop.py’s inline fallback (#6132 shared/fallback pair).

2. No alert fired on the unknown state itself. p4_claude_sub_blocked was 1, but nothing alerts on it and it conflates unknown (down) with rejected (normal depletion). check-sub-usage.py now emits a dedicated per-account gauge, pushed on both the full and token-only paths (so it survives the Pushgateway group-replace during an outage):

p4_claude_sub_status_unknown{account="ecotrans",machine="bms-4"} 1   # account DOWN / unconfirmable

- alert: ClaudeSubStatusUnknown     # severity: critical — down >=15 min, push-fresh + enabled-host
  expr: |
    ( ( p4_claude_sub_status_unknown == 1
          and on (job) (time() - push_time_seconds < 1500) )
      and on (machine)
        label_replace(p24_server_max_workers_prime, "machine", "$1", "server_label", "(.+)") )
  for: 15m

Low-headroom early-warning (binding-window p4_claude_sub_usage_pct; with only 2 subscriptions, thin headroom is the early signal the fleet is one depletion from a single-account cliff):

- alert: ClaudeSubHeadroomLow       # severity: info  (headroom <20% ⟺ usage 80–92%), for: 30m
- alert: ClaudeSubHeadroomVeryLow   # severity: warning (headroom <8% ⟺ usage >92%),  for: 15m

The info tier is routed to the batched, non-paging warning-digest (added to alertmanager.yml.tpl by #7675). ClaudeSubStatusUnknown is not gated on p4_claude_auth_valid (unlike ClaudeRunnerTokenExpired): unified_status=unknown already means the API itself would not confirm the account, so there is no self-healing false positive to suppress. Response: an account reported down is almost always an expired/revoked token — go to Option 0 (passive refresh) first, then Option A/B if there is no refresh token.

nightly-infra-check integration

The nightly infra check (.claude/commands/nightly-infra-check.md) should include a claude-runner auth check:

# Check claude-runner auth on vps-i1
vps_i1_auth=$(ssh root@217.154.82.162 "runuser -l claude-runner -c 'claude -p \"ping\" --print 2>&1 | head -1'")
if echo "$vps_i1_auth" | grep -qi "401\|unauthorized\|auth"; then
  # Fire alert
fi
 
# Check claude-runner auth on bms-4
bms4_auth=$(ssh root@54.36.123.110 "runuser -l claude-runner -c 'claude -p \"ping\" --print 2>&1 | head -1'")
if echo "$bms4_auth" | grep -qi "401\|unauthorized\|auth"; then
  # Fire alert
fi

---

Cron Dedup on bms-4

Issue (observed 2026-06-22): bms-4 had 3 duplicate hourly triage entries in root’s crontab, spawning 3 concurrent triage agents every hour.

Check for duplicates:

ssh root@54.36.123.110 "crontab -l | grep -n triage"

Fix (leave exactly one entry in claude-runner’s crontab, remove all from root’s crontab):

# Remove all triage entries from root crontab on bms-4
ssh root@54.36.123.110 "crontab -l | grep -v triage | crontab -"
 
# Verify claude-runner crontab has exactly one entry
ssh root@54.36.123.110 "crontab -u claude-runner -l | grep triage"

Expected output: exactly one line like:

0 * * * * /path/to/hourly-triage-runner.sh

---

Revoked vs. stale — and why the alert now repeats only every 6h (#4467)

Two failure modes look identical in Discord but need completely different responses:

Symptom	Meaning	Fix
403 + locally expired, clears on the next hourly run	Transient — the refresh window race (#4076); a sibling host or concurrent run just rotated the token	None. It self-heals; the debounce already suppresses it.
403 + locally expired, survives a refresh boundary (creds mtime does not advance)	Stale/revoked — the refresh token no longer works at Anthropic’s end	Human. Option 0 will NOT help — go to Option A/Option B.
HTTP 400 / 401 on the refresh POST	Outright rejected — revoked or malformed	Human. Same as above.

refresh-claude-token.py cannot heal either of the bottom two rows: a refresh call against a revoked refresh token returns 403/401 no matter how often it retries. Before #4467 the hourly cron re-paged Discord + Telegram on every run for exactly that unfixable condition (bms-4 account claude-runner-2 paged hourly for days).

What changed: repeat chat alerts for a terminal failure are now rate-limited to one per TERMINAL_ALERT_INTERVAL_H (6h), tracked as last_terminal_alert_ms in the per-account state file (/var/lib/p24/claude-token-state[-<user>].json). The stamp is cleared automatically on recovery — a successful refresh, or a benign 403 with a healthy local token — so a genuinely new failure always pages immediately.

Operator consequences — read these before triaging:

• A quiet Discord channel does NOT mean resolved. Between pages the condition is very likely still broken. Confirm with p4_claude_auth_valid{account="<user>"} or Option 0’s status check, never by absence of alerts.
• The cron still exits non-zero every run. /var/log/claude-token-refresh.log remains the ground truth for how long the condition has persisted.
• Prometheus alerting is untouched. ClaudeRunnerTokenExpired / ClaudeRunnerTokenExpiringSoon are driven by the pushgateway metric, not by this notification path, so they keep firing normally.
• The GitHub human-action issue is still created on first occurrence and deduped per (machine, account) by gh_issue_unless_exists() — that issue, not the Discord cadence, is the thing to work off.

Both bms-4 accounts are enrolled in the hourly refresh cron (/etc/cron.d/claude-token-refresh, templated from ansible/roles/claude-runner/templates/claude-token-refresh.j2 with bms-4’s claude_accounts=[claude-runner, claude-runner-2] — #5189 — so it renders one line for claude-runner, one for --user claude-runner-2), deployed by ansible/playbooks/bms-4.yml. A recurring alert for account-2 therefore means a revoked token, never a missing cron entry — do not “fix” it by re-adding the cron line.

---

Missing (not just expired) refreshToken — the >2h loud escalation (#6717)

There is a third failure mode, distinct from the two in the table above: the refreshToken key can be absent from credentials.json entirely (empty / removed), not merely stale or rejected by Anthropic. When that happens there is nothing to POST a refresh_token grant with, so the account is dark until an interactive browser re-auth restores a token.

The 2026-08-19 → 2026-09-01 incident (#6717). credentials.json for bms-4 account claude-runner had no refreshToken for 13+ days. The cron detected it correctly the whole time (grep -c "No refreshToken in credentials.json$" /var/log/claude-token-refresh.log → 488), but the old code only:

• log.warning()’d the condition on the health-check path (no alert at all on the majority of ticks, where the access token was still locally valid), and
• fired a single un-throttled Discord line on the separate “refresh is due + missing” path.

So the loudest signal was an easily-missed Discord message, and it went unactioned for 13 days until it finally blocked real work. The sibling account claude-runner-2 on the same host was healthy throughout — this was account-specific, not a host-wide cron/OS fault.

What changed (this PR). refresh-claude-token.py now centralises the missing-refreshToken path in check_missing_refresh_token() (runs every tick) and tracks how long the condition has persisted via missing_rt_first_seen_ms in the per-account state file:

Persisted duration	Channels
< MISSING_RT_ESCALATE_H (2h)	quiet Discord only (may be a transient partial write; the next tick confirms)
>= MISSING_RT_ESCALATE_H (2h)	loud: Discord + Telegram + a GH human-action issue

The loud escalation is rate-limited by the same last_terminal_alert_ms / TERMINAL_ALERT_INTERVAL_H (6h) cadence used by the revoked/stale terminal path, so a stuck condition does not re-page on every 15-min tick. The marker (and the shared cadence stamp) are cleared automatically the moment a refreshToken re-appears. The duplicate un-throttled Discord on the “refresh due + missing” path was removed — that path now just logs and exits non-zero, since the missing token was already alerted this tick.

Operator response is the same as any other terminal auth failure: this is human-gated — do Option 0 (won’t help if the token is truly absent) then Option A/Option B browser re-auth. The GH human-action issue, not the Discord cadence, is the thing to work off.

Root cause — confirmed (bms-4 infra-task investigation, 2026-09-01, #6717)

The delegated server-side investigation (log forensics on bms-4; no credentials.json contents were ever read) reached a coherent, evidenced conclusion. It is server-side token invalidation, not local file corruption by our own tooling.

1. Our crons are exonerated as the wiper. The only writers of /home/claude-runner/.claude/.credentials.json on bms-4 are (a) refresh-claude-token.py, which rewrites atomically (tmp + os.replace) and preserves the refreshToken field — it never removes it — and (b) the Claude Code CLI itself. claude-auth-synthetic-check.py and check-sub-usage.py only read the access token and write their own state/status files; spawn-worker.sh’s ensure_claude_session() only runs claude auth status / claude auth login and never copies creds between accounts. So the empty-refreshToken state is written by the CLI, downstream of a dead token — not by a p24-infra script.

2. The driver is server-side rejection — HTTP 403 … treating refresh_token as stale in /var/log/claude-token-refresh.log. This is chronic (low-level since mid-July), escalated from ~2026-08-16, and became continuous from 2026-08-27. Both bms-4 accounts hit it at similar rates (~330 claude-runner vs ~341 claude-runner-2). Independent corroboration from the separate synthetic-auth probe (claude-auth-synthetic-check.py, a real authenticated API ping): both radieu (=claude-runner) and ecotrans (=claude-runner-2) failed auth on nearly every tick 2026-08-28 → 08-31 — i.e. both tokens were server-invalid, not merely locally absent.

3. Why only claude-runner showed the “No refreshToken” symptom (the file-wipe asymmetry). claude-runner is the primary dispatch account — the CLI is invoked on it first for every job, with claude-runner-2 used only as the fallback (#5654). Once the account’s refresh token is server-invalidated, the constantly-invoked CLI clears the dead refreshToken from credentials.json, producing “No refreshToken” on every tick (891 wipe-ticks for claude-runner vs 85 for claude-runner-2). claude-runner-2 is invoked far less, so its file kept a stale-but-present token — it logged 403-stale but not the wipe, and its refresh-claude-token.py expiry line still read “~88.5 days” (a blind mtime-based estimate, not a server-validity check — do not trust it as proof of health).

4. Corrected onset / timeline. The sustained claude-runner outage ran 2026-08-27 → 2026-09-01 10:30 UTC (continuous, every tick). The “first occurrence 2026-08-19” in the original report conflated the two accounts that share this one log file: the 2026-08-19 ~7.5 h wipe episode was actually claude-runner-2, and it self-healed. At the 2026-09-01 fix both accounts were re-authed (both show fresh ~90-day RTs at 10:45 UTC); zero “No refreshToken” since.

5. Most likely trigger + ADR-002 refinement. Anthropic uses rotating, effectively single-use refresh tokens: a successful refresh returns a new RT and invalidates the prior one server-side. radieu@gmail.com is used both on bms-4 (claude-runner) and the Windows dev laptop — so when the laptop refreshes, bms-4’s stored RT is rotated out and bms-4’s next refresh gets 403-stale. This refines ADR 002’s 2026-08-01 claim that independent same-account logins don’t invalidate each other: that held at low concurrency but not under the 2026-08-27+ usage. Caveat: claude-runner-2 is a dedicated, non-interactive account yet its 403-stale rate also escalated from 08-27, so a broader server-side session-invalidation event around that date likely contributed too — the multi-device rotation race is the best-supported but not provably the sole factor (fully proving it would need Anthropic-side session data we cannot see). Net: the RT loss is server-side invalidation — an Anthropic-side rotation/session behaviour largely outside our direct control, mitigated by account topology plus the new >2h escalation.

Durable mitigation.

• Isolate the primary worker account. Keep bms-4’s primary account (claude-runner / radieu) off interactive use on any other device — mirror the isolation that keeps claude-runner-2 healthy (the ADR-002 direction). If radieu must stay shared with the Windows dev laptop, expect recurrence; the >2h escalation shipped in #6721 now surfaces it loudly (Discord + Telegram + GH human-action) within 2 h instead of the 13 days this incident ran.
• Graceful degradation already exists: the dispatcher’s per-account fallback (#5654) routes new jobs to claude-runner-2 when the primary’s token is dead, so throughput degrades rather than stops.
• No further detection code is needed — PR #6721 shipped it; this investigation adds no code.

---

Both bms-4 accounts simultaneously blocked by DIFFERENT mechanisms — the silent fleet stall (#7575)

The #6717 section above covers a single account going dark on the missing-refreshToken path, with the sibling account healthy and the per-account fallback (#5654) keeping throughput up. The 2026-10-08 incident (#7575) is the compound case that fallback cannot absorb: both bms-4 Claude accounts were unable to run jobs at the same time, for two unrelated reasons, so every dev-issue / continue-issue job failed ~3 s after spawn with err_1 runtime=3s and the queue stalled fleet-wide from ~12:58 UTC (last good job 12:23).

Account	Why it could not run	Dispatcher symptom
claude-runner-2 (ecotrans.automation)	credentials.json had no refreshToken, access token expired ~170 h (file last written 2026-10-01). Hourly claude-token-refresh ran but cannot passively refresh without a refresh token (the #6717 class).	Tried first; killed post-spawn — OAuth token revoked detected post-spawn … exit 7 (#6447). This is the err_1 runtime=3s.
claude-runner (radieu)	Token healthy, but 7-day subscription usage_pct=97.0. spawn-worker.sh _check_subscription blocks dispatch at ≥95 % (exit 5).	Fallback account also refuses → job requeued/failed.

No other host could take over: windows-dev / dev-laptop / lap2 were rejected by the dispatcher host-level HEALTH-GATE (server_status stale/unknown). bms-4 itself stayed healthy at the host level, so the dispatcher’s existing “Zero healthy dispatch servers” fleet-stopped alert (queue-dispatcher-loop.py main(), feat/2225) did not fire — that gate is host-level and has no visibility into “this host is up but all of its Claude accounts are blocked”.

Why no clean alert surfaced it

• runner-2 missing-refreshToken → the #6717 loud escalation exists, but its Discord/Telegram/GH page is rate-limited to the shared 6 h TERMINAL_ALERT_INTERVAL_H cadence, so after the first page (runner-2 went dark ~2026-10-01) it was easy to miss across ~7 days.
• runner 97 % usage → a normal, expected transient block (exit 5, requeue-with-backoff), not an alertable fault on its own.
• The combination (both blocked at once) had no dedicated signal: ClaudeRunnerTokenExpired (monitoring/prometheus/rules/claude-token-expiry.yml) needs a fresh negative gauge unless p4_claude_auth_valid==1; a no-refreshToken file tends to push a stale/absent gauge (ClaudeRunnerTokenGaugeStale, warning only). The authoritative backstop is ClaudeAuthSyntheticFailing (p4_claude_auth_valid==0 for 30m).

Recovery (human / sys-admin — not a dev-coder action)

1. Re-auth claude-runner-2 in a browser — Telegram /auth-worker claude-runner-2 + /auth-code <code> (Option 0.5), or Option A / Option B above. Never copy .credentials.json between accounts (Option C — forbidden copy-vector).
2. claude-runner usage resets on its 7-day window (this incident: 2026-10-13 10:00 UTC). It is not re-authed — it just needs the window to roll over or usage to drop below 95 %. Re-authing claude-runner-2 alone restores the fleet because it then carries jobs while claude-runner is capped.
3. Verify: claude -p say-ok as claude-runner-2.
4. Re-queue the exhausted rows whose retry_count hit the max during the stall (this incident: dev_r_worker_queue id 12087 = p24-infra#7571, id 12089 = et-operational-platform#2340).

Latent bugs this incident surfaced, and the still-open hardening

Two latent code bugs that muddied the outage were fixed under #7575 (PR #7580) — full RCA in worker-queue-err1-3s-subscription-exhaustion.md:

• spawn-worker.sh label-count crash ([: 0\n0: integer expression expected) from grep -c … || echo 0 under pipefail.
• exit-5 requeue writing a CHECK-invalid server_preference → Postgres 23514 / HTTP 400, silently stranding rows.

Still open under #7575 (do NOT assume these are done):

1. Alert for the missing-refreshToken state with a shorter page cadence than the shared 6 h terminal window — monitoring/prometheus/rules/* + refresh-claude-token.py. Touches the heavily-constrained alert rule and needs a vps-i1 deploy → sys-admin scope.
2. Account-health-aware dispatch. The dispatcher should treat an account with a non-renewable token (missing refreshToken) like the existing depleted flag — skip/deprioritise it instead of spending a ~3–10 s spawn only to hit exit 7, and emit a distinct “account X cannot run jobs” signal (queue-dispatcher-loop.py _account_order / _mark_account_depleted machinery).
3. Fleet-stopped alert at the ACCOUNT level. The feat/2225 “Zero healthy dispatch servers” alert is host-level; it should also fire (and escalate per the Error-Notification Standard) when every healthy host has no Claude account able to run a job — the exact state that was silent here. Items 2 + 3 are one coherent dispatcher change and warrant a dedicated design task.

---

Prevention

1. Add claude-runner auth check to nightly-infra-check so expiry is detected 2–8 hours before it affects agents (access tokens auto-refresh when Claude Code is actively used).

2. ✅ DONE (#3420) — scripts/check-sub-usage.py reads expiresAt from ~/.claude/.credentials.json on each agent host and pushes claude_runner_token_expires_seconds{account,machine} to the Pushgateway. (Implemented there rather than in credential-exporter because that cron already runs per account.)

3. ✅ DONE (#3420) — ClaudeRunnerTokenExpiringSoon (2h) + ClaudeRunnerTokenExpired (0) live in monitoring/prometheus/rules/claude-token-expiry.yml, routed via the existing Alertmanager warning/critical receivers.

4. Schedule a daily Playwright-based pre-emptive refresh via the nightly agent: at 02:00 UTC, if expiresAt < 6h, attempt Playwright auto-reauth proactively.

---

Related Documentation

• docs/ai-agent-operations.md — agent fleet overview, SSH commands, monitoring
• docs/claude-agent-setup.md — original IONOS agent setup reference
• docs/servers/p4-ovh-bms-4-ns3101999-operations.md — bms-4 operations
• docs/vps-i1-operations.md — IONOS VPS operations
• CLAUDE.md §Claude Code Auth on VPSes — token lifecycle, provisioning procedure
• Reauth scripts: d:\tmp\reauth-ionos.py, d:\tmp\reauth-bms4.py (local Windows machine only)