claude-oauth-reauth

Playbook: Claude OAuth Token Re-Authentication on VPS Agents

Category: Agent Operations
Severity when triggered: P1 (auth expired, agents dark) / P2 (expiry imminent)
Affected hosts: vps-i1 (217.154.82.162), bms-4 (54.36.123.110)
Related issue: #981
Last verified: 2026-06-23

---

What Triggers This Problem

Claude Code on VPS agents (vps-i1, bms-4) authenticates via OAuth subscription (Claude Max), not via ANTHROPIC_API_KEY. The OAuth access token has a TTL of approximately 8–12 hours. The refresh token lasts months.

Expiry happens when:

• The claude-runner user has not run any Claude Code session for >12 hours (no auto-refresh)
• The refresh token is revoked (very rare; requires manual deauthorization on claude.ai)
• Credentials file is deleted or corrupted

Observed impact (2026-06-21): bms-4 token expired at 01:00 UTC. The hourly triage agent was dark for ~30 hours with 401 failures. No alert fired because no health check existed for claude-runner auth state.

---

How to Confirm the Problem

Quick check (from local Windows machine)

# vps-i1
ssh root@217.154.82.162 "runuser -l claude-runner -c 'claude -p \"say-ok\" --print 2>&1 | head -3'"
 
# bms-4
ssh root@54.36.123.110 "runuser -l claude-runner -c 'claude -p \"say-ok\" --print 2>&1 | head -3'"

Expected (auth valid): model response text
Expected (auth expired): 401 Unauthorized or Authentication required or empty output

Read token expiry from credentials file

# On the VPS (run as root, pipe to avoid printing the token value)
python3 -c "
import json, datetime
c = json.load(open('/home/claude-runner/.claude/.credentials.json'))
exp = c.get('expiresAt', 0) / 1000
dt = datetime.datetime.utcfromtimestamp(exp)
now = datetime.datetime.utcnow()
diff = (dt - now).total_seconds()
print(f'Expires: {dt.isoformat()}Z')
print(f'Seconds until expiry: {int(diff)}')
print(f'Status: {\"VALID\" if diff > 0 else \"EXPIRED\"}')"

---

Step-by-Step Fix

Option 0: Passive refresh — TRY THIS FIRST (no browser, no human)

When the credentials file still has a valid refreshToken, this is the whole fix. The OAuth access token is short-lived (~8–12h) and lapses whenever the account runs no Claude Code session for >12h, but the refresh token lasts months. Claude Code silently exchanges the refresh token for a new access token on the next invocation — so a single claude -p as the affected account restores auth with no browser, no Playwright, no 2FA.

Confirm a refresh token is present (read-only, prints no secret) and that only the access token lapsed, then trigger the refresh. Substitute the affected account’s Linux user (claude-runner → radieu, claude-runner-2 → ecotrans):

# On the affected host as root (bms-4 / vps-i1). ACCT = claude-runner | claude-runner-2
ACCT=claude-runner-2
python3 - "$ACCT" <<'PY'
import json, datetime, sys
c = json.load(open(f"/home/{sys.argv[1]}/.claude/.credentials.json"))
o = c.get("claudeAiOauth", c)
diff = o.get("expiresAt", 0)/1000 - datetime.datetime.utcnow().timestamp()
print("access token:", "VALID" if diff > 0 else "EXPIRED", "| has refresh token:", bool(o.get("refreshToken")))
PY
 
# If "has refresh token: True", trigger the auto-refresh — this alone fixes it:
timeout 90 runuser -l "$ACCT" -c 'claude -p "say-ok" --print'   # expect: OK / model text
 
# Push the fresh metric so ClaudeRunnerTokenExpired clears without waiting for the 10-min cron:
runuser -l "$ACCT" -c 'python3 /opt/p24-infra/scripts/check-sub-usage.py'

If claude -p returns OK, re-check expiresAt — it now sits ~8h in the future and you are done. Only if claude -p still returns 401 / Authentication required (the refresh token itself was revoked) do you need the full browser flow below.

Verified 2026-07-14 (#4157): ecotrans/claude-runner-2 on bms-4 was 6 days dark; the passive refresh restored it in one command — no browser needed.

Option A: Playwright Autonomous Re-Auth (browser re-auth — only if the refresh token is revoked)

This is the intended path for normal token expiry. The Playwright approach navigates the OAuth flow headlessly using a pre-authenticated browser profile.

Prerequisites:

• Playwright MCP tools available in the current Claude Code session (local Windows machine)
• Browser profile at C:\Users\konar\AppData\Local\ms-playwright\ with claude.ai session cookies still valid
• SSH access to the affected VPS

Steps:

1. Start claude auth login on the VPS inside a tmux session

   For bms-4:

   ssh root@54.36.123.110
   tmux kill-session -t claude_auth 2>/dev/null || true
   tmux new-session -d -s claude_auth -x 220 -y 50
   tmux send-keys -t claude_auth "runuser -l claude-runner -c 'claude auth login'" Enter

   For vps-i1:

   ssh root@217.154.82.162
   # vps-i1 uses a process-based approach (no tmux required, see reauth-ionos.py)

2. Capture the OAuth URL from tmux output

   # Poll until URL appears (up to 30s)
   for i in $(seq 30); do
     pane=$(tmux capture-pane -t claude_auth -p 2>/dev/null)
     url=$(echo "$pane" | grep -o 'https://[^ ]*oauth[^ ]*')
     [ -n "$url" ] && { echo "URL: $url"; break; }
     sleep 1
   done

3. Navigate Playwright to the OAuth URL

   Using Playwright MCP tool from the local session:

   • Navigate to the captured URL
   • The browser opens with the user already logged into claude.ai
   • Click the “Authorize” button
   • Capture the redirect URL — it contains ?code=<value>

4. Inject the authorization code into the tmux session

   # Extract code from redirect URL (displayed in Playwright browser address bar)
   CODE="<paste-code-from-redirect>"
    
   # bms-4:
   ssh root@54.36.123.110 "tmux send-keys -t claude_auth '$CODE' Enter"
    
   # Wait 8s for claude auth login to complete
   sleep 8
    
   # Verify
   ssh root@54.36.123.110 "runuser -l claude-runner -c 'claude -p \"say-ok\" --print 2>&1 | head -3'"

5. Restart dependent services

   # On bms-4 — restart n8n workers that call claude via n8n nodes (if applicable)
   ssh root@54.36.123.110 "cd /opt/p24-infra/bms-4 && docker compose restart n8n-worker 2>/dev/null || true"
    
   # On vps-i1 — nothing extra needed (GitHub Actions runner picks up fresh credentials per run)

6. Clean up tmux session

   ssh root@54.36.123.110 "tmux kill-session -t claude_auth 2>/dev/null || true"

---

Option B: Manual Re-Auth via Local Python Scripts

Use this when Playwright is unavailable or Option A fails twice.

Prerequisites:

• Python with paramiko installed locally (pip install paramiko)
• Local SSH key at C:\Users\konar\.ssh\id_ed25519
• A browser window logged into claude.ai

For bms-4 (54.36.123.110):

python d:\tmp\reauth-bms4.py

What the script does:

1. SSH into bms-4 as root
2. Creates a fresh tmux session named claude_auth
3. Runs claude auth login as claude-runner inside tmux
4. Polls tmux pane output until the OAuth URL appears (up to 30s)
5. Prints the URL and prompts you to paste the authorization code
6. Injects the code via tmux send-keys
7. Waits 8 seconds and verifies auth with claude -p 'say: bms4 online'
8. Restarts claude-proxy systemd service (if still running)
9. Cleans up the tmux session

What you do:

1. Run the script
2. When the OAuth URL appears, open it in your browser
3. Click “Authorize” on the claude.ai OAuth page
4. Copy the code from the redirect URL (format: code=XXXX)
5. Paste it at the script prompt

For vps-i1 (217.154.82.162):

python d:\tmp\reauth-ionos.py

What the script does:

1. SSH into vps-i1 as root
2. Writes a Python helper to /tmp/auth_helper.py
3. Runs claude auth login as claude-runner via subprocess (no tmux — uses /proc/<pid>/fd/4)
4. Polls /tmp/auth_out until the OAuth URL appears
5. Prints the URL and prompts for the code
6. Writes the code to /tmp/the_code.txt and pipes it to the helper process stdin
7. Verifies auth with claude -p hello

What you do: same as bms-4 — open URL, authorize, paste code.

---

Option C: Copy Credentials from Local — FORBIDDEN, do not use

This option is banned, not “emergency only.” Copying one host’s .credentials.json to a second host is the documented mechanism of the recurring multi-host OAuth refresh-token race analyzed in docs/adr/002-claude-account-topology.md (see §“The mechanism — what actually races”, and the copy-vector citation of this exact section). Distributing one refresh token across hosts is what turns an isolated, per-grant token rotation into a fleet-wide race: whichever host refreshes next presents a token the server has already rotated away, and the sibling host gets a 403 with a locally-expired access token — the ”🔴 refresh_token stale” failure mode (#4178, #4394, #4401, #4326, #3803).

# DO NOT RUN — kept here only to show what is now forbidden:
# scp C:\Users\konar\.claude\.credentials.json root@<host>:/tmp/creds.json
# ssh root@<host> "cp /tmp/creds.json /home/claude-runner/.claude/.credentials.json ..."

If you find yourself reaching for this because Option A/B are failing or feel slow: stop. Use Option 0 (passive refresh) first — it requires no browser and fixes the vast majority of cases in one command. If Option 0 doesn’t clear it, use Option A (per-host browser re-auth) — every host gets its own independent OAuth grant via its own claude auth login, which is what keeps per-host rotation from colliding in the first place. Never bridge a gap by importing another host’s token.

---

Escalation Path When Playwright Fails

If Playwright-based auto-reauth fails twice (two consecutive attempts within 10 minutes), escalate immediately:

1. Fire Discord alert

curl -s -X POST "$P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -d '{
    "embeds": [{
      "title": "CRITICAL: claude-runner auth expired on <host> — manual re-auth required",
      "color": 15158332,
      "description": "Auto-reauth failed twice.\n\nOAuth URL: <URL>\nHost: <host>\ntmux session: claude_auth\n\nRun: python d:\\\\tmp\\\\reauth-<host>.py",
      "url": "https://github.com/radieu/p24-infra/issues"
    }]
  }'

2. Create GitHub issue

gh issue create --repo radieu/p24-infra \
  --title "CRITICAL: claude-runner auth expired on <host> — manual re-auth required" \
  --label "human-action" \
  --label "bug" \
  --milestone "Triage" \
  --body "## Claude OAuth token expired — auto-reauth failed
 
**Host:** <host>
**Time detected:** $(date -u +%Y-%m-%dT%H:%M:%SZ)
**Auto-reauth attempts:** 2 (both failed)
 
### OAuth URL (open in browser)
\`\`\`
<URL from claude auth login>
\`\`\`
 
### Re-auth steps
1. \`python d:\\tmp\\reauth-<host>.py\` on local Windows machine
2. Open the URL in browser, authorize, paste code
3. Verify: \`ssh root@<ip> \"runuser -l claude-runner -c 'claude -p say-ok --print'\"\`
4. Close this issue
 
### Tmux session state
\`\`\`
<paste tmux capture-pane output>
\`\`\`"

3. Check Telegram notification

If the Telegram Claude bot is configured (see docs/telegram-claude-bot-operations.md), the hourly triage agent will also send a P1 Telegram notification when it detects claude -p say-ok returns 401.

---

Monitoring: Detecting Expiry Before It Happens

Manual expiry check (copy-paste ready)

# Run on each VPS agent host to see token status
python3 -c "
import json, datetime, sys
try:
    c = json.load(open('/home/claude-runner/.claude/.credentials.json'))
    exp = c.get('expiresAt', 0) / 1000
    dt = datetime.datetime.utcfromtimestamp(exp)
    now = datetime.datetime.utcnow()
    diff = (dt - now).total_seconds()
    print(f'Expires: {dt.isoformat()}Z ({int(diff/3600)}h {int((diff%3600)/60)}m from now)')
    sys.exit(0 if diff > 0 else 1)
except Exception as e:
    print(f'ERROR reading credentials: {e}')
    sys.exit(2)
"

Prometheus metric + alerts (IMPLEMENTED — #3420)

Shipped 2026-07-09. The metric is pushed by scripts/check-sub-usage.py (not credential-exporter) — that cron already runs per Linux account (claude-runner → radieu, claude-runner-2 → ecotrans) every 10 min and already reads ~/.claude/.credentials.json. It reads claudeAiOauth.expiresAt (epoch ms), computes seconds-to-expiry, and pushes it to the vps-i1 Pushgateway (job claude-sub-usage-<account>) alongside the existing sub-usage gauges:

# HELP claude_runner_token_expires_seconds Seconds until claude-runner OAuth token expires (negative = expired)
# TYPE claude_runner_token_expires_seconds gauge
claude_runner_token_expires_seconds{account="radieu",machine="bms-4"} 28800
claude_runner_token_expires_seconds{account="ecotrans",machine="bms-4"} 14400

The machine label defaults to bms-4; a vps-i1 deployment sets CLAUDE_SUB_MACHINE=vps-i1 in the cron env. The expiry is read straight from the credentials file, so an already-expired token still reports (as a negative value) even when the API returns HTTP 401 — which is exactly the silent failure mode this closes.

Alert rules live in monitoring/prometheus/rules/claude-token-expiry.yml (loaded by Prometheus via rule_files: /etc/prometheus/rules/*.yml). The warning is bounded > 0 so it never overlaps the expired critical:

- alert: ClaudeRunnerTokenExpiringSoon
  expr: |
    (claude_runner_token_expires_seconds > 0 and claude_runner_token_expires_seconds < 7200)
      and on (job) (time() - push_time_seconds < 1500)
  for: 5m
  labels: { severity: warning }
  # -> warning-digest route (email + n8n/Discord)
 
- alert: ClaudeRunnerTokenExpired
  expr: |
    claude_runner_token_expires_seconds <= 0
      and on (job) (time() - push_time_seconds < 1500)
  for: 15m          # #4202 — was 1m; must outlast one 10-min push cycle
  labels: { severity: critical }
  # -> critical route (email + n8n/Discord)
 
- alert: ClaudeRunnerTokenGaugeStale
  expr: time() - max by (job) (push_time_seconds{job=~"claude-sub-usage-.*"}) > 1800
  for: 15m
  labels: { severity: warning }
  # -> warning-digest route

Why the push_time_seconds join (#4323). The Pushgateway retains the last pushed value forever, and check-sub-usage.py pushes best-effort over SSH with failures swallowed. If pushing stops, the last sample — possibly a negative one caught mid-refresh — is scraped as fresh indefinitely, so for: at any duration eventually elapses and the alert fires against a perfectly valid token (#4321 fired 122 min while expiresAt was ~68 min in the future). The freshness join makes both rules ignore samples older than 1500s (2.5 push cycles).

Consequence: a stale gauge suppresses token-expiry alerting for that account. ClaudeRunnerTokenGaugeStale exists to surface that, and ClaudeAuthSyntheticFailing — an independent real-auth ping on a separate push job — remains the authoritative broken-auth backstop. If you see the stale warning, fix the push path; do not assume the token is fine.

Routing is automatic — no Alertmanager change was needed; the existing severity=warning / severity=critical routes carry these to email + the n8n/Discord bridge.

nightly-infra-check integration

The nightly infra check (.claude/commands/nightly-infra-check.md) should include a claude-runner auth check:

# Check claude-runner auth on vps-i1
vps_i1_auth=$(ssh root@217.154.82.162 "runuser -l claude-runner -c 'claude -p \"ping\" --print 2>&1 | head -1'")
if echo "$vps_i1_auth" | grep -qi "401\|unauthorized\|auth"; then
  # Fire alert
fi
 
# Check claude-runner auth on bms-4
bms4_auth=$(ssh root@54.36.123.110 "runuser -l claude-runner -c 'claude -p \"ping\" --print 2>&1 | head -1'")
if echo "$bms4_auth" | grep -qi "401\|unauthorized\|auth"; then
  # Fire alert
fi

---

Cron Dedup on bms-4

Issue (observed 2026-06-22): bms-4 had 3 duplicate hourly triage entries in root’s crontab, spawning 3 concurrent triage agents every hour.

Check for duplicates:

ssh root@54.36.123.110 "crontab -l | grep -n triage"

Fix (leave exactly one entry in claude-runner’s crontab, remove all from root’s crontab):

# Remove all triage entries from root crontab on bms-4
ssh root@54.36.123.110 "crontab -l | grep -v triage | crontab -"
 
# Verify claude-runner crontab has exactly one entry
ssh root@54.36.123.110 "crontab -u claude-runner -l | grep triage"

Expected output: exactly one line like:

0 * * * * /path/to/hourly-triage-runner.sh

---

Revoked vs. stale — and why the alert now repeats only every 6h (#4467)

Two failure modes look identical in Discord but need completely different responses:

Symptom	Meaning	Fix
403 + locally expired, clears on the next hourly run	Transient — the refresh window race (#4076); a sibling host or concurrent run just rotated the token	None. It self-heals; the debounce already suppresses it.
403 + locally expired, survives a refresh boundary (creds mtime does not advance)	Stale/revoked — the refresh token no longer works at Anthropic’s end	Human. Option 0 will NOT help — go to Option A/Option B.
HTTP 400 / 401 on the refresh POST	Outright rejected — revoked or malformed	Human. Same as above.

refresh-claude-token.py cannot heal either of the bottom two rows: a refresh call against a revoked refresh token returns 403/401 no matter how often it retries. Before #4467 the hourly cron re-paged Discord + Telegram on every run for exactly that unfixable condition (bms-4 account claude-runner-2 paged hourly for days).

What changed: repeat chat alerts for a terminal failure are now rate-limited to one per TERMINAL_ALERT_INTERVAL_H (6h), tracked as last_terminal_alert_ms in the per-account state file (/var/lib/p24/claude-token-state[-<user>].json). The stamp is cleared automatically on recovery — a successful refresh, or a benign 403 with a healthy local token — so a genuinely new failure always pages immediately.

Operator consequences — read these before triaging:

• A quiet Discord channel does NOT mean resolved. Between pages the condition is very likely still broken. Confirm with p4_claude_auth_valid{account="<user>"} or Option 0’s status check, never by absence of alerts.
• The cron still exits non-zero every run. /var/log/claude-token-refresh.log remains the ground truth for how long the condition has persisted.
• Prometheus alerting is untouched. ClaudeRunnerTokenExpired / ClaudeRunnerTokenExpiringSoon are driven by the pushgateway metric, not by this notification path, so they keep firing normally.
• The GitHub human-action issue is still created on first occurrence and deduped per (machine, account) by gh_issue_unless_exists() — that issue, not the Discord cadence, is the thing to work off.

Both bms-4 accounts are enrolled in the hourly refresh cron (/etc/cron.d/claude-token-refresh, templated from ansible/roles/claude-runner/templates/claude-token-refresh.j2 with bms-4’s claude_accounts=[claude-runner, claude-runner-2] — #5189 — so it renders one line for claude-runner, one for --user claude-runner-2), deployed by ansible/playbooks/bms-4.yml. A recurring alert for account-2 therefore means a revoked token, never a missing cron entry — do not “fix” it by re-adding the cron line.

---

Prevention

1. Add claude-runner auth check to nightly-infra-check so expiry is detected 2–8 hours before it affects agents (access tokens auto-refresh when Claude Code is actively used).

2. ✅ DONE (#3420) — scripts/check-sub-usage.py reads expiresAt from ~/.claude/.credentials.json on each agent host and pushes claude_runner_token_expires_seconds{account,machine} to the Pushgateway. (Implemented there rather than in credential-exporter because that cron already runs per account.)

3. ✅ DONE (#3420) — ClaudeRunnerTokenExpiringSoon (2h) + ClaudeRunnerTokenExpired (0) live in monitoring/prometheus/rules/claude-token-expiry.yml, routed via the existing Alertmanager warning/critical receivers.

4. Schedule a daily Playwright-based pre-emptive refresh via the nightly agent: at 02:00 UTC, if expiresAt < 6h, attempt Playwright auto-reauth proactively.

---

Related Documentation

• docs/ai-agent-operations.md — agent fleet overview, SSH commands, monitoring
• docs/claude-agent-setup.md — original IONOS agent setup reference
• docs/servers/p4-ovh-bms-4-ns3101999-operations.md — bms-4 operations
• docs/vps-i1-operations.md — IONOS VPS operations
• CLAUDE.md §Claude Code Auth on VPSes — token lifecycle, provisioning procedure
• Reauth scripts: d:\tmp\reauth-ionos.py, d:\tmp\reauth-bms4.py (local Windows machine only)