002-claude-account-topology

ADR 002: Claude Account Topology for claude-runner Hosts

Status: Rejected (Option 1) — proceeding with Phase 0/1 mitigations only, 2026-08-01. Human ruled: no new Claude Max subscriptions; ToS risk accepted as a monitored condition rather than eliminated. Phase 0 experiment completed 2026-08-01 — mechanism (A) confirmed (see §Open questions Q1): fresh independent claude auth login on vps-h1 did not affect vps-i1’s validity. The race was purely the copy-vector, not an account-wide server-side limitation — validates that rejecting Option 1 does not leave the race unresolved. Date: 2026-07-17 Issue: #4212 Supersedes/relates: #3803, #4076, #4202, #4210, #4178

Context

bms-4, vps-i1 and vps-h1 each run scripts/refresh-claude-token.py against the same Claude Max account (radieu@gmail.com — docs/elements.md:117, purpose “Claude Code on all VPSes + local”). The OAuth refresh_token grant rotates the refresh token server-side. Whichever host refreshes next presents a token the server has already rotated away and gets HTTP 403 while its local access token has expired — the recurring ”🔴 refresh_token stale” alert (#4178 on vps-i1).

The radieu refresh token currently exists in at least six places:

#	Location	Reference
1	bms-4 /home/claude-runner/.claude/.credentials.json	docs/elements.md:186
2	vps-i1 same path	docs/elements.md:186
3	vps-h1 same path	docs/elements.md:187
4	vps-i1 openclaw-gateway — auth recorded literally as “Claude Max OAuth (shared)“	docs/elements.md:188
5	radieu workstation C:\Users\konar\.claude\.credentials.json	docs/elements.md:189
6	AI-Dev-IO1 / AI-Dev-HS1 dev agents	docs/elements.md:190-191

claude-runner-2 (ecotrans) exists only on bms-4 and is therefore not in the race today. It is the existing precedent that a second, independent account is workable.

What has already been tried — all of it detective, none preventive

Issue	Change	Effect on the race
#3803	Host detection, versioned cron, claude-auth-synthetic-check.py (15-min real API ping), ClaudeAuthSyntheticFailing alert	Detects it faster. Does not remove it.
#4076	Debounce: escalate only when 403+expired survives a refresh boundary with credentials.json mtime unchanged	Tolerates sibling-host rotation as expected background noise (scripts/refresh-claude-token.py:60-67)
#4202	ClaudeRunnerTokenExpired for: 1m → 15m	Suppresses the false positives the race generates
#4210	Adds the missing --user claude-runner-2 cron line on bms-4 (not yet merged to main)	Orthogonal — per-account cron gap, not the race

The race is currently a known, accepted, monitored condition. This ADR decides whether to keep accepting it.

The existing “never distribute credentials” posture is deliberate

Any option that distributes tokens reverses an explicit, documented decision:

• ansible/roles/claude-runner/README.md:16 — “.credentials.json is NOT seeded by Ansible … Putting it in Ansible would risk leaking the OAuth refresh token.”
• ansible/roles/claude-env-sync/defaults/main.yml:21-24 — cites that decision as precedent for other key material.
• The Claude OAuth token is not in SOPS and not in GH Secrets (docs/elements.md:263 — backup column is —). SOPS’ relationship to these hosts is the inverse: claude-runner is an age recipient (.sops.yaml:3-7).

The mechanism — what actually races

This distinction drives the whole decision and is not currently established anywhere in the repo:

• (A) Per-grant rotation (RFC 9700 / OAuth 2.1 refresh-token rotation): each authorization grant yields an independent refresh token. Rotating host A’s token does not invalidate a token host B obtained from its own claude auth login. Under (A), the race is caused solely by copying one token to several hosts — and the repo documents exactly that copy vector: docs/playbooks/claude-oauth-reauth.md:228-241 (Option C, scp of the workstation .credentials.json to bms-4, flagged “a bridge, not a solution”), plus openclaw-gateway’s “shared” auth.
• (B) Per-account rotation: the server keys one refresh token per (client_id, account), so even independent logins collapse onto one token. Under (B), per-host logins cannot help and only separate accounts fix it.

We cannot distinguish (A) from (B) from the code, and the difference is worth the entire cost of the decision: under (A) the race is fixable at zero subscription cost. Phase 0 below settles it empirically before any money is spent.

Why the Claude Code CLI makes “single-writer” largely unenforceable

The refresh grant is not issued only by scripts/refresh-claude-token.py. The Claude Code CLI performs it autonomously on ordinary use:

“Claude Code silently exchanges the refresh token for a new access token on the next invocation — so a single claude -p as the affected account restores auth with no browser, no Playwright, no 2FA.” — docs/playbooks/claude-oauth-reauth.md:59-66 (Option 0, verified 2026-07-14, #4157)

Every host in scope exists in order to run Claude Code sessions. So every host’s CLI will call the refresh grant whenever an agent runs. A single-writer design would have to stop the CLI on non-writer hosts from ever refreshing — there is no supported configuration knob for that, and it would mean wrapping or patching the vendor CLI on every host. This is decisive and is why Option 2 is rejected below on grounds stronger than mere cost or taste.

Options considered

Option 1 — Per-host Claude sub-accounts (one account ↔ one host)

Each claude-runner host binds to its own Claude account. No account spans two hosts.

• ✅ Removes the race by construction under both (A) and (B) — no shared token to rotate.
• ✅ Preserves the documented “never distribute credentials” posture unchanged.
• ✅ Isolates rate-limit quota. Today one Max quota is shared by 3 hosts + workstation + openclaw. Quota contention is already a live, instrumented pain: p4_claude_sub_blocked, p4_claude_sub_usage_pct{,_5h,_7d}, docs/playbooks/claude-account-depletion-tracking.md, docs/playbooks/queue-failed-weekly-limit.md. This benefit is independent of the race.
• ✅ Failure isolation: one host’s revoked token cannot stall the fleet (the #3803 failure mode).
• ✅ Removes account-sharing exposure under Anthropic’s consumer terms (see §Open questions).
• ✅ Precedent exists: claude-runner-2/ecotrans on bms-4 already works this way.
• ✅ Self-sustaining in steady state: rotation resets the 90-day RT clock on every refresh (scripts/refresh-claude-token.py:318-333), so an actively used, unshared account effectively never reaches RT expiry. The race is what forces today’s manual browser re-auth.
• ⚠️ Cost: one subscription per host. Evidence base is ambiguous — see §Open questions.
• ⚠️ Re-auth toil scales with account count for any account left idle >90 days.

Option 2 — Single-writer refresh + distribution

One host runs the refresh grant on a schedule and pushes rotated tokens to the others.

• ❌ Largely unenforceable — the Claude Code CLI refreshes on its own on every host that runs a session (see above). The writer cannot hold exclusivity.
• ❌ Reverses an explicit documented security decision (claude-runner/README.md:16).
• ❌ Creates a hot credential-movement channel running at ≤2h cadence between three hosts, carrying a token that grants full access to the Claude account. This is a permanent, high-value distribution path — a materially larger blast radius than today’s per-host-only storage.
• ❌ SOPS+git is infeasible as the transport: rotation lands every ~8–12h, so it implies a commit and a secrets-sync.yml run per rotation.
• ❌ Adds a new SPOF (the writer host) on top of the already-documented bms-4 dispatcher SPOF (audit §5). Writer down → whole fleet goes stale together — a worse failure mode than today, where hosts fail independently.
• ❌ Does not fix quota contention or account sharing.
• ✅ Only merit: no additional subscription cost.

Option 3 — Per-host independent grants on the same account (surfaced; not in the issue)

Keep one subscription; ban copying; each host runs its own claude auth login.

• ✅ Zero cost. Under mechanism (A) this alone removes the race.
• ✅ No distribution; posture unchanged.
• ❌ No effect under mechanism (B).
• ❌ Does not fix quota contention (one Max cap still shared by 5–6 consumers).
• ❌ Does not address account sharing.

This is not a competing end-state — it is the immediate stop-the-bleeding step and the experiment that prices Option 1. It is Phase 0 below.

Option 4 — ANTHROPIC_API_KEY instead of OAuth (surfaced; rejected)

• ✅ Dissolves the problem entirely: API keys have no rotation semantics and no race.
• ❌ Reverses a deliberate architecture choice — docs/ai-agent-operations.md:33: “Claude Code authenticates via OAuth subscription (Claude Max), not via ANTHROPIC_API_KEY.”
• ❌ Replaces flat-rate with metered billing for the fleet’s highest-volume workload; the “zero marginal cost” property is asserted across docs/ai-risk-data-governance.md:68, docs/environments/bms-4.md:65, docs/environments/vps-i1.md:62 and others.

Recorded for completeness; not recommended.

Trade-off summary

	Opt 1 per-host accounts	Opt 2 single-writer	Opt 3 per-host grants, 1 account	Opt 4 API key
Removes race under (A)	✅	⚠️ unenforceable	✅	✅
Removes race under (B)	✅	⚠️ unenforceable	❌	✅
Subscription cost	N×	1×	1×	metered
Distributes secrets	no	yes (≤2h cadence)	no	no
Reverses documented decision	no	yes	no	yes
Fixes quota contention	✅	❌	❌	✅
Fleet-wide SPOF	none	writer host	none	none
Fixes account sharing	✅	❌	❌	✅

Decision

Recommend Option 1 (per-host Claude sub-accounts) as the target state, reached in phases, with the subscription spend gated on a zero-cost experiment that may show most of the benefit is available immediately.

Option 2 is rejected — not primarily on cost, but because the Claude Code CLI refreshes tokens autonomously on every host, so single-writer exclusivity cannot be enforced without patching the vendor CLI; and because it would build a permanent hot credential-distribution channel in direct reversal of claude-runner/README.md:16.

This ADR is Proposed, not Accepted. It requires human ratification, because it carries a recurring cost decision (§Open questions Q2) and a terms-of-service question (Q3) that an agent must not settle unilaterally. Ratify by approving the PR and flipping Status to Accepted. Phase 0 is safe to execute before ratification; Phase 2 must not start until Q2 and Q3 are answered by a human.

Outcome — 2026-08-01

The human (radieu, repo owner) rejected Option 1 (per-host Claude sub-accounts). No new Claude Max subscriptions will be purchased — Q2 (real subscription cost) and Q3 (ToS comfort with one account across 3 hosts) are answered by this decision, not resolved: the ToS risk identified in Q3 is accepted as a monitored condition rather than eliminated, and Q2 is moot since no spend is authorized. Phase 2 (new accounts, per-host provisioning) is out of scope and must not be started by any agent absent a future, separate human decision reopening it.

Phase 0 (zero-cost mitigations: ban the copy vector, run the vps-h1 mechanism-(A)/(B) experiment, re-home openclaw-gateway) and Phase 1 (doc correction: docs/environments/vps-h1.md, and confirming #4210’s cron fix is already merged) proceed as originally designed — both were written to be independent of the Phase 2 gate. See #4807 for the execution tracking issue and its PR for what was completed vs. left as human-only follow-up (the two interactive claude auth login steps that this agent cannot perform).

Migration plan

Per the issue guardrail, no credential is touched by this ADR. Every step that moves or creates credentials is executed by secret-manager; server operations by sys-admin. This document is the plan only.

Phase 0 — Stop the bleeding + settle mechanism (A) vs (B) — zero cost

1. Ban the copy vector. Rewrite docs/playbooks/claude-oauth-reauth.md Option C (:228-241) from “emergency only” to forbidden: copying one .credentials.json to a second host is the documented mechanism of this incident class. Point it at Option 0 (passive refresh) then Option A (per-host browser re-auth) instead.
2. Experiment (sys-admin, on vps-h1 — lowest-risk host; “free general-purpose AI worker”, no production dispatch):
   • Record the current refresh_token_hash on vps-h1 and vps-i1. These are already logged — truncated sha256[:16], not secret material (scripts/refresh-claude-token.py:155-156 and scripts/refresh-claude-token.py:338).
   • Perform a fresh, independent claude auth login on vps-h1 for the same account.
   • Force a refresh on vps-h1; confirm its hash changes.
   • Read p4_claude_auth_valid{machine="vps-i1"} and run claude-auth-synthetic-check.py on vps-i1.
   • vps-i1 stays valid → mechanism (A): grants are independent; the race is a copying artifact. vps-i1 goes 401/403 → mechanism (B): rotation is account-global.
   • No new instrumentation is needed — #3803 already built the exact observability this test wants.
3. Re-home openclaw-gateway (docs/elements.md:188): it shares the radieu OAuth with claude-runner on the same host, so vps-i1 races against itself. Give it its own grant (or its own account under Phase 2).
4. Record the Phase 0 result on #4212 and update this ADR’s Status.

Phase 0 execution log — 2026-08-01 (sys-admin, read-only SSH)

Step 1 (ban the copy vector) is done — see docs/playbooks/claude-oauth-reauth.md Option C.

Step 2 (the vps-h1 experiment) is not completed — it hinges on a fresh, independent claude auth login on vps-h1, which is an interactive OAuth browser flow (URL + human click + paste-back code) that no agent can perform. What was done instead, read-only over SSH (no credential file was read or modified, only the already-safe truncated hash and timestamps in /var/lib/p24/claude-token-state*.json):

• vps-i1 baseline: refresh_token_hash = dcf03bc34e4ede64 (refresh_token_set_at_ms/refresh_token_expires_at_ms present and consistent with the ~90-day RT lifetime assumption). Confirmed via /var/lib/p24/claude-token-state.json.
• vps-h1 baseline: not available. /var/lib/p24/claude-token-state.json does not exist on vps-h1, and neither does /etc/cron.d/claude-token-refresh — despite ansible/roles/claude-runner/tasks/main.yml deploying that cron file unconditionally for every host the claude-runner role is applied to (ansible/playbooks/vps-h1.yml:24), and despite .credentials.json and the claude-runner Linux user both existing on vps-h1 (confirmed via id claude-runner). In other words: the host has the agent and the credential, but the ansible-managed hourly refresh cron that would populate the state file was never actually provisioned there — a gap independent of this ADR. Filed as #4854; not fixed in this PR because remediating it means installing a new cron file on the WAHA-era-labelled vps-h1 host, which is outside this session’s sys-admin role boundary for that host without human sign-off.
  • Practical effect on the experiment: until that gap is closed (or a state file is bootstrapped manually), there is no existing baseline hash to diff against post-login on vps-h1. Whoever performs the interactive login below should also either (a) get the missing cron deployed first so the state file populates automatically, or (b) read claudeAiOauth.refreshToken’s hash manually (never the raw token) before and after login as a one-off substitute.
  • What needs to run, by whom: a human, interactively, on vps-h1 (ssh root@72.60.32.61): runuser -l claude-runner -c 'claude auth login', complete the browser OAuth flow for the radieu account, then force a refresh (claude -p "say-ok" --print) and compare p4_claude_auth_valid{machine="vps-i1"} / re-run claude-auth-synthetic-check.py on vps-i1 to read the mechanism (A) vs (B) result per the step-2 procedure above.

Step 3 (re-home openclaw-gateway) — investigated, not implemented (same interactive-login blocker). Confirmed the exact sharing mechanism: services/openclaw/docker-compose.yml:16-17 bind-mounts vps-i1’s claude-runner credentials directly and read-only into the openclaw-gateway/openclaw-cli containers:

- /home/claude-runner/.claude:/home/node/.claude:ro
- /home/claude-runner/.claude.json:/home/node/.claude.json:ro

This is not a second independent grant sharing the same account (which would at least be an (A)-mechanism-safe design) — it is literally the same on-disk credential file, read-only, so openclaw never persists its own refresh. If code inside the container ever exercises the refresh grant (e.g. an internal Claude Code SDK call triggered by an expired local access token) the OAuth server still honors that HTTP call and rotates the refresh token server-side, but the container cannot write the rotated value back to the read-only mount — silently invalidating the claude-runner cron’s copy of the refresh token on the same host, independent of bms-4/vps-h1. This is a plausible root cause for at least some of the vps-i1-side incidents in the issue list above and is worse than a normal cross-host race because there is no independent write path to recover from.

• Concrete fix (not applied here): stop bind-mounting claude-runner’s directory; give openclaw-gateway its own credential path (e.g. a new ${OPENCLAW_CLAUDE_CONFIG_DIR} volume) and populate it via its own claude auth login — still the same radieu account, no new subscription, consistent with the ADR’s rejected-Option-1 outcome.
• What needs to run, by whom: a human, interactively: run claude auth login for a session rooted at the new credential path (e.g. via docker compose run openclaw-cli once the compose volume is repointed, or directly on the host before repointing), then update services/openclaw/docker-compose.yml to mount that path instead of claude-runner’s, and redeploy (docker compose up -d --force-recreate on vps-i1 — brief container restart, no WhatsApp session data affected since that lives in ${OPENCLAW_CONFIG_DIR}, not the .claude mount). Left undone here because it requires the same interactive OAuth step as above, plus a live container redeploy this session is not authorized to perform without a human decision on timing.

Phase 1 — Correct the record (independent of outcome)

5. ✅ DONE (#4807, 2026-08-01). docs/environments/vps-h1.md:49-51 was stale — it stated “vps-h1 has no claude-runner user. Claude Code is not installed here. … Do not install Claude Code … on this host”, contradicted by ansible/playbooks/vps-h1.yml:24 (which applies the claude-runner role), .sops.yaml:4 (recipient “vps-h1 claude-runner”), scripts/cron.d/claude-token-refresh:2, scripts/refresh-claude-token.py:75-84 and docs/elements.md:187. Corrected to state the host does run claude-runner. (Investigating this also surfaced that the OAuth-refresh cron itself is not actually deployed on vps-h1 — a separate, previously-unknown gap, tracked as #4854, not fixed here.)
6. ✅ ALREADY DONE before this PR. #4210 (fix/4210-claude-runner-2-cron) is merged and closed; scripts/cron.d/claude-token-refresh already ships both the default claude-runner line and the --user claude-runner-2 line. No action taken here — confirmed only, not re-done.

Phase 2 — Per-host accounts (gated on human answers to Q2 + Q3)

7. Human: confirm seat/cost model and provision one account per claude-runner host (bms-4 already has ecotrans; vps-i1 and vps-h1 need their own).
8. secret-manager: per-host claude auth login for the new account. .credentials.json stays hand-provisioned and out of Ansible and out of SOPS — Option 1 requires no change to that posture.
9. ✅ DONE (#5189). The cron is now templated from the claude-runner role (templates/claude-token-refresh.j2) with a claude_accounts host var — one refresh line per account — replacing the static scripts/cron.d/claude-token-refresh copy (now removed). Under Option 1 each host has exactly one account, so the single default line (claude_accounts defaults to [claude-runner]) is correct for vps-i1/vps-h1; bms-4 overrides it to [claude-runner, claude-runner-2] in host_vars and keeps two (#4210). This ends the drift where the bms-4-shaped static file’s --user claude-runner-2 line was copied onto single-account hosts.
10. ansible/roles/claude-runner: no structural change — the role creates the user and installs cron; credentials remain manual by design (README.md:16).
11. Populate dev_r_server_capacity.claude_accounts per host. The column already exists and already falls back to ["claude-runner"] when NULL (docs/playbooks/claude-account-depletion-tracking.md:30-31).
12. Monitoring needs no new work: the per-account label scheme from #3932 already emits one p4_claude_rt_expires_days{machine,account} series per (account, machine) (scripts/refresh-claude-token.py:170-187), and Grafana worker-queue-v1 panels 104–111 are already keyed by account.
13. Update docs/elements.md:117 (subscription count), :185-191 (agent inventory), :263 (credential inventory).
14. Retire the #4076 debounce only once the race is confirmed gone. It is a correct guard against a genuinely transient 403 and should outlive the race; re-evaluate, do not reflexively remove.

Phase 3 — Close out

15. Close #4178 as resolved-by once Phase 0 (mechanism A) or Phase 2 (mechanism B) lands.
16. Revisit the ClaudeRunnerTokenExpired for: 15m window (#4202) and ClaudeAuthSyntheticFailing for: 30m — both were widened to absorb race-generated false positives. With the race gone they can likely tighten, improving MTTD for genuine auth failure.

Consequences

• Under mechanism (A), Phase 0 removes the race for zero subscription cost, and Option 1’s remaining value is quota isolation + ToS compliance — a separate, weaker business case that the human can then price honestly rather than under alert pressure.
• Under mechanism (B), Phase 2 is the only real fix and the cost is unavoidable.
• Either way the fleet stops treating a preventable condition as monitoring noise. #4076’s debounce currently encodes “sibling host rotated my token” as expected; that is a design smell worth retiring.
• Per-host accounts increase the number of 90-day re-auth events only for idle accounts. An actively-used unshared account renews its own RT clock on every rotation and should not need manual re-auth in steady state.
• .credentials.json remains outside Ansible, SOPS and git in every recommended phase. This ADR does not weaken the credential posture; Option 2 would have.

Open questions — human input required

• Q1 (settled by Phase 0, no human needed): is refresh-token rotation per-grant (A) or per-account (B)? ANSWERED 2026-08-01 — mechanism (A), per-grant. Baseline refresh_token_hash recorded on both hosts before the experiment (vps-i1: dcf03bc34e4ede64, vps-h1: 175b5ac41b61f459). Performed a fresh, independent claude auth login on vps-h1 (tmux + manual OAuth authorize, per docs/playbooks/claude-oauth-reauth.md Option A/B hybrid — no Playwright MCP available in this session, so the user completed the browser authorize step manually and relayed the code back for injection). vps-h1’s hash changed to 724c52da359b32d6 (confirmed independent grant) and claude -p "say-ok" verified working there. vps-i1 immediately re-tested and still valid (claude -p "say-ok" → OK, no reauth needed) — proves the two hosts’ refresh tokens rotate independently. The race in 3803 was caused solely by the copy vector (Option C, now forbidden), not by an inherent per-account server-side limitation.
• Q2 (human, blocks Phase 2): what does an additional Claude Max subscription actually cost? The only seat-economics data in the repo is docs/ai-agent-operations.md:230-235 ($10 / 20 accounts flat, ~€9.25/agent, ~€27.75/month for 3 agents) and it is ambiguous — it does not reconcile with Claude Max’s published per-seat pricing, and no doc states how many subscriptions are actually purchased. This line is the entire evidence base for “buy N more” and it must be confirmed with the real billing account before Phase 2.
• Q3 (human, blocks Phase 2): does running one Claude Max account concurrently on three servers comply with Anthropic’s terms? No doc anywhere states that multi-host use of one account is permitted or prohibited. If it is not permitted, Option 1 stops being a cost trade-off and becomes mandatory, and Q2’s answer no longer matters.
• Q4: should the radieu workstation and the AI-Dev-IO1/AI-Dev-HS1 dev agents (docs/elements.md:189-191) also be separated, or is the human workstation an accepted co-tenant?

Related

• #3803 — original P0 fleet-stuck incident (closed) — built the detection this ADR’s Phase 0 reuses
• #4076 — transient-403 debounce — tolerates the race
• #4202 — for: window widening — absorbs race false positives
• #4210 — per-account cron line (merged, closed) — orthogonal, shipped independently
• #4178 — vps-i1 symptom alert — closed by this work
• #4854 — vps-h1 claude-token-refresh cron never deployed (discovered during this ADR’s Phase 0 experiment, filed 2026-08-01, not fixed by this PR)
• docs/playbooks/claude-oauth-reauth.md — re-auth paths; Option C is the copy vector to remove
• docs/playbooks/claude-account-depletion-tracking.md — quota contention this ADR also relieves