p24-infra Docs

plan-4815-fleet-root-credential-isolation

12 min read

Plan: Move fleet-root credentials off the shared claude-runner account (C-R6(b) Option B)

Issue: #4815 Type: Code-change-design (plan only — no credential move in this PR) Status: Draft — iteration 1 Last updated: 2026-08-01 Author: Claude Opus 4.8 (worker session bms4-cw-1 on bms-4, role dev-coder) Related: #4724 (C-R6(b) Option A — --allowedTools allowlist, shipped) · #4706 (role-capability enforcement — ROLE_CAP_ENFORCE, the env-level SSH/SOPS withholding this plan completes) · #4245 / #4065 (GitHub App-token fleet migration — already covers most of the GitHub-PAT dimension) · #2075 / docs/plans/plan-per-role-worker-credentials.md (per-role worker credentials) · docs/adr/002-claude-account-topology.md (#4212 — second-account precedent)

Scope discipline. This document is a design/scoping artefact, exactly as the issue requires (“this issue is for scoping/designing the approach, not a same-day credential move”). It contains no secret values — key/file names only. Every execution step is delegated to sys-admin (OS users, systemd-run, dispatcher deploy) and secret-manager (gh auth login switch, any SOPS touch). A dev-coder worker authored it and cannot perform those steps inline.

0. Summary — recommendation

The issue offers three candidate approaches (1: move vps_root_key to a distinct OS user; 2: switch gh auth to a scoped bot account; 3: per-agent OS-user isolation). Grounding them against the actual code changes the sequencing, and reveals that half the work is already built or in flight:

DimensionIssue optionReality on the host todayRecommendation
GitHub tokenOption 2 (gh auth → AI-Dev-BMS4-1)The git path already mints 1h App-installation tokens via bms-4/git-credential-gh-token.sh (PAT fallback), migration owned by 4065. What is still radieu’s owner-level PAT is the interactive gh auth login used by telegram-claude-bot’s /ask and the gmail daily agent.Do the login switch (Phase 0) — it is cheap and standalone — but do NOT re-plan the git-token path; ride 4065.
Fleet-root SSHOption 1 (vps_root_key → distinct OS user)#4706 already derives ROLE_DENY_SSH and, when ROLE_CAP_ENFORCE=1, unsets SSH_AUTH_SOCK + sets GIT_SSH_COMMAND=/bin/false for SSH-denied roles. Its own guard comment (spawn-worker.sh:651-657) states it cannot revoke on-disk read of ~/.ssh/vps_root_key and defers that to “per-role unix users” as a follow-up. This issue is that follow-up.Primary work: job-type → OS-user split so only infra-task/infra-alert workers get the key on disk (Phase 1).
Always-on agentsOption 3 (per-agent OS user)telegram-claude-bot + gmail daily agent run as claude-runner (workspace-isolated #3708, but same home dir, same vps_root_key, same age key).Give them a low-privilege OS user with neither the SSH key nor the age key (Phase 2).

Why the ordering matters. The failure mode of getting Phase 1 wrong is “every queue worker on bms-4 loses SSH simultaneously”. So the plan front-loads the two low-risk, independently-shippable wins (Phase 0 gh login; enabling the already-built #4706 env guard) and gates the OS-user restructuring behind an explicit human decision (§6 ARCH GATE) and a staged, reversible rollout.

Recommended path, lowest-blast-radius first:

  1. Phase 0 — gh auth login → AI-Dev-BMS4-1 bot (Option 2). Contains the GitHub blast radius from “everything the owner can do” to a scoped repo grant. No queue-worker dependency. secret-manager + sys-admin.
  2. Phase 1a — flip ROLE_CAP_ENFORCE=1 (#4706 is coded and dry-run-clean today). Zero new code — closes the env-inherited SSH/SOPS paths for roles that don’t need them. Fully reversible (=0).
  3. Phase 1b — job-type → OS-user split (Option 1): a dedicated claude-infra user owns vps_root_key; the dispatcher routes only infra-task/infra-alert/infra-task-request to it. claude-runner* workers no longer have the key on disk — closing the residual gap #4706 explicitly left open.
  4. Phase 2 — per-agent low-priv users (Option 3) for telegram-claude-bot + gmail daily agent, plus systemd-run PrivateTmp/BindPaths hardening as defence-in-depth.

Phases 0, 1a, 1b, 2 are each independently valuable and independently revertible. The issue can be closed after Phase 0+1a+1b land; Phase 2 may be split to a child issue if the OS-user work proves large.

1. Problem restatement (grounded)

docs/ai-risk-data-governance.md C-R6(b) sets the ceiling: an agent invoked as claude --dangerously-skip-permissions (gmail daily agent) — or any process that reaches the claude-runner UID through any path (a queue-worker bug, a dependency vuln, the telegram /ask handler) — gets arbitrary command execution as claude-runner on bms-4. That account’s home holds:

Credential (name only)PathReach
Fleet-root SSH key~/.ssh/vps_root_keyroot SSH to bms-1/2/3/4, vps-i1, vps-h1
SOPS age key~/.age/p24-infra-keys.txtdecrypts every secrets/*.env.sops
GitHub login~/.config/gh/ (active gh auth)owner-level PAT today

C-R6(b) is explicit that no in-run mitigation exists for the host dimension: “a file on disk readable by claude-runner stays readable” — unsetting env vars closes inherited paths only. The --allowedTools allowlist shipped in #4724 (Option A) closes one agent’s own tool surface but does nothing for a compromise arriving through a different path. Only three fixes actually reduce C-R6(b), and the issue names all three; this plan is the design pass for them.

The three durable references agree on the framing: docs/ai-risk-data-governance.md C-R6(b) · docs/gmail-tools-daily-agent-operations.md §Activation checklist blocker 3 (Option A/B/C) · docs/priorities.md §P1 (telegram-claude-bot entry).

2. What already exists (do not rebuild)

Concrete, so the plan extends rather than duplicates:

  • scripts/spawn-worker.sh:648-673 — #4706 SSH guard. When ROLE_CAP_ENFORCE=1 and the role’s capabilities.cannot implies server ops (ROLE_DENY_SSH=1), it unset SSH_AUTH_SOCK and export GIT_SSH_COMMAND=/bin/false. dev-coder.cannot includes “SSH into a production server”, so a dev-issue worker is already SSH-denied at the env level — but the guard’s own comment (lines 651-657) states it “does NOT revoke read access to an on-disk key under ~/.ssh… Closing that residual gap needs per-worker filesystem isolation (systemd-run PrivateTmp/BindPaths, or per-role unix users) and is deliberately out of scope here — tracked as a follow-up in docs/worker-role-scoping-operations.md.” #4815 is that follow-up.
  • scripts/spawn-worker.sh:566-568 — #4706 SOPS guard. Symmetric: ROLE_DENY_SOPS=1 unsets SOPS_AGE_KEY_FILE. Same on-disk caveat applies to ~/.age/p24-infra-keys.txt.
  • scripts/spawn-worker.sh:333-339 — job-type → role map already exists: infra-task|infra-alert|nc-alert-batch → sys-admin, dev-issue|continue-issue → dev-coder, review-* → dev-reviewer, secret-manager → secret-manager. Phase 1b reuses this exact mapping to pick the OS user, so there is no second source of truth for “which job type needs fleet SSH”.
  • scripts/queue-dispatcher-loop.py:1287-1306 — spawn path already runs su -s /bin/bash {worker_user} -c '{spawn_script}' and iterates worker_users from _account_order(server) (today claude-runner, claude-runner-2 for subscription balancing). The user is already a parameter — Phase 1b adds an infra-vs-general dimension to _account_order, it does not invent a new spawn mechanism.
  • **bms-4/git-credential-gh-token.sh + 4065 — the git-over-HTTPS token path is already App-token-first (1h TTL) with PAT fallback scheduled for removal. Phase 0 only touches the interactive gh auth login, a different surface.
  • docs/adr/002-claude-account-topology.md + claude-runner-2 (ecotrans) — a second independent OS/Claude account on bms-4 is already precedent; Phase 1b/2 add users, not a new topology class.
  • docs/plans/plan-per-role-worker-credentials.md — per-role SOPS files + signing identity; this plan is the OS-user / filesystem complement to that secret-scoping work. Keep them aligned: the OS user a role runs as (here) should be the same axis the per-role SOPS files key on (there).

3. Only infra-* job types genuinely need fleet-root SSH

The load-bearing fact for Phase 1b. From the spawn path, worker prompts, and job-type map:

Job type(s)Fleet-root SSH needed?Why
infra-task, infra-alert, infra-task-requestYESSSH root@bms-1/2/3, vps-i1 for MongoDB/Docker/systemd/remediation (infra-task-request-worker.md; scripts/remediation/*; spawn-infra-task-worker.sh)
dev-issue, continue-issuenogit over HTTPS (gh credential helper); no server shell
review-pr, review-plannoGH API only
nc-alert-batch, alert-triage-batchnolocal Prometheus/Supabase REST on the host they run on
secret-managerno fleet SSHreads SOPS + distributes to the local env files / Vercel; does not need root SSH to other bare-metal boxes for its core job (verify against worker-secret-manager.md before Phase 1b cutover)

So a single dedicated OS user (claude-infra, name TBD) can hold vps_root_key, and every other worker class runs as a user that never has the key on disk. This is Option 1 “done right”: scoped by the already-existing job-type→role classification rather than by trusting each agent to self-restrict.

4. Design — phase by phase

Phase 0 — gh auth login → AI-Dev-BMS4-1 bot (Option 2 / issue Option 2)

Goal: the active gh auth login on bms-4 stops being radieu’s owner-level PAT. Owner: secret-manager (mint/store the scoped bot PAT in the correct SOPS file) + sys-admin (gh auth login on the host as the worker user, or deploy the credential). Changes:

  • Provision/confirm the AI-Dev-BMS4-1 bot account and a scoped fine-grained PAT (repo scope on the repos workers actually touch; no admin:org/owner scope). Store the key name in a SOPS file per the SOPS map (candidate: secrets/n8n-bms4.env.sops, which already holds GH_TOKEN; confirm with secret-manager — do not invent a new file without checking plan-per-role-worker-credentials.md).
  • Switch the interactive gh login used by telegram-claude-bot /ask and the gmail daily agent to that account. Verify git-credential-gh-token.sh App-token path is unaffected (it is a separate helper, not the gh login). Regression risk: any automation relying on owner-level scope via the gh CLI login breaks. Audit gh api/gh calls in dispatcher, github-runner role, queue-analyst, n8n-workflow-export for scopes the bot PAT lacks (runner registration may need admin on the repo — verify before cutover). Reversible: re-run gh auth login with the prior credential.

Phase 1a — flip ROLE_CAP_ENFORCE=1 (enable the already-built #4706 env guard)

Goal: stop env-inherited SSH/SOPS reach for roles that don’t need it, with zero new code. Owner: sys-admin (set ROLE_CAP_ENFORCE=1 in the worker environment / dispatcher spawn env). Changes: none to code — #4706 ships dry-run today (spawn-worker.sh:425 logs DRY-RUN … nothing withheld). Flip the flag, watch the [role-cap] … (ENFORCING) log line and mezmo_log role_cap_ssh_withheld events for a cycle. Regression risk: low but real — if any dev-issue/review worker quietly depended on SSH_AUTH_SOCK or the age key, it now fails closed. The mapping treats that as a bug to surface, not to tolerate. Validate on one worker user first. Reversible: ROLE_CAP_ENFORCE=0.

Phase 1a alone does not close C-R6(b): env withholding ≠ filesystem isolation. It is a cheap, reversible hardening that makes Phase 1b’s blast-radius smaller by ensuring nothing legitimately uses the inherited paths before the on-disk key moves.

Phase 1b — job-type → OS-user split for vps_root_key (Option 1, the core fix)

Goal: vps_root_key exists on disk only under a dedicated user; general workers cannot read it. Owner: sys-admin (OS users, key relocation, dispatcher deploy) + secret-manager (if the key is re-sourced from SOPS during the move). Target state:

  • New OS user (candidate claude-infra) on bms-4 (and vps-i1 where infra workers run). Owns ~/.ssh/vps_root_key at mode 0600. claude-runner* home dirs no longer contain the key.
  • scripts/queue-dispatcher-loop.py _account_order() / the spawn loop (:1287-1306) selects the OS user by the same job-type→role classification already in spawn-worker.sh:333-339: infra-task|infra-alert|infra-task-request → claude-infra; everything else → claude-runner*. (Keep the subscription-balancing account rotation within each class.)
  • spawn-infra-task-worker.sh (the n8n path) targets claude-infra.
  • Age key ~/.age/p24-infra-keys.txt: decide whether it also moves to claude-infra or stays broadly readable. Recommendation — keep it out of claude-runner* too, letting ROLE_DENY_SOPS + the per-role SOPS files (plan-per-role-worker-credentials.md) govern who decrypts what. Flag for the human decision (§6): this couples #4815 to the per-role-SOPS rollout and may widen scope. Regression risk (highest in the plan): mis-routing a job type that does need SSH to a keyless user strands every such job; conversely, if the subscription-balancing rotation is not preserved per-class, dispatch capacity drops. Mitigation: staged rollout — (i) create claude-infra and copy the key, keep claude-runner’s copy; (ii) route only infra-alert to the new user, watch a full cycle; (iii) add infra-task/infra-task-request; (iv) only then remove the key from claude-runner*. Each step is independently revertible until step (iv). Reversible until final key removal: re-add the key to claude-runner*; restore prior _account_order.

Phase 2 — per-agent low-privilege users for always-on agents (Option 3)

Goal: telegram-claude-bot /ask and the gmail daily agent run as users holding neither vps_root_key nor the age key. Owner: sys-admin. Changes: dedicated OS users for each always-on agent; update their systemd units (claude-session-manager.service, the gmail agent invocation su -s /bin/bash claude-runner … in docs/gmail-tools-daily-agent-operations.md). Add systemd-run/unit hardening (PrivateTmp=yes, ProtectHome=/BindPaths= scoped to the agent’s own workspace) as defence-in-depth so even a shared user cannot traverse to another agent’s credential files. Regression risk: agent workspaces, auth caches (~/.claude/.credentials.json), and log paths (/var/log/p24-infra-workers/… owned by claude-runner) must be re-permissioned for the new users. Coordinate with ADR 002 account topology (avoid multiplying Claude-account refresh-token copies). Reversible: revert unit User=/hardening.

5. Compliance & documentation deliverables (for the implementation PRs, not this plan)

  • Update docs/worker-role-scoping-operations.md — replace the “residual gap / follow-up” note with the delivered filesystem isolation.
  • Update docs/ai-risk-data-governance.md C-R6(b) — downgrade the host dimension once Phase 1b+2 land (or record explicit owner acceptance of any residual).
  • Update docs/gmail-tools-daily-agent-operations.md §Activation checklist blocker 3 (Option B resolved) and docs/priorities.md §P1 (human-supervised — flag, do not self-edit priorities).
  • New/updated ops doc for the claude-infra user and per-agent users; dev_r_services row if a new service identity is introduced.

6. ARCH GATE — human decisions required before implementation

Per the issue (“go through the plan workflow — design → review-plan → implement, not straight to code”) and the CLAUDE.md architecture gate (new architecture), the following need an explicit decision before any implementation issue is opened:

  1. Approve the phased sequencing (0 → 1a → 1b → 2) and confirm Phase 0 + 1a may ship immediately (both low-risk, reversible) while Phase 1b awaits this ratification.
  2. Phase 1b OS-user model: one shared claude-infra user for all infra-* job types (recommended, simplest) vs. per-role users. Confirm the user name and which hosts get it (bms-4 only, or vps-i1 too).
  3. Age-key placement (Phase 1b): does ~/.age/p24-infra-keys.txt also move off claude-runner* (couples to plan-per-role-worker-credentials.md, widens scope) or stay for now (narrower, but the SOPS blast-radius half of C-R6(b) remains open)?
  4. Phase 0 bot-PAT scope + SOPS home: confirm AI-Dev-BMS4-1 scopes and the SOPS file, and that no owner-scope automation (runner registration, org ops) depends on the current gh login.
  5. Issue split: close #4815 on Phase 0+1a+1b, or keep it open through Phase 2? (Recommend: split Phase 2 to a child issue if the OS-user work is large.)

7. Out of scope (explicit)

  • Do not touch the --allowedTools allowlists from #4724 — a separate, complementary control that stays regardless (issue non-goal).
  • Do not re-plan the git-over-HTTPS App-token migration — owned by 4065.
  • No credential move happens in this PR. This is the design; execution is delegated to sys-admin
    • secret-manager via follow-up implementation issues once §6 is decided.

Role: dev-coder (design/scoping only; execution delegated to sys-admin + secret-manager). Auto-generated plan — review via /review-plan 4815 before opening implementation issues.

Graph View

Backlinks