role-pat-to-github-app

Plan: replace the role-scoped GitHub PATs with dedicated GitHub App installation tokens

Status: investigation + human checklist. Code (section 5 steps 3-4) implemented behind a default-OFF flag (feat/7481-role-app-profiles): gh-app-token --app sysadmin|secretmgr, and spawn-worker.sh P24_ROLE_APP_TOKENS=1 (see “Implementation status” under section 5). The Apps, their SOPS files and steps 1, 2, 5, 6 are NOT done. Tracks: #7481 (follow-up effort named in its last paragraph), plan #4065 (Wave A), #7478 (p24-reviewer — the pattern to copy) Scope: ROLE_SYS_ADMIN_GITHUB_CLASSIC_PAT (secrets/role-sys-admin.env.sops) and ROLE_SECRET_MANAGER_GITHUB_PAT (secrets/role-secret-manager.env.sops) Hard rule: NEVER reuse App 2109526 (pull_requests / contents / administration / secrets / workflows write). Each role gets its own NEW App with least privilege.

Key names only. No secret value was read or decrypted while writing this: key names come from grep -o '^[A-Z_]*=' on the SOPS ciphertext (dotenv SOPS keeps key names in the clear) and from plain-text repo grep. Anything not provable that way is marked UNVERIFIED.

---

0. Headline findings

1. No code reads either PAT by name. Plain-text grep for ROLE_SYS_ADMIN_GITHUB_CLASSIC_PAT / ROLE_SECRET_MANAGER_GITHUB_PAT finds only docs, the key templates, one test string and one documentation PowerShell fragment (docs/playbooks/sops-autonomous-reset.md:89, assigns $env:SM_GH_PAT which nothing else in the repo reads). spawn-worker.sh merely sources the whole role env file (scripts/spawn-worker.sh:727-737) so the variable lands in the worker environment, but gh does not use it: gh reads GH_TOKEN / GITHUB_TOKEN. So today’s worker GitHub identity is not these PATs — it is GH_TOKEN from the env files / default App 2109526 token (scripts/spawn-worker.sh:286-300, gh auth git-credential helper at :337,:354) or the claude-runner OS user’s stored gh auth login. The role PATs are effectively dormant credentials that sit decrypted on up to three hosts — pure liability. Consequence: “replacing” them has two parts that must not be confused: (a) wiring the roles to a dedicated App token so they stop borrowing App 2109526 / the shared GH_TOKEN (new code, section 5), and (b) deleting the dormant PATs (revocation, section 6).
2. The only documented hard dependency on the sys-admin classic PAT is admin:ssh_signing_key (registering the role commit-signing public keys on the user account, docs/plans/plan-per-role-worker-credentials.md:49,111-116,164). An installation token cannot call POST /user/ssh_signing_keys (user-identity-only). Keys are already registered (IDs 1037694-1037698), so this only bites at the next signing-key rotation -> keep a human step.
3. The project scope is unused by anything in the repo (grep for gh project|projectV2|read:project outside docs: 0 hits). Do not carry it over. If it is ever needed, user-owned Projects v2 are not reachable by an installation token.
4. secret-manager’s only GitHub write-sensitive need is gh secret set on radieu/p24-infra (infra/agent-prompts/worker-secret-manager.md:702; scripts/rotate/*.ps1 likewise, repo always radieu/p24-infra). That is secrets:write — the single highest-risk permission in this plan; the new App must be installed on p24-infra only and key-isolated from every other role.
5. No hard job runtime cap was found (systemd-run --scope sets only MemoryMax/CPUWeight, scripts/spawn-worker.sh:1239-1242; no RuntimeMaxSec/timeout). A worker holds one environment for its whole life, so a token exported once at spawn dies after 60 min. See section 4.

---

1. Inventory

1.1 Where the two keys and their env files live

Item	Location (file:line)
Key ROLE_SYS_ADMIN_GITHUB_CLASSIC_PAT	secrets/role-sys-admin.env.sops:1 (the file holds only this key + ROLE_GIT_SIGNING_KEY)
Key ROLE_SECRET_MANAGER_GITHUB_PAT	secrets/role-secret-manager.env.sops:1 (file also holds ~20 other admin creds: Supabase mgmt token, Mailgun/OpenAI/Sentry admin, CF global key, Wasabi admin, IONOS, MongoDB rs0 admin, VPS SSH key, GCP SA JSON, ConvertAPI, n8n API key, GITLAB_SECRET_MANAGER_PAT)
Key templates (tier 3, “rotate annually”)	docs/sops-templates/role-sys-admin.keys:11, docs/sops-templates/role-secret-manager.keys:11
Design / scopes as originally planned	docs/plans/plan-per-role-worker-credentials.md:40-41,49,53 (note: plan says secret-manager PAT is fine-grained, secrets:write on p24-infra while the template says classic — UNVERIFIED which it really is; check via section 6 step 0)
Playbook map	docs/playbooks/secret-manager.md:134-135
Other mentions	docs/plans/monitoring-env-sops-reorganisation.md:89, docs/playbooks/sops-autonomous-reset.md:89,393, scripts/tests/test_log_op.py:119 (redaction test string only)
SOPS recipient rules	.sops.yaml:135 (role-sys-admin), .sops.yaml:151 (role-secret-manager)

1.2 How the env files reach workers

Step	Evidence
secrets-sync.yml triggers on both files	.github/workflows/secrets-sync.yml:201-202,225-226,243-244
vps-i1: decrypt + scp to /opt/p24-infra/vps-i1/role-{sys-admin,secret-manager}.env	secrets-sync.yml:438-451,512-515,539-543
bms-4: decrypt + install 0640 root:claude-runner at /opt/p24-infra/bms-4/role-{sys-admin,secret-manager}.env	secrets-sync.yml:1339-1352,1415-1418,1477-1480
vps-h1 copies are checked for permissions	scripts/check-env-file-perms.sh:98-99,104-105 (whether CI still deploys them to vps-h1 is UNVERIFIED; vps-h1 is not a worker host since 2026-08-06)
job_type -> role mapping and source of the file	scripts/spawn-worker.sh:713-737 (`infra-task
Same mapping for scope-key audit	scripts/lib/secret-scope-map.json:33-37
Dev workstation (Windows) read path	docs/playbooks/sops-autonomous-reset.md:89 (doc fragment only)

Conclusion: the PAT is exported into the worker env as a variable no tool consumes. Worker gh/git identity comes from GH_TOKEN/gh-app-token (scripts/spawn-worker.sh:286-300,335-354).

1.3 GitHub operations each role actually performs

sys-admin (job types infra-task, infra-alert, nc-alert-batch; prompts infra/agent-prompts/infra-task-request-worker.md, .claude/agent-prompts/roles/sys-admin.md):

Operation	Evidence
gh issue view / comment / edit (labels) / close / list on radieu/p24-infra	infra-task-request-worker.md:110-157,237-265,354-355,517-532,591-614 ; roles/sys-admin.md:87 (gh issue create --label bug)
Same on a return_to_repo (requester’s repo — et-operational-platform, brandpilot, Art-Agency, whatsup, …)	infra-task-request-worker.md:265,540,550,614
Branch push + gh pr create to main (n8n workflow snapshot sync)	infra-task-request-worker.md:405-420
git clone / fetch / push over HTTPS via gh auth git-credential	scripts/spawn-worker.sh:335-354
Commit signing with ROLE_GIT_SIGNING_KEY (SSH key, not a GitHub API op)	scripts/spawn-worker.sh:924-973
Error-notification issue creation	CLAUDE.md §Error Notification Standard, scripts/spawn-worker.sh:1108-1114
gh pr merge	not performed by the infra-task prompt (grep: 0 hits)
gh workflow run / gh run / gh api repo admin	not in the infra-task prompt (0 hits); ad-hoc use by a live worker session is UNVERIFIED
Register SSH signing public keys (POST /user/ssh_signing_keys)	planning-time only, plan-per-role-worker-credentials.md:111-116; user-identity-only
GitLab (bms-1 W3/W4)	uses GITLAB_* PATs, not GitHub — out of scope

secret-manager (job type secret-manager; prompts infra/agent-prompts/worker-secret-manager.md, .claude/agent-prompts/roles/secret-manager.md):

Operation	Evidence
gh issue view/comment/edit/close/create on $ORIGIN_REPO (any requester repo) and radieu/p24-infra	worker-secret-manager.md:57-63,206,252,296,359,437-446,499-505,523,1050-1064,1092-1107
gh pr list + gh pr merge --squash --delete-branch of the CI rotator PR (best effort, no --admin)	worker-secret-manager.md:199,285-292
Commit SOPS change + git push origin HEAD of a rotation branch; rotation-log commit + push	worker-secret-manager.md:675-681,1022-1030
gh secret set KEY --repo radieu/p24-infra	worker-secret-manager.md:699-703,989; scripts/rotate/rotate-now.ps1:243, update-sops-keys.ps1:142, n8n-bms4-api-key.ps1:151 (all hard-coded to p24-infra)
gh api repos/radieu/p24-infra/contents/<playbook> (read docs)	worker-secret-manager.md:348-352,915
gh secret list gap audit (manual)	worker-secret-manager.md:1114
Rotating the GitHub PAT family itself (GH_TOKEN, GITHUB_PAT_*) with Playwright + TOTP	worker-secret-manager.md:868-995 — browser/user-identity, not an API op
gh pr review / approve	not performed (reviewer App #7478)
Vercel / Supabase / Mailgun etc.	not GitHub — out of scope
gh secret set for other repos (et-op, brandpilot)	none found in scripts/prompts; those go to Vercel. UNVERIFIED for ad-hoc rotations

---

2. Operation -> minimal App permission mapping

Installation tokens support per-request narrowing (--permissions, --repos in scripts/gh-app-token). The App’s installed grant is the ceiling; keep it equal to the union below and narrow further per call.

2.1 sys-admin App (p24-sysadmin-worker)

Operation	Repository permission	Repos
issue comment/edit/close/create/list, labels	issues: write	p24-infra + each repo that can be return_to_repo
PR create (and comment)	pull_requests: write	p24-infra
git clone/fetch/push of a branch	contents: write	p24-infra (writes); other repos read-only — see note
mandatory baseline	metadata: read	all installed
editing files under .github/workflows/** in a push (e.g. a fix touching workflows)	workflows: write	omit by default; add only if a recurring need is proven. A push touching workflow files is rejected without it — acceptable fail-safe

Do not grant: administration, secrets, actions: write (no gh workflow run use found), pull_requests approvals are not needed, environments, organization_*.

Per-repo split for cross-repo callbacks: an App permission is per-App not per-repo, but the install scope is per-repo. Install on p24-infra plus the explicit list of requester repos. Per-repo restriction of contents: write is then done at mint time with --repos p24-infra --permissions contents:write,issues:write,pull_requests:write, and --repos <return_to_repo> --permissions issues:write for callbacks.

2.2 secret-manager App (p24-secretmgr-worker)

Operation	Repository permission	Repos
gh secret set / list (repo Actions secrets)	secrets: write	p24-infra only
rotation branch push + rotation-log commit	contents: write	p24-infra
open/merge rotation PR	pull_requests: write	p24-infra
issue callbacks (origin repo) and gap issues	issues: write	p24-infra + requester repos
mandatory baseline	metadata: read	all installed

Do not grant: administration, workflows, actions, environments, variables, dependabot_secrets. Prefer two mint profiles from the same App at runtime so the dangerous permission is only held while needed: callbacks mint issues:write on the origin repo; the gh secret set step mints secrets:write on p24-infra for that single call (the helper already supports --repos/--permissions, never cached).

2.3 What an installation token CANNOT do (needs a different answer)

Capability	Why	Answer
Register / rotate SSH signing keys (/user/ssh_signing_keys), SSH auth keys, GPG keys	user-identity endpoints; Apps have no user token for these	Keys already registered. Future rotation = human step, or a short-lived fine-grained PAT with the “SSH signing keys” user permission (UNVERIFIED that this permission exists for fine-grained PATs — check at github.com/settings/personal-access-tokens/new; if not, a classic PAT created just-in-time and deleted after)
Create / regenerate / revoke PATs and Apps themselves; generate App private keys	browser + 2FA only	Stays Tier 3 / existing Playwright+TOTP path (worker-secret-manager.md:868-995); App private-key rotation is a human action (plan #4065 §4)
User-owned Projects v2 (project scope)	installation tokens have no user-project access	not used today; if ever needed, human or user PAT
Repos on another owner/org	installation is bound to the radieu account	UNVERIFIED whether any return_to_repo is outside radieu/* (all repos cited in docs are radieu/*); a repo outside the owner needs its own installation or a PAT
Branch-protection bypass / gh pr merge --admin	needs administration (forbidden) and bypass-list membership	secret-manager’s rotator-PR merge is already “best effort” (worker-secret-manager.md:292); if blocked it reports to a human. Do not add the App to the bypass list
Counting as the PR approver	a PR pushed by App X cannot be approved by App X (require_last_push_approval)	unchanged; approvals stay with p24-reviewer (#7478)
Commit “Verified” badge for App-pushed HTTPS commits	commit signature comes from the git user.signingkey, matched to the user’s registered key, independent of the push token	unchanged by this migration (author email noreply+... may already show Unverified — UNVERIFIED)
Fine-grained-PAT-only features (e.g. a user’s private-repo ACL differences)	n/a	none identified
Org-level secrets/variables/runners	account is personal (radieu), none in use	n/a

---

3. Dedicated App identities (names, scope, storage)

	sys-admin	secret-manager
GitHub App name	p24-sysadmin-worker	p24-secretmgr-worker
Repository permissions	Issues RW, Pull requests RW, Contents RW, Metadata R	Secrets RW, Contents RW, Pull requests RW, Issues RW, Metadata R
Webhook	Active unchecked	Active unchecked
Where installable	Only on this account	Only on this account
Install scope	”Only select repositories”: p24-infra + requester repos (UNVERIFIED list — derive from return_to_repo values actually used; start with p24-infra, et-operational-platform)	“Only select repositories”: p24-infra for secrets; requester repos only if a callback to them is wanted (otherwise route callbacks via sys-admin or the reviewer App)
SOPS env key names	SYSADMIN_APP_ID, SYSADMIN_APP_PRIVATE_KEY_B64, SYSADMIN_APP_INSTALLATION_ID	SECRETMGR_APP_ID, SECRETMGR_APP_PRIVATE_KEY_B64, SECRETMGR_APP_INSTALLATION_ID

Key naming follows the REVIEWER_APP_* precedent (scripts/gh-app-token:4-6 docstring) because gh-app-token reads three keys per profile; the user-mandated generic names GITHUB_APP_ID / GITHUB_APP_PRIVATE_KEY_B64 / GITHUB_APP_INSTALLATION_ID are already taken by App 2109526 in n8n-bms4.env.sops and /opt/p24-infra/bms-4/.env, so reusing them inside a role file would let the default profile (scripts/gh-app-token ENV_CANDIDATES) silently pick the wrong key. Use the prefixed names. (If the generic names are mandatory, the role env file must never be a member of ENV_CANDIDATES/GH_APP_ENV — fragile; not recommended.)

Placement decision (secret-manager to confirm):

• Recommended: two new narrow files, secrets/github-sysadmin-app.env.sops and secrets/github-secretmgr-app.env.sops, 2 recipients (developer + bms-4 infra-task worker key; add lap2 only if it ever runs those roles), excluded from secrets-sync.yml, read by the worker with a direct sops -d and exported into the child env — exactly the github-reviewer-app.env.sops pattern (docs/playbooks/reviewer-github-app.md §3). Avoids shipping an App private key in plaintext to every host that gets the role env file (vps-i1 / vps-h1 / bms-4).
• Alternative: add the three keys into the existing role files. Pro: one file per role as the roadmap states, zero new .sops.yaml rules. Con: secrets-sync.yml writes the whole file in plaintext (0640 root:claude-runner) to bms-4 + vps-i1 (and possibly vps-h1), widening where a sys-admin key could leak, and any source of the file exposes the key to every child process. role-secret-manager.env.sops additionally holds ~20 unrelated admin credentials.

---

4. Long-running job audit (token lifetime = 60 min)

scripts/gh-app-token caches the default (unscoped) token for 50 min (CACHE_TTL, scripts/gh-app-token:57); scoped (--repos/--permissions) tokens are never cached. Rule from docs/playbooks/github-app-token-integration.md:136-140,252-256: mint immediately before use, never persist.

Workflow	Holds a GitHub token for	Risk
Spawn-time clone (spawn-worker.sh:335)	seconds	none
Persistent credential.helper=!gh auth git-credential in the clone (spawn-worker.sh:353-354)	whole worker life — resolves GH_TOKEN of the worker process env at each git call	High if GH_TOKEN is exported once at spawn: any git push / gh call after 60 min -> 401. Needs a dynamic helper: a git credential helper (and gh wrapper) that calls gh-app-token --app <role> per invocation instead of a static env var
sys-admin infra-task: multi-step server work + callback at the end (infra-task-request-worker.md:517-614)	minutes to hours; callback comes last	Medium: the final gh issue comment/close is the call most likely to land after 60 min. Mint at callback time
sys-admin n8n live-fix -> export -> PR (infra-task-request-worker.md:405-420)	tens of minutes	Low-Medium; mint at push time
secret-manager Tier 2 Playwright/TOTP PAT rotation (worker-secret-manager.md:868-995; comment at :945 says the flow “can pause on 2FA/CAPTCHA long enough for a stale-reclaim”)	can exceed 50-60 min between the lock/claim and the closing gh secret set + callback	High: mint a fresh token at each of gh secret set, git push, callback
secret-manager Tier 1 rotation (mint -> SOPS -> git push -> gh secret set -> server env -> callback; worker-secret-manager.md:675-703,1050-1064)	typically < 30 min, but waits on CI (secrets-sync)	Medium; mint per step
secret-manager waiting on the rotator PR merge / human merge	indefinite	do not hold a token across waits; re-mint
Dispatcher-level job duration	no RuntimeMaxSec/timeout found (spawn-worker.sh:1239-1242); stale reclaim keys off heartbeat (60 s loop in the same block)	A silent-hang worker can outlive any token; this is a reason to mint on demand, not at spawn

Design consequence (code, section 5 step 3): do not export a role-App token as a static GH_TOKEN at spawn. Install a tiny wrapper: GH_TOKEN is unset in the worker env; gh is shimmed (or GH_TOKEN is set per call by a helper) to call gh-app-token --app <role> (50-min cache for the unscoped mint is fine and already refreshes on expiry); git uses a credential.helper that prints username=x-access-token / password=$(gh-app-token --app <role>).

---

5. Implementation steps (after the human checklist in section 6A)

All of these are separate PRs/tasks; owners follow CLAUDE.md §Role Enforcement.

secret-manager (Class A, SOPS writes — one task per role file):

1. Create the two narrow SOPS files with the three prefixed keys each (values pasted by the human into sops edit; scaffold with PLACEHOLDER_PENDING_HUMAN_PASTE first, as for the reviewer App). Add the .sops.yaml rules (2 recipients), the CLAUDE.md SOPS-file-map rows, and the exclusion from secrets-sync.yml. Update docs/sops-templates/*.keys. Canary-decrypt before git add.
2. After section 6B verification: remove ROLE_SYS_ADMIN_GITHUB_CLASSIC_PAT and ROLE_SECRET_MANAGER_GITHUB_PAT from their SOPS files (leave all other keys), regenerate/trigger secrets-sync so the plaintext copies on bms-4/vps-i1(/vps-h1) lose the line, then verify with grep -c on key names only.

dev-coder (code): 3. scripts/gh-app-token: add two profiles to APP_PROFILES (sysadmin, secretmgr) with their *_APP_ID/*_APP_PRIVATE_KEY_B64/*_APP_INSTALLATION_ID keys, env_first=True, own cache path, own *_APP_ENV override (mirror reviewer, scripts/gh-app-token:89-109), plus tests next to the existing reviewer tests. (This is the only change needed to the helper; it already supports --check, --repos, --permissions.) 4. scripts/spawn-worker.sh: per role (ROLE_CRED), decrypt the narrow file with the bms-4 worker key, export only the three profile vars, do not export a static GH_TOKEN; install the git credential helper + gh shim from section 4 using gh-app-token --app <profile>; keep the current default-App path as fallback behind a feature flag (rollback, section 7). Note ROLE_CAP_ENFORCE / ROLE_DENY_SOPS guard at :719-726: roles that cannot hold SOPS material must stay unchanged. 5. Worker prompts: in infra-task-request-worker.md and worker-secret-manager.md replace “ambient GH_TOKEN” assumptions with “mint at point of use” for the callback / gh secret set / push steps (section 4). gh secret set call sites in scripts/rotate/*.ps1 run on the Windows dev box and keep the developer’s own gh login — unchanged. 6. Docs: update docs/playbooks/secret-manager.md:134-135, docs/playbooks/github-app-token-integration.md (add the two Apps and their permission grants next to “App 2109526” / “reviewer”), docs/playbooks/github-pat-rotation.md (role PATs removed), docs/playbooks/sops-autonomous-reset.md:89,393 (drop SM_GH_PAT; fix the “use ROLE_SECRET_MANAGER_GITHUB_PAT instead of App” fallback row), docs/plans/plan-per-role-worker-credentials.md §2.2.

sys-admin (live verification, read-only for secrets): see 6B.

Implementation status (code, Partially implements: #7481)

Step	State
3 gh-app-token profiles sysadmin / secretmgr	Done. SYSADMIN_APP_* / SECRETMGR_APP_*, env_first, own cache + *_APP_ENV override. Never falls back to App 2109526: missing creds => exit 1, --check prints no token. Tests: scripts/tests/test_gh_app_token.py.
4 spawn-worker.sh per-call token plumbing	Done, opt-in, default OFF (P24_ROLE_APP_TOKENS=1). scripts/lib/role-app-token.sh (decrypts the narrow secrets/github-{sysadmin,secretmgr}-app.env.sops in a subshell, mints), role-app-gh-shim.sh (gh first on PATH, GH_TOKEN set for the single exec only), role-app-git-credential.sh (prepended to the clone’s credential helpers). No static GH_TOKEN is exported. Only role-sys-admin / role-secret-manager jobs are affected. secretmgr: gh secret ... mints secrets:write on p24-infra for that call only; everything else mints without secrets. Tests: scripts/tests/test_role_app_tokens.py.
1, 2 SOPS files / PAT removal	Not done (secret-manager). Apps do not exist yet, so with the flag on the helpers find no creds and silently fall back to the pre-change identity.
5 prompts, 6 docs	Not done.

Rollback of the code: unset P24_ROLE_APP_TOKENS (or set 0) and re-dispatch; with the flag off the new block is skipped entirely and spawn behaviour is unchanged. Not yet verified live: real mint against the Apps, git push/gh through the shim on bms-4, the sops -d path under the worker’s age key.

---

6. Human checklist

6A. Create the Apps (GitHub UI, account radieu, 2FA required) — repeat for each App

Do sys-admin first, then secret-manager. Have the PEM handling plan ready: nothing is pasted into chat, issues, PR bodies or logs.

1. github.com/settings/apps/new (personal account). Fill:
   • Name: p24-sysadmin-worker / p24-secretmgr-worker
   • Homepage URL: https://github.com/radieu/p24-infra
   • Webhook: uncheck Active
   • Repository permissions: exactly the table in section 3. Everything else No access. (sys-admin: Contents RW, Issues RW, Pull requests RW, Metadata R. secret-manager: Contents RW, Issues RW, Pull requests RW, Secrets RW, Metadata R.) No account or organization permissions.
   • Where can this App be installed? Only on this account
2. Create GitHub App. Note the App ID (General page).
3. General -> Private keys -> Generate a private key. A .pem downloads. (This key is the credential; treat it like the old PAT.)
4. Install App tab -> install on radieu -> Only select repositories -> choose per section 3 (secret-manager: radieu/p24-infra only). After install, read the Installation ID from the URL github.com/settings/installations/<ID>.
5. Produce the base64 value locally (Windows PowerShell, no echo of the result): [Convert]::ToBase64String([IO.File]::ReadAllBytes("<path>.pem")) | Set-Clipboard (or base64 -w0 <path>.pem). Keep it in the clipboard only.
6. Hand the three values to secret-manager via sops edit on the new narrow file (SYSADMIN_APP_* / SECRETMGR_APP_*; section 3) — paste into the editor, do not type them in a shell command or chat. Then delete the .pem (secure-delete / empty Recycle Bin) and clear the clipboard.
7. Record App ID (not secret) in the PR for the SOPS change so the maintainers can map settings/apps/<slug>.
8. Confirm in the App’s Advanced tab that “Recent deliveries” is empty (no webhook) and that Permissions & events shows only the intended grants. Screenshot-free: read it back from the API in 6B instead.

6B. Verification before revoking anything (no secret printed)

• scripts/gh-app-token --app sysadmin --check and --app secretmgr --check on bms-4 (exit 0; the --check mode never prints the token).
• Permission read-back: gh api /app/installations/<id> with an App JWT shows the granted set; compare with section 3 (any extra permission = fix at github.com/settings/apps//permissions).
• Negative tests, expect 403/404: secret-manager App minted with --repos <other repo>; sys-admin App gh secret list; either App gh api repos/radieu/p24-infra/actions/workflows -X POST-style admin calls; a mint requesting administration fails.
• Positive tests: one real infra-task on bms-4 (issue comment, branch push, PR create) and one real secret-manager dry run (gh secret set on a throwaway secret name in p24-infra, then delete it).
• Long-job test: leave a worker idle for >60 min and perform a callback; it must succeed (proves mint-on-demand).
• Prove the dormant-PAT claim before deleting: grep -rn of worker logs / a one-week window with the PAT keys removed from a canary host’s env file, no 401s (or run the X-OAuth-Scopes check: gh api -i /user shows the classic PAT’s scopes; ask secret-manager to run it via scripts/sops-invoke.ps1 so the value is redacted — this also settles the “classic vs fine-grained” question for ROLE_SECRET_MANAGER_GITHUB_PAT).

6C. Revoke the classic PATs (end, human, 2FA)

Only after 6B and one full week of clean worker runs on the App path.

1. secret-manager removes the two keys from the SOPS files and redeploys (section 5 step 2); confirm the plaintext role env files on the hosts no longer carry the names.
2. Human, github.com/settings/tokens (classic) and /settings/personal-access-tokens (fine-grained): find the tokens by their GitHub display names (they are not named after the SOPS keys — see the display-name collision warning in docs/playbooks/github-pat-rotation.md, #6247). Delete the sys-admin classic PAT and the secret-manager PAT. Do not touch GH_TOKEN (p24-bms4-github-rw-key), GITHUB_PAT_*, or the CI-only GH_TOKEN PAT.
3. Check github.com/settings/security-log for the deletions; check Settings -> SSH and GPG keys that the five p24-worker-<role> signing keys (IDs 1037694-1037698) are still registered (they are independent of the PAT).
4. Update docs/sops-templates/*.keys and the access matrix (section 5 step 6) and log the change in the secrets rotation log (docs/ rotation-log convention).

---

7. Rollback

• Phase 1 (Apps created, SOPS files added, code behind a flag): rollback = flag off; workers fall back to the pre-change identity (GH_TOKEN/default App) exactly as today. Apps can stay installed unused.
• Phase 2 (PAT keys removed from SOPS but PATs still alive on GitHub): git revert the SOPS-removal commit and redeploy via secrets-sync — restores the old env file contents with no GitHub change. This is why 6C step 2 (deleting the PATs on GitHub) is last and irreversible by design.
• Phase 3 (PATs deleted on GitHub): not reversible; a replacement PAT must be re-created by a human (Tier 3) and re-added by secret-manager. Mitigation: the dormant-PAT proof in 6B must be complete first. For an immediate emergency (App compromised): Suspend the App at github.com/settings/installations/ (instant, reversible) or delete the private key in the App settings; installation tokens already minted expire within 60 minutes.
• App private key compromise response: generate a new key in the App settings, delete the old one, update the single SOPS file (secret-manager), no other host holds it.

---

7A. Risk acceptance: installation scope = all repositories

Date: 2026-10-08. Decided by: the repository owner (radieu), explicitly, after being offered the “Only select repositories” option from section 3 / 6A step 4. Status: accepted risk, not a blocker; do not re-ask, revisit only when the owner chooses to tighten (below).

	sys-admin App	secret-manager App
App	p24-sysadmin-worker, App ID 5242909	p24-secretmgr-worker, App ID 5242942
Installation ID	169392610	169393430
repository_selection	all (deviates from section 3)	all (deviates from section 3)
Permissions (verified live 2026-10-08 via App JWT, GET /app and GET /app/installations)	Contents RW, Issues RW, Pull requests RW, Metadata R	Secrets RW, Contents RW, Pull requests RW, Issues RW, Metadata R
Events / webhook	none	none

What the deviation means: every repository on the radieu account (including future ones) is reachable by a token minted from either installation, within that App’s permission set. The worst case for the secret-manager App is secrets:write on any radieu repo; for the sys-admin App, contents/issues/PR write on any radieu repo. Neither can reach organisation resources, SOPS-encrypted data, or any GitHub admin surface.

Compensating controls (all verified or enforced in code, 2026-10-08):

1. Per-call narrowing of every mint. scripts/gh-app-token --repos <name> and --permissions k:v produce a token limited to the named repo(s) and the named permission subset (never cached). The role-app shim (scripts/lib/role-app-gh-shim.sh) mints one token per gh invocation and never exports a long-lived GH_TOKEN.
2. secrets:write is minted only per gh secret call, always with --repos p24-infra. The shim maps secretmgr:secret to --permissions secrets:write --repos p24-infra and every other gh or git operation to contents:write,pull_requests:write,issues:write (no secrets). Tested on bms-4 as claude-runner: a secrets:write --repos p24-infra token sees exactly one repository (radieu/p24-infra) via GET /installation/repositories, and a throwaway gh secret set / gh secret delete round trip succeeded. Gap found and tracked: role-app-token.sh itself forwards arbitrary arguments, so a caller that bypasses the shim could still ask for an unscoped secrets:write; the fix is a guard in that helper (reject secrets:write unless --repos is present), see the PR that introduces the helper (#7636).
3. Narrow SOPS files, 3 recipients each. secrets/github-sysadmin-app.env.sops and secrets/github-secretmgr-app.env.sops hold only the three *_APP_* keys; recipients are the developer key, lap2-windows and the bms-4 infra-task worker key. AGE_KEY_GHA and the vps-i1 runner are not recipients; both files are excluded from secrets-sync.yml, self-test.yml and sops-pr-canary.yml, so no plaintext copy of either private key exists on any server.
4. Minimal permission sets. Neither App has administration, workflows, actions, environments, pages, deployments, members or any account permission, and neither subscribes to events. Negative tests on bms-4: minting administration:write and workflows:write both fail (HTTP 422). This is the main limit on blast radius; “all repositories” widens where, not what.
5. Identity separation. Neither App is App 2109526, and the PR approver is a third App (p24-reviewer); main still requires an approving review from an identity other than the last pusher, and the role workers never approve PRs.
6. Instant revocation. Suspend the installation at github.com/settings/installations/ (reversible) or delete the App’s private key; minted tokens expire within 60 minutes.

How to tighten later (no code change needed): open github.com/settings/installations/169392610 (sys-admin) and github.com/settings/installations/169393430 (secret-manager), under Repository access choose Only select repositories, and pick radieu/p24-infra plus the requester repos from section 8 item 2 (secret-manager: radieu/p24-infra only). New tokens honour it immediately; the per-call --repos logic already in place keeps working inside the narrower set.

---

7B. Canary plan and PAT revocation

Prerequisites (human/merge order): PR #7636 (profiles + shim + spawn-worker wiring, flag default OFF) merged; the sysadmin and secretmgr SOPS files merged; the repo checked out at /opt/p24-infra on bms-4 so scripts/lib/role-app-*.sh are executable there. Nothing changes behaviour until the flag is set.

Which host first: bms-4 only. It is the primary dispatch host, the only host whose worker key is a recipient of both narrow files, and the only place the flag can take effect (vps-i1 and vps-h1 are not recipients, so P24_ROLE_APP_TOKENS=1 there degrades to the pre-change identity by design). lap2 stays off until bms-4 has a clean week. Start with the sys-admin profile alone for the first 2 days (lower blast radius: no secrets), then enable secret-manager.

Enable (real mechanism): a systemd drop-in is NOT enough, and neither is anything set only in the dispatcher’s own environment: queue-dispatcher-loop.py spawns workers via ssh root@host "su -s /bin/bash <user> -c 'export ...; exec spawn-worker.sh ...'", and only the variables the dispatcher lists in env_exports reach spawn-worker.sh. The dispatcher now forwards P24_ROLE_APP_TOKENS (default 0) and P24_ROLE_APP_TOKENS_ROLES (default sysadmin,secretmgr) from its own environment (values validated to [A-Za-z0-9_,], anything else falls back to the defaults). The dispatcher unit reads EnvironmentFile=/opt/p24-infra/bms-4/.env, so add these two lines there:

P24_ROLE_APP_TOKENS=1
P24_ROLE_APP_TOKENS_ROLES=sysadmin

/opt/p24-infra/bms-4/.env is rendered by the sync-bms-4 job of secrets-sync.yml from SOPS (primary source secrets/n8n-bms4.env.sops), so a hand edit on the host is overwritten at the next sync. Either make the change in the source SOPS via secret-manager, or add a separate non-synced env file as an additional EnvironmentFile= on the dispatcher unit. Restart the dispatcher unit, then dispatch one infra-task job; the worker log must contain [role-app] per-call App tokens enabled: profile=sysadmin. A secret-manager job in this phase must log profile secretmgr not in P24_ROLE_APP_TOKENS_ROLES and keep the old identity. After ~2 clean days widen to P24_ROLE_APP_TOKENS_ROLES=sysadmin,secretmgr (or drop the line, the default is both) and verify the profile=secretmgr log line on a secret-manager job.

Rollback (seconds): remove the two lines (or set P24_ROLE_APP_TOKENS=0), restart the dispatcher and re-dispatch; workers return to the old identity. Nothing else to revert; the Apps, installations and SOPS files may stay unused. For a suspected compromise additionally suspend the installation (section 7 / 7A item 6).

As enabled (bms-4, 2026-10-09 08:30Z, by sys-admin): the flags live in a separate non-synced file, not in the SOPS-rendered .env:

• /opt/p24-infra/bms-4/role-app.env (root:root 0644, no secrets), exactly P24_ROLE_APP_TOKENS=1 and P24_ROLE_APP_TOKENS_ROLES=sysadmin.
• Drop-in /etc/systemd/system/p24-queue-dispatcher.service.d/role-app.conf: [Service] + EnvironmentFile=-/opt/p24-infra/bms-4/role-app.env (listed after .env and worker-queue.env, so it wins; the leading - makes a missing file harmless). Then systemctl daemon-reload; no restart needed (the dispatcher is a 2-minute oneshot timer, the next cycle picks it up). Original unit backed up at /root/p24-queue-dispatcher.service.bak-20261009.
• Drift risk: no Ansible role manages p24-queue-dispatcher.service (the unit was copied by hand), so neither file is Ansible- or sync-managed. They survive secrets-sync.yml and deploy-vps-scripts.yml but not a host rebuild; re-create them from this section.
• Rollback: rm /etc/systemd/system/p24-queue-dispatcher.service.d/role-app.conf && systemctl daemon-reload (or set P24_ROLE_APP_TOKENS=0 in role-app.env). In-flight workers keep their environment; new jobs revert to the old identity.
• Widen to secret-manager after ~2 clean days: set P24_ROLE_APP_TOKENS_ROLES=sysadmin,secretmgr in role-app.env (no reload needed, the file is read each cycle). Done 2026-10-11 ~07:54Z (by sys-admin): after 7 clean sysadmin jobs (0 mint errors, 0 gh auth errors), the line is now P24_ROLE_APP_TOKENS_ROLES=sysadmin,secretmgr; next dispatcher cycle Result=success. Backup: role-app.env.bak-20261011T0753Z (same dir). File owner/ACL is actually claude-runner:claude-runner 0644 (not root:root as written above). Rollback: cp -a the backup over role-app.env, or set the line back to sysadmin.
• role-dev-* jobs log [role-app] P24_ROLE_APP_TOKENS=1 but not applicable ... default identity; that is expected, only sys-admin and secret-manager roles map to a profile.

What to watch for 7 days:

• Worker logs: the [role-app] ... not applicable and WARN: could not install gh shim lines (a silent fallback to the old identity means the canary is not testing anything), and Mezmo role_app_tokens_enabled counts per profile.
• Any 401/403/404 from gh or git in infra-task and secret-manager jobs; callback comments missing on originating issues; failed gh secret set / rotation PR merges.
• github.com/settings/security-log and each App’s Advanced tab: unexpected API activity, new installations, key creation.
• Commit authorship and signing on pushes from workers are unchanged (the SSH signing keys are independent of the App tokens).
• At least one job that outlives 60 minutes and still posts its callback (proves mint-on-demand).
• Section 6B negative and positive tests re-run once mid-week, and the installation’s repository list still equals the owner-accepted setting.

Revocation of the classic PATs (human, only after a clean week): if none of the above fired for 7 consecutive days with the flag on for both roles, proceed with section 6C: secret-manager removes ROLE_SYS_ADMIN_GITHUB_CLASSIC_PAT and ROLE_SECRET_MANAGER_GITHUB_PAT from their SOPS files, then the owner deletes the two tokens at github.com/settings/tokens and /settings/personal-access-tokens by their GitHub display names as shown in the UI (not by SOPS key name; see the collision warning in docs/playbooks/github-pat-rotation.md). Do not touch p24-bms4-github-rw-key (SOPS GH_TOKEN), GITHUB_PAT_* or the CI-only PAT.

---

8. Open / UNVERIFIED items to settle before step 5

1. Real scopes/type of both PATs (X-OAuth-Scopes) — expected from plan: sys-admin classic repo+workflow+project+admin:ssh_signing_key; secret-manager “fine-grained, secrets:write on p24-infra” vs template “classic”.
2. List of repos that can appear as return_to_repo / $ORIGIN_REPO and whether any is outside the radieu account.
3. Whether live sys-admin sessions run gh workflow run, gh pr merge, or gh api admin calls that are not written into the prompts (grep only sees the committed prompts). Mitigation: 1-week canary window in 6B.
4. Whether secrets-sync.yml still deploys the role env files to vps-h1 (the perms check lists them).
5. Whether lap2 workers (docker-run, ADR-006) run infra-task/secret-manager jobs and therefore need to be SOPS recipients of the new narrow files.
6. Fine-grained PAT “SSH signing keys” user permission availability (for future signing-key rotation).

---

9. Decision: one shared App or two? (2026-10-08, recommendation: TWO)

Question. Can sys-admin and secret-manager share ONE new dedicated App (per-call --permissions narrowing keeps sys-admin away from secrets:write), or do they need two (section 3 as written)?

Recommendation: keep two Apps (p24-sysadmin-worker, p24-secretmgr-worker). No change to PR #7636’s profile names (--app sysadmin, --app secretmgr) or its SOPS files (secrets/github-sysadmin-app.env.sops, secrets/github-secretmgr-app.env.sops).

Why per-call narrowing of ONE App is not enough

1. The private key is the credential, not the token. --permissions only narrows the token that the caller chooses to mint. Whoever can read the App private key can mint any subset of the App’s installed grant. If one App carries secrets:write (needed by secret-manager), then any sys-admin worker that holds that key can mint secrets:write on p24-infra, by a bug, a prompt injection or a hostile infra-task payload. “Sys-admin never holds secrets:write” would then be a convention in our shim, not a property enforced by GitHub. Two Apps make it a GitHub-enforced ceiling: the sys-admin key simply cannot produce that permission.
2. Permissions are per App, install scope is per repo. A single App must carry the union of permissions and be installed on the union of repos. sys-admin needs contents/issues/pull_requests on several requester repos (return_to_repo); secret-manager needs secrets:write on p24-infra only. One App gives secrets:write ceiling on every repo sys-admin is installed on, which widens the highest-risk permission in the plan from one repo to all of them. (Per-call --repos narrows it back only for honest callers - same point 1.)
3. Blast radius of a key leak and of an emergency stop. Two keys, two suspend switches: a leaked or misbehaving sys-admin key (broad repo reach, handles untrusted infra-task text) can be suspended without halting rotations, and a secret-manager incident does not take down all infra-task callbacks. Matches the section 3 key-isolation rule and the reviewer App precedent (#7478/#7625: one purpose per identity).
4. Audit trail. GitHub attributes actions to the App slug. Two slugs make secrets:write activity trivially attributable to secret-manager in the security log; one App mixes them.
5. SOPS recipients. The two narrow files may have different recipient sets (sys-admin may later need lap2; secret-manager likely not). One shared file forces the wider set on the key that matters most.

Where per-call narrowing IS worth using (defence in depth, applies to both designs)

Keep what #7636 already does: secretmgr mints secrets:write only for gh secret ..., scoped --repos p24-infra, never cached; all other calls mint issues/contents only. The two-App split sets the ceiling, per-call narrowing minimises the window inside it.

Operational cost of two (small)

• One extra App creation: the section 6A steps, about 10 minutes of human UI work, done once more.
• One extra narrow SOPS file and .sops.yaml rule (the scaffolding is already in section 5 step 1).
• One extra --check and key-rotation task (Tier 3, once per rotation interval). Rotation of one key never touches the other role.
• No code cost: #7636 already implements both profiles with independent caches and *_APP_ENV overrides.

If one App is chosen anyway (not recommended; exact changes required)

Only acceptable if the human accepts that point 1 is convention-only. Then #7636 would change as follows:

• Create ONE App p24-role-worker (union grant: Contents RW, Issues RW, Pull requests RW, Secrets RW, Metadata R), installed on p24-infra + requester repos.
• One SOPS file secrets/github-roleapp.env.sops with ROLEAPP_ID, ROLEAPP_PRIVATE_KEY_B64, ROLEAPP_INSTALLATION_ID (or keep the two profile names but point both at the same three keys).
• scripts/gh-app-token: profiles sysadmin and secretmgr become aliases of one credential set with separate caches; the secrets:write restriction would have to be enforced in the shim (scripts/lib/role-app-*.sh) by refusing secrets in sys-admin mode, which is again a software guard, not a boundary.
• spawn-worker.sh must withhold the key from sys-admin jobs for secrets:write to mean anything; that is impossible when sys-admin needs the same key for its own tokens, so the guard cannot be made strong.

Decision summary

Criterion	One App	Two Apps
sys-admin provably cannot get secrets:write	no (convention)	yes (GitHub-enforced)
secrets:write reach	all installed repos	p24-infra only
Independent suspend / rotate	no	yes
Extra human work	none	one App creation (about 10 min), one SOPS file
Code changes to #7636	several	none

Tracks #7481 (Partially implements; Apps themselves are still human/secret-manager steps in section 6A).