Plan 4556 — Role- and repo-scoped SOPS secret access
Issue: #4556
Status: Draft — awaiting review
Author: worker agent (dev-issue, vps-i1)
Created: 2026-07-29
Role note: This document is a design artifact only. Every live secrets/*.env.sops
write described here MUST be executed by /role-secret-manager (or filed as an
infra-task for a sys-admin/secret-manager session). A generic worker never re-encrypts
a SOPS file.
0. TL;DR
Today, one age keypair per machine grants that machine decrypt access to ~24 of the 26
secrets/*.env.sops files. bms-4 — the most autonomous, least-reviewed code-execution host
in the fleet — can therefore decrypt MongoDB rs0 admin, Vercel, Wasabi, Supabase
service-role, GitHub App signing keys, and every application repo’s secrets, on any job,
regardless of what that job is for.
This plan replaces per-machine keys with one age keypair per SOPS-file scope, and
delivers each scope key to a worker ephemerally at dispatch time based on the job’s already
tracked role + repo_context. The blast radius of a compromised worker job shrinks from
“the entire estate” to “the one scope that job legitimately needs”.
What this plan does NOT do: it does not reduce what a compromised developer workstation or a compromised GitHub Actions environment can reach. See §11 (Residual risk).
1. Current state — verified, not assumed
All facts below were verified against this branch on 2026-07-29 by inspecting file contents and key names only. No secret value was decrypted, printed, or logged.
1.1 .sops.yaml — three creation_rules
| Rule | Recipients | Files matched |
|---|---|---|
secrets/administration.env.sops | developer key only | 1 |
secrets/pinbox24-radieu-w4-auth.env.sops | developer key only | 1 |
secrets/.*\.env\.sops (catch-all) | 6: developer, AGE_KEY_GHA (CI), bms-4 runner, vps-i1 runner, dev-laptop, vps-h1 runner | 25 |
secrets/ currently holds 27 *.env.sops files. Note this includes the dotfile
secrets/.canary.env.sops, which a plain ls secrets/ hides but which does match the
catch-all regex (^|[/\\])secrets[/\\].*\.env\.sops$ — so it is one of the 25. Any tooling
that enumerates scopes must glob dotfiles too, or it will silently skip it.
Two files carry .sops.yaml comments describing them as “admin-only” —
bms-servers.env.sops and gcp.env.sops — but both fall under the flat 6-recipient rule.
They are deployment-excluded (dropped by secrets-sync.yml’s path filter), not
recipient-isolated. Any host’s universal key decrypts them directly. This is the single
clearest illustration of the gap this plan closes: the policy layer says admin-only, the
crypto layer says everyone.
1.2 Concrete blast radius — secrets/n8n-bms4.env.sops
~68 keys, routinely decrypted by bms-4. Key names present include:
MONGODB_RS0_ADMIN_PASSWORD · MONGODB_RS0_PROMETHEUS_PASSWORD ·
GITHUB_APP_ID · GITHUB_APP_PRIVATE_KEY_B64 · GITHUB_APP_INSTALLATION_ID ·
VERCEL_TOKEN · SUPABASE_SERVICE_ROLE_KEY · SUPABASE_DB_PASSWORD ·
WASABI_ACCESS_KEY · WASABI_SECRET_KEY ·
RADEK_BRAND_LINKEDIN_ACCESS_TOKEN · RADEK_BRAND_MAILERLITE_API_KEY
— alongside the genuine n8n/Redis/Atrax operational secrets that justify the file’s
existence. The RADEK_BRAND_* pair belongs to a different repo entirely.
MONGODB_RS0_ADMIN_PASSWORD is confirmed still present in this file on both this branch
and origin/main, matching the issue’s premise.
1.3 One keypair per machine
ansible/roles/claude-env-sync/tasks/main.yml generates exactly one keypair per host at
/home/claude-runner/.age/p24-infra-keys.txt (age-keygen -o <path>, when: not age_key_stat.stat.exists), and prints only the public key in a debug task. There is no
existing mechanism to scope more narrowly than “this machine”.
1.4 Dispatch already knows role + repo — but SOPS access ignores both
infra-src/meta-dispatcher/src/classify.ts classifies every queued task into
{ role, repo_context, server_preference }, with
DEV_REPOS = { et-operational-platform, art-agency, brandpilot, radekkonarski-personal-brand }.
scripts/queue-dispatcher-loop.py carries those fields through to
scripts/spawn-worker.sh, which receives ROLE ($7), REPO_CONTEXT ($8) and
JOB_PROFILE ($9).
And yet, in scripts/spawn-worker.sh:
export SOPS_AGE_KEY_FILE="${HOME}/.age/p24-infra-keys.txt" # one key, every jobThe SOPS_FILE case statement immediately below maps repo → SOPS file, but it only
exports a path string. Nothing enforces it. A dev-issue job for brandpilot is handed
the universal key and can sops -d any of the 25 catch-all files.
Evidence that the mapping is unenforced: two of its seven entries point at files that do not exist (verified 2026-07-29):
| Mapped repo | SOPS_FILE target | Exists? |
|---|---|---|
radekkonarski-personal-brand | secrets/radekkonarski-brand.env.sops | MISSING |
amazon-kdp-tango | secrets/kdp-tango.env.sops | MISSING |
brandpilot, art-agency, p24-infra, et-operational-platform, whatsup-android-chat-puller | — | OK |
Because nothing validates the path, these fail silently — the job proceeds with the universal key and full catch-all access, which is strictly worse than failing closed. This is a pre-existing bug, not caused by this plan; it should be filed as its own follow-up issue rather than fixed in this doc-only PR. It is recorded here because Phase 4 replaces this exact code path, and the scoped replacement must fail loud where this one fails silent (§7.3).
1.5 The role axis is CI-mediated — and currently bypassable
Five secrets/role-*.env.sops files exist (sys-admin, secret-manager, dev-coder,
dev-reviewer, dev-tester). secrets-sync.yml decrypts them server-side with
AGE_KEY_GHA and scps the plaintext to /opt/p24-infra/{server}/role-{role}.env;
spawn-worker.sh then sources the file matching the job’s JOB_TYPE.
That path is sound. The hole is that all five files also sit under the flat 6-recipient rule, so any host’s universal key can decrypt them directly, bypassing the CI-mediated delivery entirely. Fixing this needs no new keypairs — only recipient removal.
1.6 The ephemeral-delivery precedent already exists
spawn-worker.sh (l.458–487) materializes ROLE_GIT_SIGNING_KEY from the role env into a
600-mode file under ${XDG_DATA_HOME:-$HOME/.local/share}/p24-worker/signing-keys, runs a
usability preflight, hands ownership to the detached worker, and sweeps orphans older than
12h. This is the shape to reuse — including its scar tissue (§7.3).
2. Coordination — what already exists, and what this plan defers to
Searched and confirmed on 2026-07-29:
| Ref | State | Relationship |
|---|---|---|
| #2732 | OPEN | ”Remove duplicate MONGODB_RS0_ keys from n8n-bms4.env.sops (moved to mongodb-bms.env.sops)”. This is the pre-existing execution issue for what the #4556 body calls “Phase 0”. |
| #3178 | OPEN | monitoring.env.sops role-based reorganisation, plan already written at docs/plans/monitoring-env-sops-reorganisation.md. The content axis. |
| #2835 | OPEN, P0 | Full SOPS rebuild — recreate all files with fresh keys. Hard sequencing constraint. |
| #2416 | CLOSED | Per-service credential isolation plan delivered, execution tracked via #2732 / #2728. |
| #4068 | CLOSED, shipped | GitHub App installation tokens — the validated ephemeral-credential precedent. |
2.1 Phase 0 is not re-derived here
The #4556 body asks to “execute that specific move as Phase 0”. #2732 already owns that work and is open. Re-deriving it in this plan would create exactly the duplicate the issue warns against. This plan therefore:
- Declares #2732 a hard prerequisite of Phase 5, and does not restate its steps.
- Contributes one thing #2732 does not currently assert: that after the move, the
recipient set of
mongodb-bms.env.sopsmust be narrowed too — otherwise the credential has moved file but not shrunk blast radius, since both files sit under the same flat rule.
2.2 Sequencing against #2835 (the decision that gates everything)
#2835 (P0) plans to recreate every secrets/*.env.sops with fresh values. A recipient
migration and a full value rebuild both rewrite every file. Running them concurrently risks
a lost-update collision on files touched by both, with the failure mode being an
undecryptable production secret.
Recommendation: fold the recipient change into #2835 rather than racing it. #2835 already
recreates each file from scratch; writing it out under a narrowed creation_rule costs one
extra .sops.yaml edit per file and zero extra sops updatekeys runs. Doing them separately
means encrypting every file twice.
This is the plan’s single biggest open decision and needs a human call before Phase 5 begins. Phases 1–4 (key generation, tooling, verification harness) are **independent of 2835 and can proceed immediately in either case — they add capability without changing any recipient.
3. Threat model — be precise about what is being defended
In scope. A worker job on a dispatch host (primarily bms-4) executes attacker-influenced input — a crafted issue body, a malicious dependency pulled during implementation, a prompt-injected file in a cloned repo — and attempts to read credentials beyond its task.
Why bms-4 specifically. It is the primary dispatch node, runs the least-reviewed code
paths (headless Claude executing arbitrary issue instructions), and holds vps_root_key
(root SSH to the whole fleet — a standing P1 concern in docs/priorities.md).
Out of scope, by design.
- Compromise of the developer workstation (holds the break-glass key).
- Compromise of GitHub Actions (holds
AGE_KEY_GHA). - OS/container-level sandboxing of worker jobs — explicitly out of scope per the issue; tracked as future hardening (§12).
- An attacker with root on a dispatch host during a job that legitimately holds a scope key. Scoping bounds what one job reaches; it does not defend a fully-owned host.
What “shrink the blast radius” concretely means here: a role=dev-coder, repo=brandpilot job moves from 24 files decryptable to 2 (brandpilot.env.sops plus
its role credential set, the latter delivered as plaintext by CI, not decrypted on-host).
4. Target architecture
4.1 Unifying principle
One age keypair per SOPS-file scope, not per host.
The file layout already encodes the right granularity. Reuse it instead of inventing a new taxonomy.
4.2 Recipient formula
For each secrets/<scope>.env.sops, the creation_rule recipients become exactly:
- Developer key — human break-glass identity. Unchanged from today. Present on every rule.
AGE_KEY_GHA— only ifsecrets-sync.ymlactually deploys that file. Files already excluded by its path filter (administration,bms-servers,ovh-api,gcp,whatsup,rotate-secret,pinbox24-radieu-w4-auth) do not get CI as a recipient. This alone is a meaningful reduction: CI currently holds keys to files it never deploys.- That scope’s own dedicated keypair — private half installed only on hosts that
actually run jobs needing it (per
classify.tsrouting: most scopes only ever need bms-4 today).
The four universal per-host keys are removed only after the scope key is installed and verified (§7.2 dual-key window).
4.3 No standing master key for automation
Deliberately no “secret-manager master key” as a permanent recipient on every file. That would relocate the blast-radius problem onto one new identity rather than solve it. secret-manager access splits into two tiers:
| Tier | Who | How access is obtained |
|---|---|---|
Routine — job_type=secret-manager queue rows naming a specific scope (“rotate N8N_ENCRYPTION_KEY”, “add KEY_X to brandpilot.env.sops”) | dispatched worker | Resolves scope from the request, gets the same ephemeral scope key any dev-coder job gets, plus re-encrypt capability for that one scope. |
Cross-cutting — new scope creation, fleet-wide sops updatekeys, executing this migration | human-operated /role-secret-manager session, or CI | Uses the developer key or AGE_KEY_GHA. No new identity. |
Post-migration, no automated or dispatched identity holds standing at-rest access to every file. Only the developer key and CI do — exactly as today, and both are outside this plan’s threat model.
4.4 Role axis: recipient removal only
Role credentials are delivered exclusively via the CI-mediated pre-decrypt-and-deploy path
(§1.5). No live sops -d of a role-*.env.sops file ever happens on a host. Therefore the
five role files need no scope keypair — only removal of the four universal host keys,
leaving developer + AGE_KEY_GHA.
This is the cheapest, lowest-risk, highest-value change in the whole plan. It should ship first (Phase 3) and independently.
4.5 Ephemeral delivery (repo axis)
At dispatch, spawn-worker.sh resolves (role, repo_context, job_type) → scope, materializes
that scope’s private key to a 600-mode file under the existing
${XDG_DATA_HOME:-$HOME/.local/share}/p24-worker/ tree, exports SOPS_AGE_KEY_FILE to point
at it, hands ownership to the detached worker, and the worker shreds it in its own cleanup.
Orphan sweep mirrors the existing 12h signing-key sweep.
The scope private keys themselves are stored at rest in a new
secrets/scope-keys.env.sops — recipients developer + AGE_KEY_GHA only — and deployed by
secrets-sync.yml to each dispatch host as /opt/p24-infra/{server}/scope-keys.env, mode
600, owned by claude-runner. This reuses the exact delivery path already proven for
role-*.env, and means the host no longer holds a broad age key at all — it holds a
bundle from which the spawner extracts one entry per job.
Honest limitation, stated plainly. A host holding the scope-key bundle can read any key in it. This design bounds what a job reaches, not what a host reaches. Reducing the latter requires per-job OS isolation (§12) or a network key broker (§12). This is a real and deliberate limitation of the ephemeral-materialization approach; it should not be described as full isolation. The gain is genuine but bounded: it removes at-rest, always-on, whole-estate access from the process that executes untrusted input, and it makes every scope access an explicit, loggable event (§9).
4.6 New scope: secrets/platform-integrations.env.sops
Extract from n8n-bms4.env.sops the keys that are platform-wide integrations rather than
n8n operational secrets: GITHUB_APP_ID, GITHUB_APP_PRIVATE_KEY_B64,
GITHUB_APP_INSTALLATION_ID, VERCEL_TOKEN, SUPABASE_SERVICE_ROLE_KEY,
SUPABASE_ANON_KEY, SUPABASE_URL, WASABI_ACCESS_KEY, WASABI_SECRET_KEY.
MONGODB_RS0_* is #2732’s business, not this plan’s.
RADEK_BRAND_* — destination must be decided, not assumed. These keys
(RADEK_BRAND_LINKEDIN_ACCESS_TOKEN, RADEK_BRAND_MAILERLITE_API_KEY) belong to
radekkonarski-personal-brand, whose SOPS file lives in that repo, not here —
secrets/radekkonarski-brand.env.sops does not exist in p24-infra/secrets/ (verified
2026-07-29). So there is no in-repo destination to move them to today. Two options, needing
a human call in Phase 5:
- (a) Relocate them to the
radekkonarski-personal-brandrepo’s own SOPS file and delete them here. Cleanest, but a cross-repo secret move — must go through secret-manager and verify no p24-infra consumer reads them first (grep -rl RADEK_BRAND_before removal). - (b) Create
secrets/radekkonarski-brand.env.sopsinp24-infraas a proper scope, which would also makespawn-worker.sh’s existing (currently dead — §1.4) mapping resolve.
Do not silently drop them into platform-integrations.env.sops: they are neither
platform-wide nor n8n-operational, and folding them in would re-create in a new file exactly
the “unrelated repo’s secrets riding along” problem this plan exists to remove.
Leave secrets/n8n-bms4-gh.env.sops alone (GH_TOKEN, GITHUB_PAT_ALL_WRITES,
GITHUB_PAT_P24_INFRA) — already a separate, isolated concern, out of this pilot’s scope.
5. ⚠️ Verified breakage risk — scripts/gh-app-token
This is the finding most likely to break production if Phase 5 is executed naively.
scripts/gh-app-token (208 lines, Python) does not read from SOPS. It reads
GITHUB_APP_ID / GITHUB_APP_PRIVATE_KEY_B64 / GITHUB_APP_INSTALLATION_ID from
$GH_APP_ENV, else from the first of:
ENV_CANDIDATES = [
"/opt/p24-infra/bms-4/.env",
"/opt/p24-infra/monitoring/.env",
]that actually carries GITHUB_APP_PRIVATE_KEY_B64 — i.e. from deployed plaintext env
files. Consumers found: bms-4/git-credential-gh-token.sh,
scripts/setup-git-credential-helper.sh, scripts/spawn-worker.sh, secrets-sync.yml.
Two consequences:
-
Moving the
GITHUB_APP_*keys intoplatform-integrations.env.sopsis not enough.secrets-sync.ymlmust be updated so those keys still land in the deployed env file thatENV_CANDIDATESreads — orENV_CANDIDATES/GH_APP_ENVmust be updated to the new deployed path. If neither happens, token minting breaks fleet-wide, taking git push, the credential helper, and worker GitHub access with it. -
A pre-existing discrepancy, flagged for verification.
scripts/gh-app-token’s docstring states the App keys are deployed to/opt/p24-infra/monitoring/.envon vps-i1. Butsecrets/monitoring.env.sopscontains zeroGITHUB_APP_*keys, and no workflow, template, orrole-*.env.sopsfile references them — they exist only inn8n-bms4.env.sops. So on vps-i1 neither candidate should carry the key. Scope of this check: SOPS sources and repo files only. The live server file was not inspected from this worker. A sys-admin session should confirm whethergh-app-tokencurrently works on vps-i1 before Phase 5 assumes either answer.
Phase 5 gate: scripts/gh-app-token --check must pass on every dispatch host both
before and after the move, with the before-result recorded in the migration log.
6. Phases
Phases 1–4 add capability and change no recipients — they are safe to run regardless of the #2835 decision. Phases 5–8 change recipients and are gated on §2.2.
| Phase | Work | Role | Gated on |
|---|---|---|---|
| 0 | MongoDB admin credential move | secret-manager | **Deferred to 2732 |
| 1 | Generate scope keypairs; create secrets/scope-keys.env.sops; extend claude-env-sync to install the bundle; register new scopes in dev_r_services | secret-manager + dev-coder | — |
| 2 | scripts/verify-sops-scope-isolation — pass/fail matrix per (key, file), values never printed | dev-coder | — |
| 3 | Role-file recipient narrowing — drop the 4 universal host keys from the 5 role-*.env.sops rules | secret-manager | Phase 2 |
| 4 | spawn-worker.sh scope resolution + materialization + preflight + shred; queue-dispatcher-loop.py passes scope through | dev-coder | Phase 1 |
| 5 | Pilot — brandpilot.env.sops narrowed end-to-end; create platform-integrations.env.sops | secret-manager | §2.2, #2732, §5 gate |
| 6 | Pilot verification — positive + negative decrypt, live job run | sys-admin | Phase 5 |
| 7 | Roll out to remaining scopes, one file per PR, visible checklist | secret-manager | Phase 6 |
| 8 | Migration log; wire Phase 2 script into CI on every .sops.yaml change | dev-coder | Phase 7 |
| 9 | Decrypt-event audit logging + staleness detection | dev-coder | Phase 8 |
6.1 Phase 3 detail — the quick win
For each of the five secrets/role-*.env.sops, add an explicit creation_rule above the
catch-all with recipients developer + AGE_KEY_GHA, then sops updatekeys --yes that file.
Verification: on bms-4, sops -d secrets/role-dev-coder.env.sops must fail with “no matching
recipient” (exit code + stderr text only — never the decrypted body). secrets-sync.yml must
still deploy all five successfully, since CI retains its recipient.
Rollback: re-add the host keys to the rule, sops updatekeys, push. One commit, one minute.
6.2 Phase 5 detail — pilot on brandpilot.env.sops
Chosen because it is a dev repo (exercises the repo_context axis end-to-end), it is
deployed by CI (exercises the AGE_KEY_GHA retention path), and it is low-traffic —
a mistake does not take production monitoring or MongoDB down.
Ordering within the phase is non-negotiable:
- Add the
brandpilotscope key as an additional recipient.sops updatekeys. Canary. - Verify a real
repo=brandpilotjob decrypts via the scope key. - Only then remove the four universal host keys.
sops updatekeys. Canary. - Verify again, positive and negative.
Steps 1 and 3 are separate commits, separate PRs, separate reviews. Never one batch edit.
Rollback (applies to every recipient-removal step, Phases 5 and 7 alike). The removal commit is always revertible in isolation, because the addition of the scope key landed in an earlier, separate commit:
- Re-add the removed recipients to that file’s rule in
.sops.yaml(orgit revertthe removal commit — it touches only.sops.yamlplus the re-encrypted file). sops updatekeys --yes secrets/<file>.env.sops.- Canary:
sops -dthe file, assert exit code 0 only — never print the output. - Push. Distribution follows the normal
secrets-sync.ymlpath.
The developer key is never removed from any rule at any point in this migration, so break-glass recovery from the workstation is always available even if a rollback is itself botched. If a file is discovered undecryptable on a host mid-job, the immediate mitigation is this rollback — not a re-key — because the plaintext values are unchanged throughout; only recipients move.
7. Risks and mitigations
7.1 Lockout during transition
Risk: a recipient removed before its scope key is installed → file undecryptable on that host, mid-job.
Mitigation: mandatory dual-key window per file (§6.2). Every removal is its own reviewed PR. The developer key is never removed from any rule, so break-glass recovery is always available from the workstation.
A file not yet migrated behaves exactly as today — strictly no worse. This must be stated in the migration log so partial rollout is never mistaken for partial breakage.
7.2 Migration stall after the pilot
Risk: Phase 6 succeeds, Phase 7 never completes, most of the estate stays on the broad model while the pilot PR makes it look done.
Mitigation: Phase 7 is a visible checklist — one task-list line per file on the tracking issue, one PR per file. The Phase 2 verify script prints a per-file migrated/not-migrated column so incompleteness is machine-visible, not a memory item.
7.3 Ephemeral-delivery fragility — inherited, not hypothetical
The ROLE_GIT_SIGNING_KEY pattern being reused has already caused two production incidents:
- #3385 — the spawner’s
EXITtrap deleted the key before the detached worker read it. Applied to age keys, the equivalent is a worker’ssops -dfailing mid-job because its scope key was already shredded. - #3540 — a malformed / passphrase-encrypted key silently broke signing with no clear error. The age equivalent is worse: a corrupt scope key degrades to “no secrets”, which a worker may misread as “this job has no secrets” and proceed with wrong behaviour rather than failing.
Mitigation:
- Copy the ownership-handoff discipline exactly: no
EXITtrap, narrowTERM/INTtrap cleared after successful spawn, persistent XDG dir (not/tmp, which is subject tosystemd-tmpfilesGC), 12h orphan sweep. - Add an age preflight analogous to the existing
ssh-keygen -y -P ''check (searchspawn-worker.shfor that literal string rather than a line number — line numbers drift). Validate the materialized key by attempting a decrypt of a tiny known-good canary file for that scope, asserting exit code only. - Fail loud, never silent. If the scope key is missing or invalid, the spawner must
refuse to start the job and mark the queue row failed with a distinct
failure_reason— never fall through to “running without secrets”. The existingSOPS_FILEfallback already logs[WARN] … proceeding without secrets; that permissive behaviour must not be carried into the scoped path.
7.4 Re-widening regression
Risk: a rule silently broadens after the migration. Two distinct routes, both real:
a later unrelated .sops.yaml edit that re-widens a recipient list; or a path-string
mismatch that causes a narrower file’s content to match the broad catch-all rule instead —
which is exactly what #3275 was
(sops-set.ps1 temp-file naming caused administration.env.sops content to be re-encrypted
under the wrong, broader rule). More narrow rules means more surface for that second route.
A one-time Phase 6 test proves isolation then, not later.
Mitigation: Phase 8 wires scripts/verify-sops-scope-isolation into CI as a required check
on every .sops.yaml change. Continuous enforcement, not a point-in-time fact.
7.5 Key sprawl / zombie scope keys
Risk: ~6 keys becomes ~20–30. Two failure modes: (a) a key generated in Phase 1 but never wired into the spawner lookup — a dead recipient sitting on a file forever; (b) a retired repo’s scope key never revoked.
Mitigation: Phase 2’s script cross-checks every .sops.yaml recipient against the
spawner’s lookup table and flags orphans. Phase 9’s audit logging doubles as staleness
detection — flag any scope key with zero decrypt events over N months for review.
7.6 Collision with #2835
Covered in §2.2 — the gating decision.
8. Secret-handling rules for the implementation itself
The migration must not itself become an exposure vector. Non-negotiable, every phase:
- Key generation (Phase 1):
age-keygen -o <path>writes private keys directly to disk. Only the public key may be echoed, logged, or reported (age public keys are non-secret, like an SSH pubkey). No task may pipe a private key file through a step that could echo it. - Canary / decrypt verification (Phases 0, 2, 5, 6, 7, 8): assert on exit code and
stderr text only (“no matching recipient” vs success). Never
sops -d | Out-String, nevercat/Readdecrypted plaintext, never pipe decrypted content into a log, GH issue, or PR comment. - Negative-decrypt tests (Phase 6): capture exit code / stderr only. If the attempt
unexpectedly succeeds — the exact bug the test exists to catch — the failure report
must say
UNEXPECTED SUCCESS — scope leak (file: X, key-id: Y)and still not print the decrypted content. - Verify script output (Phase 2): a matrix of pass/fail marks per (key, file) pair.
Never a decrypted
KEY=valueline. - Migration log (Phase 8): file names, dates, which recipients were added/removed, who performed the step. Never key values.
- PR / issue / commit text throughout: reference credentials by key NAME only
(
MONGODB_RS0_ADMIN_PASSWORD,VERCEL_TOKEN). Never a value, not even truncated. - Windows-side edits:
[System.IO.File]::WriteAllText($path, $content, [System.Text.UTF8Encoding]::new($false))only — never>,Out-File, orSet-Content(CRLF/BOM corruption). Canary-decrypt before everygit add. Temp files insidesecrets/MUST be named*-tmp.env.sops— SOPS matchescreation_ruleson the input file path string, so a temp file named otherwise gets encrypted under the wrong rule. Seedocs/playbooks/sops-edit-operations.mdanddocs/playbooks/sops-windows-crlf.md. - If a secret value ever appears in a log, terminal, commit, or chat output: stop,
do not continue the migration step, follow
docs/playbooks/static-api-key-incident-rotation.mdbefore resuming.
9. Verification
Phase 2 script — scripts/verify-sops-scope-isolation. For each (scope key, SOPS file)
pair, assert decrypt succeeds where the matrix says it should and fails where it should not.
Output is a pass/fail matrix only. Exit non-zero on any deviation. Also flags (a) recipients
present in .sops.yaml but absent from the spawner lookup, (b) the converse.
Per-file cutover checks (Phases 5–7). Before removal: scope key decrypts (exit 0). After
removal: each removed host key fails (non-zero + “no matching recipient”); scope key still
succeeds; secrets-sync.yml still deploys the file if it is a deployed file; a real
dispatched job of the matching (role, repo) completes.
Continuous (Phase 8). The Phase 2 script runs in CI on every .sops.yaml change.
Phase 5 additional gate. scripts/gh-app-token --check passes on every dispatch host
before and after, per §5.
10. Compliance
Per CLAUDE.md §Compliance, each new credential-bearing element is registered in Supabase
dev_r_services with compliance_workbook='yes' and a workbook_url, as part of the
phase that creates it — not afterwards:
secrets/platform-integrations.env.sops(Phase 5)secrets/scope-keys.env.sops(Phase 1)- each per-scope keypair (Phase 1)
10.1 Error Notification Standard — mandatory for the new automations
Per CLAUDE.md §Error Notification Standard, every script, GH Action, cron job, and
automation must, on error: (1) send a Discord embed via
P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL, and (2) create a GH Issue in
radieu/p24-infra labelled bug. This plan introduces three such automations, and each
must ship with that error path in the same PR that creates it:
| Automation | Phase | Error path required |
|---|---|---|
scripts/verify-sops-scope-isolation | 2 | Discord + GH issue on any isolation-assertion failure |
CI check on .sops.yaml changes | 8 | Discord + GH issue on check failure |
| Decrypt-event audit logging + staleness detection | 9 | Discord + GH issue on ingestion failure and on a detected scope leak |
Constraint that overrides the convenience of a verbose alert: the error payload carries
file names, key NAMES, and age key IDs only — never a decrypted value, never a partial
one. A scope-leak alert reports which (key, file) pair unexpectedly decrypted, not what
it decrypted. Otherwise the notification path becomes precisely the exfiltration channel §8
exists to prevent, and Discord + a public-ish GH issue are worse places to leak than a local
log. Follow the existing pattern in scripts/backup-git-mirror.sh / scripts/alert-ingest.py.
Ops documentation lands in docs/playbooks/ alongside Phase 8’s migration log.
docs/playbooks/secret-repo-access-matrix.md — the policy layer this plan adds crypto
enforcement beneath — is updated in Phase 7 to record, per repo, which scope key backs the
policy row. docs/playbooks/secret-rotation-access-matrix.md and
docs/playbooks/credential-rotation-180d.md gain the new scopes’ rotation entries.
11. Residual risk — what this plan does NOT close
Stated plainly so this is not mistaken for a complete fix:
- The developer’s age key remains broadly scoped, by design. It is the break-glass identity every scope still trusts. A compromised developer workstation reaches everything — before and after this migration, unchanged.
AGE_KEY_GHAremains broadly scoped across all CI-deployed files. A compromised GitHub Actions environment reaches every deployed secret. Narrowed slightly by §4.2 rule 2 (CI drops off files it never deploys), but not meaningfully reduced.- A dispatch host holding the scope-key bundle can read any key in it (§4.5). Job-level scoping is bounded by host-level trust.
- A fully-compromised host during a legitimate job still reaches that job’s scope.
- This does not fix the
vps_root_keyco-location concern on bms-4 (standing P1 indocs/priorities.md) — a separate credential surface.
What it does close: at-rest, always-on, whole-estate decrypt access from the process that executes the least-reviewed, most autonomous code in the fleet. That is the actual threat model driving the issue, and it is a real reduction — not a complete one.
12. Out of scope / future hardening
- Full OS/container-level per-job sandboxing — explicitly out of scope per the issue. File as a separate follow-up; it is the only thing that closes §11 item 3.
- Network key broker — a service that hands a job a decrypted subset rather than a key, removing the bundle from the host entirely. Larger change; the natural successor if §11.3 proves unacceptable.
- Narrowing
AGE_KEY_GHAvia per-workflow OIDC-scoped keys.
13. Open decisions for the reviewer
- §2.2 — fold into #2835 or run separately? The plan’s biggest sequencing question. Recommendation: fold in. Needs a human call before Phase 5.
- §4.5 — is host-level bundle access acceptable as the stopping point for this iteration, with per-job sandboxing deferred?
- §5 — does
gh-app-tokencurrently work on vps-i1? Needs a sys-admin check before Phase 5 proceeds on either assumption. - §6.1 — ship Phase 3 (role-file narrowing) immediately and independently? It is low risk, high value, and unblocked by everything above. Recommendation: yes.
- §4.6 — where do
RADEK_BRAND_*go? Option (a) cross-repo relocation, or option (b) create the scope here. Needed before Phase 5 touchesn8n-bms4.env.sops.
14. Follow-up issues to file (not fixed by this PR)
| What | Why separate |
|---|---|
Two dead SOPS_FILE mappings in scripts/spawn-worker.sh (§1.4) | Pre-existing bug, unrelated to this design; needs a code PR, not a doc PR. |
| Full OS/container per-job sandboxing (§12) | Explicitly out of scope per the issue. |
Confirm scripts/gh-app-token behaviour on vps-i1 (§5) | Needs a sys-admin session with live server access. |