Plan: reorganize SOPS secrets from host-based to role/project-based files
Issue: #4729
Type: Code-change-design (plan only — do NOT implement the migration in this issue)
Status: Draft — iteration 1
Last updated: 2026-07-30
Author: Claude Opus 4.8 (dev-coder worker, bms-4 — design only)
Related:
#3178 (monitoring.env.sops role split — docs/plans/monitoring-env-sops-reorganisation.md; this plan subsumes it as Track M and does not restate its key-level detail),
#4556 (role-*.env.sops files + narrowed recipients — the model this plan generalises),
#4065 (GitHub PAT / App consolidation — owns the GitHub-token axis; this plan defers all GH_TOKEN/GITHUB_PAT_* moves to it),
#2835 (SOPS rebuild with fresh keys — key rotation, orthogonal to this file-organisation work).
Role note (EU AI Act Art. 13 traceability). This document is a design artifact produced by the dev-coder role. dev-coder
capabilities.cannotincludes writing/rotating/decrypting anysecrets/*.env.sopsfile. Every execution step in this plan (SOPS re-encryption, recipient changes, GH Secrets updates, Vercel env sync, runningsecrets-sync.yml, MongoDB user creation) issecret-managerand/orsys-adminwork and is flaggedhuman-action/ delegated — see §7. No secret value appears anywhere in this plan; only key NAMES and file names are referenced.
0. Summary — recommendation
Adopt a single naming rule and migrate the fleet to it in small, additive-first tracks:
The rule: a SOPS file is named for who manages it (trust tier / role) and what consumes it (service / app / role) — never for the physical host it happens to land on. A host may source several files; that is the intended outcome, not a regression.
Concretely, every secrets/*.env.sops file falls into exactly one of four families:
| Family | Prefix / convention | Managed by | CI-deployed? | Examples (target) |
|---|---|---|---|---|
| A — Admin trust tier | administration.env.sops + admin-* | secret-manager (writes), developer (reads) | No — excluded from secrets-sync.yml | administration.env.sops, admin-server-access.env.sops, ovh-api.env.sops, gcp.env.sops |
| B — Worker roles | role-*.env.sops | secret-manager | Yes → /opt/p24-infra/{host}/role-{name}.env | role-sys-admin, role-secret-manager, role-dev-coder/reviewer/tester (unchanged — #4556) |
| C — Services / stacks | bare service name | secret-manager | Yes → the service’s runtime path | n8n.env.sops, mongodb-rs0.env.sops, monitoring.env.sops (shrunk, #3178), traccar.env.sops, whatsup.env.sops |
| D — Products / apps | bare product name | secret-manager | Yes → Vercel / GitLab / worker bindings | art-agency, brandpilot, et-operational-platform, et-lager, pinbox24-w3/w4/-auth/-backends (unchanged — already the model) |
Families B and D already exist and are the model. The work is confined to the genuinely
host-named files in family C/A — n8n-bms4.env.sops, n8n-bms4-gh.env.sops,
vps-h1.env.sops, bms-servers.env.sops, mongodb-bms.env.sops — plus the two credential
families that host-grouping has scattered (MongoDB rs0, GitHub tokens). monitoring.env.sops is
already fully planned in #3178 and is carried here as Track M without re-deriving its key list.
Ship order (each track is independent; M and G are pre-existing):
Track M monitoring.env.sops role split .............. #3178 (already planned — execute as-is)
Track G GitHub token consolidation .................. #4065 (already planned — owns GH_TOKEN axis)
Track N n8n-bms4.env.sops → n8n.env.sops (+ hand off Mongo/GH keys to Track J/G)
Track J MongoDB rs0 consolidation → mongodb-rs0.env.sops (from bms-servers + mongodb-bms + admin)
Track S bms-servers.env.sops → admin-server-access.env.sops (root-password residue after Track J)
Track H vps-h1.env.sops → audit → archive or rename to real consumer
Rationale: Tracks M and G are already scoped and partly in flight — do not fork them. Tracks N/J/S/H are new and small, each touching one or two files, each shippable behind the additive-first safety rule in §3 with zero decrypt-gap. No track requires an ARCH GATE decision beyond this plan’s approval.
1. Why host-based grouping is the problem (motivating case: bms-3)
The trigger (2026-07-31 bms-3 dispatch-host assessment): bms-3’s root password lives in
secrets/bms-servers.env.sops next to bms-1/2/4 root passwords and the MongoDB rs0
replica-set credentials. That single file therefore spans:
- ≥3 roles — sys-admin (SSH root to each host), secret-manager (rotation), the MongoDB service (rs0 app + admin users);
- ≥2 projects — server access (an ops concern) and the Pinbox24 MongoDB service (an app concern).
Consequences of coarse, host-keyed grouping:
- Blast radius. A breach of one file leaks unrelated trust tiers (root SSH and DB app creds).
This is the same “admin keys in monitoring” anti-pattern #3178 fixed for
monitoring.env.sops. - Access scoping.
.sops.yamlrecipients are per-file. Host files force every recipient of the file to be a recipient of all its unrelated secrets. Role/project files let recipient lists be narrowed to exactly the consumers (as #4556 did forrole-*). - Rotation coupling. Rotating one host’s root password re-encrypts a file that also carries live DB URIs, widening the change surface and the canary/verify blast radius.
- Doc drift. Every
CLAUDE.mdin the ecosystem already describes ownership as “owned end-to-end by X role” / “credentials via SOPS+age:<file>” — a role/project layout, which the host files contradict.
The same scattering afflicts two credential families that no single host owns:
- MongoDB rs0 creds are split across
administration.env.sops(rs0 admin),bms-servers.env.sops(rs0 app users),mongodb-bms.env.sops, andn8n-bms4.env.sops(Prometheus reader user). - GitHub tokens are split across
monitoring.env.sops,n8n-bms4.env.sops,n8n-bms4-gh.env.sops, andet-operational-platform.env.sops(this axis is **owned by 4065 — see Track G).
2. Current inventory → target family (the concrete file map)
Legend: KEEP = already role/project-based (the model, no change); PLAN = covered by an existing plan; MOVE/SPLIT/RENAME/AUDIT = new work in this plan. The contents column references key NAMES/groups only.
2.1 Family D — products/apps (KEEP — already the model)
| File | Verdict | Note |
|---|---|---|
art-agency.env.sops | KEEP | project-based ✓ |
brandpilot.env.sops | KEEP | project-based ✓ |
et-operational-platform.env.sops | KEEP | project-based ✓ (GitHub token axis handled by #4065) |
et-lager.env.sops | KEEP | project-based ✓ |
pinbox24-w3.env.sops (V32_*) | KEEP | service-based ✓ (docs/w3-w4-stack-operations.md) |
pinbox24-w4.env.sops (V42_*) | KEEP | service-based ✓ |
pinbox24-w4-auth.env.sops | KEEP | canonical since #3773 Phase 2 ✓ |
pinbox24-backends.env.sops (mailgun-prod) | KEEP | service-based ✓ |
pinbox24-test-users.env.sops | KEEP | test-fixture scope ✓ |
whatsup.env.sops (p24-wa-a) | KEEP | app-based ✓ (manual sync — #2943) |
radekkonarski-brand.env.sops, gmail-tools.env.sops | KEEP | live in their own repos — already project-based, out of scope for this repo’s secrets/ |
2.2 Family B — worker roles (KEEP — already the model, #4556)
role-sys-admin · role-secret-manager · role-dev-coder · role-dev-reviewer ·
role-dev-tester — all KEEP. This plan generalises their pattern; it does not change them.
rotate-secret.env.sops (rotation tooling) — KEEP (role/tool-scoped).
2.3 Family A — admin trust tier
| Current file | Verdict | Target | Contents (key NAMES only) |
|---|---|---|---|
administration.env.sops | KEEP name | administration.env.sops | IONOS API, CF Global Key, WASABI_ADMIN, GITLAB_ADMIN_PAT, SUPABASE_ACCESS_TOKEN, GH_PAT_ADMIN, MongoDB rs0 admin (→ Track J decides: keep admin-tier here vs. move to mongodb-rs0) |
ovh-api.env.sops | KEEP | ovh-api.env.sops | OVH + SoYouStart write-scope API keys — provider-scoped, dev-only (already excluded from sync) ✓ |
gcp.env.sops | KEEP | gcp.env.sops | GCP_SA_KEY_JSON — project/provider-scoped, dev-only ✓ |
bms-servers.env.sops | SPLIT | admin-server-access.env.sops + mongodb-rs0.env.sops | root part (bms-1/2/3/4 root pw) → admin-server-access; rs0 app-user part → Track J |
bms-servers.env.sops root passwords are correctly one role (sys-admin/admin), so they stay
together — but under a name that says what they are (server root access), not which hosts.
Renaming to admin-server-access.env.sops keeps the dev-only exclusion and makes the trust tier
legible. (Track S.)
2.4 Family C — services/stacks (the host-named residue)
| Current file | Verdict | Target | Contents (key NAMES only) | Track |
|---|---|---|---|---|
monitoring.env.sops | PLAN | shrink per #3178 (→ worker-queue, cloudflare, traccar, role-*, administration) | observability connectors only | M (#3178) |
n8n-bms4.env.sops | RENAME + SPLIT | n8n.env.sops | n8n stack proper: N8N_*, Redis. Hand off: MongoDB Prometheus reader → Track J; GH_TOKEN → Track G | N |
n8n-bms4-gh.env.sops | **DEFER to 4065 | GitHub-token file per #4065 | GH_TOKEN, GITHUB_PAT_ALL_WRITES, GITHUB_PAT_P24_INFRA | G (#4065) |
mongodb-bms.env.sops | MERGE | mongodb-rs0.env.sops | rs0 connection/app creds | J |
vps-h1.env.sops | AUDIT | archive or rename to real consumer | ”Traefik, SSL, vps-h1 services” — WAHA decommissioned 2026-07-02; likely mostly stale | H |
2.5 Cross-cutting consolidation targets (new service files)
| New file | Consolidates FROM | Purpose | Track |
|---|---|---|---|
mongodb-rs0.env.sops | bms-servers (rs0 app users) + mongodb-bms + n8n-bms4 (Prometheus reader) | one home for the shared rs0 service creds, split by trust tier (admin creds stay in administration; app URIs stay per-app in pinbox24-w3/w4) | J |
n8n.env.sops | rename of n8n-bms4.env.sops after Track J/G removals | the n8n stack’s own runtime env | N |
admin-server-access.env.sops | root-password residue of bms-servers.env.sops | server root SSH access, dev-only | S |
Non-duplication guard. Per-app MongoDB URIs (
V32_NEW_MONGODB_URI,V42_NEW_MONGODB_URI,PINBOX24_MONGODB_URI) stay in their per-app files —mongodb-rs0.env.sopsholds only the shared / infra rs0 creds (admin-tier pointer, Prometheus reader), never a duplicate app URI. This closes the same duplication #3178 flagged forbms-serversmongodb_w3/w4_app_password.
Source-of-truth pin (Track J prerequisite — plan-review note, iteration 1). Because Track J touches keys currently split across four files, the Track J implementation issue must first record a single winning file per shared rs0 credential before any Step-4 removal. Proposed pinning (to be confirmed by secret-manager at implementation): rs0 admin creds → source of truth
administration.env.sops; Prometheus/monitoring reader →mongodb-rs0.env.sops; app creds → the per-apppinbox24-w3/w4.env.sopsURIs.mongodb-bms.env.sopsand thebms-servers.env.sopsrs0 app-password duplicates are consumers to be retired, never a source of truth. No key is removed from any file until its winning file is deployed and verified (§3 Step 4).
3. Migration sequencing — zero decrypt-gap guarantee
The one hard invariant: at no point may a running server or CI job be unable to decrypt a key it needs. Every track obeys the same additive-first ordering, proven by #3178:
For each key K moving from OLD.env.sops → NEW.env.sops:
1. CREATE NEW.env.sops containing K (and add its .sops.yaml recipient rule if narrower than default).
2. WIRE add/confirm the secrets-sync.yml job + target enum + push-path filter for NEW,
and MERGE that workflow change FIRST (job must exist before the SOPS file triggers it).
3. DEPLOY merge NEW.env.sops → secrets-sync runs → verify K is present at every consumer path.
4. REMOVE only now delete K from OLD.env.sops (canary-decrypt OLD before commit).
5. VERIFY re-run the affected sync + smoke test; K served from NEW, absent from OLD.
Key ordering rules carried from #3178:
- Workflow before file. A new
*.env.sopsmatchingsecrets/*.env.sopsauto-triggers on push; its sync job must already exist onmain, or the trigger fires into a no-op and the consumer starves. → thesecrets-sync.ymlPR always merges before the new-SOPS-file PR. - Never remove-before-deploy. Step 4 is gated on Step 3 verification. If verification fails, stop — OLD still has K, so nothing is broken (this is also the in-track rollback, §4).
- Recipient parity. New files inherit the 6-recipient default rule in
.sops.yamlunless they are dev-only (Family A) — those get an exclusion in the push-path filter and an isolated recipient rule (model: theadministration.env.sopsrule and therole-*narrowed rules). sops updatekeysis banned on dotenv files (SOPS 3.9.1 bug #4601). Re-keying usesscripts/sops-set.ps1 -RekeyOnly(Windows dev / secret-manager). Flagged human-action (§7).
Per-track sequence
| Order | Track | PRs (in order) | Decrypt-gap risk |
|---|---|---|---|
| 1 | M (#3178) | already sequenced in that plan (PR-D…PR-G) | none (additive-first) |
| 2 | G (#4065) | already sequenced (Wave A then B) | none |
| 3 | J MongoDB | J1 sync job for mongodb-rs0 → J2 create mongodb-rs0.env.sops (copy rs0 keys) → J3 remove rs0 dupes from bms-servers/mongodb-bms/n8n-bms4 | none if J3 after J2 verified |
| 4 | N n8n rename | N1 add n8n sync job (alongside existing sync-bms-4) → N2 create n8n.env.sops (n8n keys only, post-J/G) → N3 retire n8n-bms4.env.sops | rename = create-new + delete-old, never in-place |
| 5 | S server-access | S1 create admin-server-access.env.sops (dev-only, add exclusion + recipient rule) → S2 verify decrypt → S3 delete bms-servers.env.sops | none — file is dev-only, not server-deployed |
| 6 | H vps-h1 audit | H1 grep-audit every key’s live consumer → H2 archive dead keys per docs/decommissioned-services.md OR rename to real consumer | audit-first; no removal until consumer confirmed |
Renames are always create-new-then-delete-old (never an in-place git mv of the encrypted
blob), because the sync trigger keys on the filename: the new name must have a live sync job before
the old file is deleted.
4. Rollback plan
Rollback is cheap because every track is additive-first — OLD retains the key until NEW is verified.
- In-track (before Step 4 removal): do nothing. OLD still serves K; abandon the NEW file PR
(
git revertthe create commit, or simply don’t merge it). No consumer ever lost access. - After Step 4 removal, if a consumer breaks: revert the removal commit (restores K to OLD),
re-run the affected
secrets-sync.ymltarget, canary-decrypt OLD. Because the value never changed (move, not rotate), the restored key is byte-identical. - Recipient/
.sops.yamlrollback: re-add the removed recipient pubkey(s) to the file’s rule, re-key withscripts/sops-set.ps1 -RekeyOnly, canary, push (the exact rollback #4556 §6.1 documents for therole-*narrowing). - Full-track abort: each track is one-or-two files; reverting its PR chain returns to the prior
layout with no cross-track coupling (tracks share only
secrets-sync.yml, and each edits a distinct job block — see §5 conflict note).
Verification gate for every removal (Step 5): run .github/workflows/credential-smoke-tests.yml
(or the track’s smoke subset) + the relevant secrets-sync.yml -f target=<t> and confirm green
before the track is considered done.
5. Files to update (deliverable #4)
Renaming/splitting SOPS files touches two very different reference classes. Only the first class is rewritten; the second is deliberately left as historical record.
5.1 MUST update (authoritative references — the canonical map)
| File | What changes |
|---|---|
.sops.yaml | add creation rules for new files (mongodb-rs0, n8n, admin-server-access); dev-only files get isolated recipient rules |
.github/workflows/secrets-sync.yml | add/rename sync jobs (sync-n8n, sync-mongodb-rs0), update target enum, update push-path filter (exclusions for dev-only files) |
CLAUDE.md (repo root) | the SOPS file map table + Role Enforcement secret-manager scope rows |
/home/claude-runner/.claude/CLAUDE.md (global) | SOPS file-map reference (monitoring/bms-4/.env lookup lines) |
.claude/agent-prompts/roles/{secret-manager,sys-admin,sys-security,dev-coder}.md | file-name references in each role’s scope |
.claude/role-contexts/p24-infra/secret-manager.md, .claude/role-contexts/pinbox24/sysadmin.md | file-map references |
.claude/task-playbooks/credential-design.md, .claude/task-playbooks/network-ops.md | file-name references |
docs/playbooks/secret-manager.md, docs/playbooks/secret-manager-request.md | SOPS file map |
docs/06-secrets/README.md, docs/07-access-control/ai-sysadmin-capabilities.md, docs/13-external-services/README.md | file-map + ownership tables |
docs/w3-w4-stack-operations.md | if any W3/W4 rs0 reference moves to mongodb-rs0 |
docs/disaster-recovery-runbook.md, docs/disaster-recovery.md | restore-order references to renamed files |
ansible/inventory/group_vars/all.yml, ansible/inventory/host_vars/vps-h1.yml, ansible/playbooks/{bms-4,bms-servers}.yml | any hard-coded SOPS filename |
workflow consumers: .github/workflows/{credential-smoke-tests,deploy-meta-dispatcher,deploy-p24-auth-worker,nightly-queue-cleanup,wasabi-admin-sync,wasabi-sops-update,self-test,apply-supabase-migrations}.yml | filename references |
Enumeration recipe (run per renamed file at implementation time — do not hand-transcribe):
# Authoritative refs to a given file, EXCLUDING historical logs (see 5.2):
grep -rlE '<oldname>\.env\.sops' \
--include='*.md' --include='*.yml' --include='*.yaml' --include='*.sh' --include='*.py' . \
| grep -vE '^\./(docs/20[0-9]{2}-|docs/audits/|CHANGELOG\.md|docs/plans/)'5.2 MUST NOT rewrite (historical record — leave as-was-true-then)
Per the repo memory rule “recalled/dated docs reflect what was true when written,” do not edit:
docs/2026-07/secrets-rotation-log-*.md,docs/audits/2026-07-02-*.md— dated logs.CHANGELOG.mdprior entries (add a NEW[Unreleased]line per track instead).docs/plans/*.md(including #3178, #4065, #4556) — prior plans are frozen; this plan references, never edits them.docs/decommissioned-services.md— append-only (Track H adds vps-h1 entries; never rewrites).
Reference-count reality check:
monitoring.env.sopsappears in 225 files,n8n-bms4.env.sopsin 133,bms-servers.env.sopsin 75 — the large majority are class 5.2 historical logs. The authoritative surface per file (class 5.1) is ~15–25 files. Do not masssedacross all matches; the enumeration recipe’s exclusion filter is mandatory.
6. Regression risks
- Decrypt-gap during a rename — mitigated by additive-first §3 (create+deploy+verify before delete) and the workflow-before-file rule.
- Sync job missing for a new filename — a
secrets/*.env.sopspush auto-triggers; if no job matches, the consumer silently starves. Mitigation: §3 Step 2 merges thesecrets-sync.ymlchange first; §7 flags the verify as human-action. - Recipient loss on dev-only files — if
admin-server-access.env.sopsgets the 6-recipient default instead of the dev-only isolated rule, root passwords become decryptable by every runner. Mitigation: Track S adds the exclusion + isolated rule in the SAME PR that creates the file; security-audit of the final recipient matrix delegated to sys-security (§7). secrets-sync.ymlcross-track conflict — every track edits a distinct job block, but all add to the singletargetenum line. Mitigation (from #3178): batch enum edits, or land tracks strictly sequentially so each rebases cleanly.- Stale
vps-h1.env.sopskeys with a hidden live consumer — Track H is audit-first; no key is removed untilgrep-confirmed dead across the fleet.
7. Human-action / delegation flags (deliverable #5)
Every item below is outside dev-coder scope and must be executed by the named role. This plan does not perform any of them.
| Step | Required role | Why human-action |
|---|---|---|
All sops --encrypt / -RekeyOnly writes, every key move | secret-manager | dev-coder cannot write/decrypt SOPS |
.sops.yaml recipient add/remove + re-key | secret-manager | changes who can decrypt |
Update GH Secrets mirrors for renamed files (sync-gh-secrets) | secret-manager | writes org secrets |
| Vercel env re-sync (et-op, brandpilot, art-agency, et-lager) if a consumed key moves | secret-manager | writes deployment env |
Run secrets-sync.yml -f target=<t> per track + verify consumer paths | sys-admin | server-level deploy/verify |
Create MongoDB mongodb-rs0 service users if Track J introduces new accounts | sys-admin (mongosh on rs0 primary) | server DB op |
| Final recipient-matrix review (dev-only files not runner-decryptable) | sys-security | access-control audit |
| Track H archival decisions (which vps-h1 keys are dead) | sys-admin (fleet grep + confirm) | server-level consumer check |
Delegation path: on approval, this plan spawns per-track implementation issues (one each for
Tracks J, N, S, H) with job_type=secret-manager for the SOPS/recipient work and infra-task
(bms-4) for the deploy/verify work, per docs/playbooks/secret-manager-request.md and
worker-issue.md Step 0b-ROLE. Tracks M and G are not re-issued — they proceed under 4065.
8. Open questions for /review-plan
administration.env.sopsrename? Left as-is (name already means trust tier, not host; 73 refs). Rename toadmin-master.env.sopsonly if the review wants prefix uniformity across Family A. Recommendation: do not rename — low value, high ref-churn.ovh-api.env.sopsis provider-scoped (arguably Family A already). Keep as-is vs. fold under anadmin-provider-*convention. Recommendation: keep — provider-scoping is legible.- MongoDB rs0 admin creds — keep in
administration.env.sops(trust tier) or move tomongodb-rs0.env.sops(service)? Recommendation: keep admin-tier inadministration, put only shared non-admin rs0 creds inmongodb-rs0(blast-radius separation). - **Track ordering vs. 2835 (fresh-key rebuild): if #2835 executes first, every file is re-keyed anyway — should renames piggy-back on that re-key pass? Flag for human sequencing decision.
9. Deliverable checklist (issue #4729)
- (1) Concrete old→new file map by role×project — §2 (tables 2.1–2.5).
- (2) Migration sequencing with zero decrypt-gap — §3 (additive-first + per-track order).
- (3) Rollback plan — §4.
- (4) Explicit list of every doc/workflow file to update (+ what NOT to rewrite) — §5.
- (5) Human-action steps flagged — §7.
This issue is plan-only. Implementation is deferred to per-track secret-manager / infra-task
issues spawned on approval. No SOPS file, .sops.yaml, or secrets-sync.yml is modified by this PR.