Plan: reorganize SOPS secrets from host-based to role/project-based files

Issue: #4729 Type: Code-change-design (plan only — do NOT implement the migration in this issue) Status: Draft — iteration 1 Last updated: 2026-07-30 Author: Claude Opus 4.8 (dev-coder worker, bms-4 — design only) Related: #3178 (monitoring.env.sops role split — docs/plans/monitoring-env-sops-reorganisation.md; this plan subsumes it as Track M and does not restate its key-level detail), #4556 (role-*.env.sops files + narrowed recipients — the model this plan generalises), #4065 (GitHub PAT / App consolidation — owns the GitHub-token axis; this plan defers all GH_TOKEN/GITHUB_PAT_* moves to it), #2835 (SOPS rebuild with fresh keys — key rotation, orthogonal to this file-organisation work).

Role note (EU AI Act Art. 13 traceability). This document is a design artifact produced by the dev-coder role. dev-coder capabilities.cannot includes writing/rotating/decrypting any secrets/*.env.sops file. Every execution step in this plan (SOPS re-encryption, recipient changes, GH Secrets updates, Vercel env sync, running secrets-sync.yml, MongoDB user creation) is secret-manager and/or sys-admin work and is flagged human-action / delegated — see §7. No secret value appears anywhere in this plan; only key NAMES and file names are referenced.


0. Summary — recommendation

Adopt a single naming rule and migrate the fleet to it in small, additive-first tracks:

The rule: a SOPS file is named for who manages it (trust tier / role) and what consumes it (service / app / role)never for the physical host it happens to land on. A host may source several files; that is the intended outcome, not a regression.

Concretely, every secrets/*.env.sops file falls into exactly one of four families:

FamilyPrefix / conventionManaged byCI-deployed?Examples (target)
A — Admin trust tieradministration.env.sops + admin-*secret-manager (writes), developer (reads)No — excluded from secrets-sync.ymladministration.env.sops, admin-server-access.env.sops, ovh-api.env.sops, gcp.env.sops
B — Worker rolesrole-*.env.sopssecret-managerYes → /opt/p24-infra/{host}/role-{name}.envrole-sys-admin, role-secret-manager, role-dev-coder/reviewer/tester (unchanged — #4556)
C — Services / stacksbare service namesecret-managerYes → the service’s runtime pathn8n.env.sops, mongodb-rs0.env.sops, monitoring.env.sops (shrunk, #3178), traccar.env.sops, whatsup.env.sops
D — Products / appsbare product namesecret-managerYes → Vercel / GitLab / worker bindingsart-agency, brandpilot, et-operational-platform, et-lager, pinbox24-w3/w4/-auth/-backends (unchanged — already the model)

Families B and D already exist and are the model. The work is confined to the genuinely host-named files in family C/An8n-bms4.env.sops, n8n-bms4-gh.env.sops, vps-h1.env.sops, bms-servers.env.sops, mongodb-bms.env.sops — plus the two credential families that host-grouping has scattered (MongoDB rs0, GitHub tokens). monitoring.env.sops is already fully planned in #3178 and is carried here as Track M without re-deriving its key list.

Ship order (each track is independent; M and G are pre-existing):

Track M  monitoring.env.sops role split .............. #3178 (already planned — execute as-is)
Track G  GitHub token consolidation .................. #4065 (already planned — owns GH_TOKEN axis)
Track N  n8n-bms4.env.sops        → n8n.env.sops        (+ hand off Mongo/GH keys to Track J/G)
Track J  MongoDB rs0 consolidation → mongodb-rs0.env.sops (from bms-servers + mongodb-bms + admin)
Track S  bms-servers.env.sops     → admin-server-access.env.sops (root-password residue after Track J)
Track H  vps-h1.env.sops          → audit → archive or rename to real consumer

Rationale: Tracks M and G are already scoped and partly in flight — do not fork them. Tracks N/J/S/H are new and small, each touching one or two files, each shippable behind the additive-first safety rule in §3 with zero decrypt-gap. No track requires an ARCH GATE decision beyond this plan’s approval.


1. Why host-based grouping is the problem (motivating case: bms-3)

The trigger (2026-07-31 bms-3 dispatch-host assessment): bms-3’s root password lives in secrets/bms-servers.env.sops next to bms-1/2/4 root passwords and the MongoDB rs0 replica-set credentials. That single file therefore spans:

  • ≥3 roles — sys-admin (SSH root to each host), secret-manager (rotation), the MongoDB service (rs0 app + admin users);
  • ≥2 projects — server access (an ops concern) and the Pinbox24 MongoDB service (an app concern).

Consequences of coarse, host-keyed grouping:

  1. Blast radius. A breach of one file leaks unrelated trust tiers (root SSH and DB app creds). This is the same “admin keys in monitoring” anti-pattern #3178 fixed for monitoring.env.sops.
  2. Access scoping. .sops.yaml recipients are per-file. Host files force every recipient of the file to be a recipient of all its unrelated secrets. Role/project files let recipient lists be narrowed to exactly the consumers (as #4556 did for role-*).
  3. Rotation coupling. Rotating one host’s root password re-encrypts a file that also carries live DB URIs, widening the change surface and the canary/verify blast radius.
  4. Doc drift. Every CLAUDE.md in the ecosystem already describes ownership as “owned end-to-end by X role” / “credentials via SOPS+age: <file> — a role/project layout, which the host files contradict.

The same scattering afflicts two credential families that no single host owns:

  • MongoDB rs0 creds are split across administration.env.sops (rs0 admin), bms-servers.env.sops (rs0 app users), mongodb-bms.env.sops, and n8n-bms4.env.sops (Prometheus reader user).
  • GitHub tokens are split across monitoring.env.sops, n8n-bms4.env.sops, n8n-bms4-gh.env.sops, and et-operational-platform.env.sops (this axis is **owned by 4065 — see Track G).

2. Current inventory → target family (the concrete file map)

Legend: KEEP = already role/project-based (the model, no change); PLAN = covered by an existing plan; MOVE/SPLIT/RENAME/AUDIT = new work in this plan. The contents column references key NAMES/groups only.

2.1 Family D — products/apps (KEEP — already the model)

FileVerdictNote
art-agency.env.sopsKEEPproject-based ✓
brandpilot.env.sopsKEEPproject-based ✓
et-operational-platform.env.sopsKEEPproject-based ✓ (GitHub token axis handled by #4065)
et-lager.env.sopsKEEPproject-based ✓
pinbox24-w3.env.sops (V32_*)KEEPservice-based ✓ (docs/w3-w4-stack-operations.md)
pinbox24-w4.env.sops (V42_*)KEEPservice-based ✓
pinbox24-w4-auth.env.sopsKEEPcanonical since #3773 Phase 2 ✓
pinbox24-backends.env.sops (mailgun-prod)KEEPservice-based ✓
pinbox24-test-users.env.sopsKEEPtest-fixture scope ✓
whatsup.env.sops (p24-wa-a)KEEPapp-based ✓ (manual sync — #2943)
radekkonarski-brand.env.sops, gmail-tools.env.sopsKEEPlive in their own repos — already project-based, out of scope for this repo’s secrets/

2.2 Family B — worker roles (KEEP — already the model, #4556)

role-sys-admin · role-secret-manager · role-dev-coder · role-dev-reviewer · role-dev-tester — all KEEP. This plan generalises their pattern; it does not change them. rotate-secret.env.sops (rotation tooling) — KEEP (role/tool-scoped).

2.3 Family A — admin trust tier

Current fileVerdictTargetContents (key NAMES only)
administration.env.sopsKEEP nameadministration.env.sopsIONOS API, CF Global Key, WASABI_ADMIN, GITLAB_ADMIN_PAT, SUPABASE_ACCESS_TOKEN, GH_PAT_ADMIN, MongoDB rs0 admin (→ Track J decides: keep admin-tier here vs. move to mongodb-rs0)
ovh-api.env.sopsKEEPovh-api.env.sopsOVH + SoYouStart write-scope API keys — provider-scoped, dev-only (already excluded from sync) ✓
gcp.env.sopsKEEPgcp.env.sopsGCP_SA_KEY_JSON — project/provider-scoped, dev-only ✓
bms-servers.env.sopsSPLITadmin-server-access.env.sops + mongodb-rs0.env.sopsroot part (bms-1/2/3/4 root pw) → admin-server-access; rs0 app-user part → Track J

bms-servers.env.sops root passwords are correctly one role (sys-admin/admin), so they stay together — but under a name that says what they are (server root access), not which hosts. Renaming to admin-server-access.env.sops keeps the dev-only exclusion and makes the trust tier legible. (Track S.)

2.4 Family C — services/stacks (the host-named residue)

Current fileVerdictTargetContents (key NAMES only)Track
monitoring.env.sopsPLANshrink per #3178 (→ worker-queue, cloudflare, traccar, role-*, administration)observability connectors onlyM (#3178)
n8n-bms4.env.sopsRENAME + SPLITn8n.env.sopsn8n stack proper: N8N_*, Redis. Hand off: MongoDB Prometheus reader → Track J; GH_TOKEN → Track GN
n8n-bms4-gh.env.sops**DEFER to 4065GitHub-token file per #4065GH_TOKEN, GITHUB_PAT_ALL_WRITES, GITHUB_PAT_P24_INFRAG (#4065)
mongodb-bms.env.sopsMERGEmongodb-rs0.env.sopsrs0 connection/app credsJ
vps-h1.env.sopsAUDITarchive or rename to real consumer”Traefik, SSL, vps-h1 services” — WAHA decommissioned 2026-07-02; likely mostly staleH

2.5 Cross-cutting consolidation targets (new service files)

New fileConsolidates FROMPurposeTrack
mongodb-rs0.env.sopsbms-servers (rs0 app users) + mongodb-bms + n8n-bms4 (Prometheus reader)one home for the shared rs0 service creds, split by trust tier (admin creds stay in administration; app URIs stay per-app in pinbox24-w3/w4)J
n8n.env.sopsrename of n8n-bms4.env.sops after Track J/G removalsthe n8n stack’s own runtime envN
admin-server-access.env.sopsroot-password residue of bms-servers.env.sopsserver root SSH access, dev-onlyS

Non-duplication guard. Per-app MongoDB URIs (V32_NEW_MONGODB_URI, V42_NEW_MONGODB_URI, PINBOX24_MONGODB_URI) stay in their per-app filesmongodb-rs0.env.sops holds only the shared / infra rs0 creds (admin-tier pointer, Prometheus reader), never a duplicate app URI. This closes the same duplication #3178 flagged for bms-servers mongodb_w3/w4_app_password.

Source-of-truth pin (Track J prerequisite — plan-review note, iteration 1). Because Track J touches keys currently split across four files, the Track J implementation issue must first record a single winning file per shared rs0 credential before any Step-4 removal. Proposed pinning (to be confirmed by secret-manager at implementation): rs0 admin creds → source of truth administration.env.sops; Prometheus/monitoring readermongodb-rs0.env.sops; app creds → the per-app pinbox24-w3/w4.env.sops URIs. mongodb-bms.env.sops and the bms-servers.env.sops rs0 app-password duplicates are consumers to be retired, never a source of truth. No key is removed from any file until its winning file is deployed and verified (§3 Step 4).


3. Migration sequencing — zero decrypt-gap guarantee

The one hard invariant: at no point may a running server or CI job be unable to decrypt a key it needs. Every track obeys the same additive-first ordering, proven by #3178:

For each key K moving from OLD.env.sops → NEW.env.sops:
  1. CREATE   NEW.env.sops containing K (and add its .sops.yaml recipient rule if narrower than default).
  2. WIRE     add/confirm the secrets-sync.yml job + target enum + push-path filter for NEW,
              and MERGE that workflow change FIRST (job must exist before the SOPS file triggers it).
  3. DEPLOY   merge NEW.env.sops → secrets-sync runs → verify K is present at every consumer path.
  4. REMOVE   only now delete K from OLD.env.sops (canary-decrypt OLD before commit).
  5. VERIFY   re-run the affected sync + smoke test; K served from NEW, absent from OLD.

Key ordering rules carried from #3178:

  • Workflow before file. A new *.env.sops matching secrets/*.env.sops auto-triggers on push; its sync job must already exist on main, or the trigger fires into a no-op and the consumer starves. → the secrets-sync.yml PR always merges before the new-SOPS-file PR.
  • Never remove-before-deploy. Step 4 is gated on Step 3 verification. If verification fails, stop — OLD still has K, so nothing is broken (this is also the in-track rollback, §4).
  • Recipient parity. New files inherit the 6-recipient default rule in .sops.yaml unless they are dev-only (Family A) — those get an exclusion in the push-path filter and an isolated recipient rule (model: the administration.env.sops rule and the role-* narrowed rules).
  • sops updatekeys is banned on dotenv files (SOPS 3.9.1 bug #4601). Re-keying uses scripts/sops-set.ps1 -RekeyOnly (Windows dev / secret-manager). Flagged human-action (§7).

Per-track sequence

OrderTrackPRs (in order)Decrypt-gap risk
1M (#3178)already sequenced in that plan (PR-D…PR-G)none (additive-first)
2G (#4065)already sequenced (Wave A then B)none
3J MongoDBJ1 sync job for mongodb-rs0 → J2 create mongodb-rs0.env.sops (copy rs0 keys) → J3 remove rs0 dupes from bms-servers/mongodb-bms/n8n-bms4none if J3 after J2 verified
4N n8n renameN1 add n8n sync job (alongside existing sync-bms-4) → N2 create n8n.env.sops (n8n keys only, post-J/G) → N3 retire n8n-bms4.env.sopsrename = create-new + delete-old, never in-place
5S server-accessS1 create admin-server-access.env.sops (dev-only, add exclusion + recipient rule) → S2 verify decrypt → S3 delete bms-servers.env.sopsnone — file is dev-only, not server-deployed
6H vps-h1 auditH1 grep-audit every key’s live consumer → H2 archive dead keys per docs/decommissioned-services.md OR rename to real consumeraudit-first; no removal until consumer confirmed

Renames are always create-new-then-delete-old (never an in-place git mv of the encrypted blob), because the sync trigger keys on the filename: the new name must have a live sync job before the old file is deleted.


4. Rollback plan

Rollback is cheap because every track is additive-first — OLD retains the key until NEW is verified.

  • In-track (before Step 4 removal): do nothing. OLD still serves K; abandon the NEW file PR (git revert the create commit, or simply don’t merge it). No consumer ever lost access.
  • After Step 4 removal, if a consumer breaks: revert the removal commit (restores K to OLD), re-run the affected secrets-sync.yml target, canary-decrypt OLD. Because the value never changed (move, not rotate), the restored key is byte-identical.
  • Recipient/.sops.yaml rollback: re-add the removed recipient pubkey(s) to the file’s rule, re-key with scripts/sops-set.ps1 -RekeyOnly, canary, push (the exact rollback #4556 §6.1 documents for the role-* narrowing).
  • Full-track abort: each track is one-or-two files; reverting its PR chain returns to the prior layout with no cross-track coupling (tracks share only secrets-sync.yml, and each edits a distinct job block — see §5 conflict note).

Verification gate for every removal (Step 5): run .github/workflows/credential-smoke-tests.yml (or the track’s smoke subset) + the relevant secrets-sync.yml -f target=<t> and confirm green before the track is considered done.


5. Files to update (deliverable #4)

Renaming/splitting SOPS files touches two very different reference classes. Only the first class is rewritten; the second is deliberately left as historical record.

5.1 MUST update (authoritative references — the canonical map)

FileWhat changes
.sops.yamladd creation rules for new files (mongodb-rs0, n8n, admin-server-access); dev-only files get isolated recipient rules
.github/workflows/secrets-sync.ymladd/rename sync jobs (sync-n8n, sync-mongodb-rs0), update target enum, update push-path filter (exclusions for dev-only files)
CLAUDE.md (repo root)the SOPS file map table + Role Enforcement secret-manager scope rows
/home/claude-runner/.claude/CLAUDE.md (global)SOPS file-map reference (monitoring/bms-4/.env lookup lines)
.claude/agent-prompts/roles/{secret-manager,sys-admin,sys-security,dev-coder}.mdfile-name references in each role’s scope
.claude/role-contexts/p24-infra/secret-manager.md, .claude/role-contexts/pinbox24/sysadmin.mdfile-map references
.claude/task-playbooks/credential-design.md, .claude/task-playbooks/network-ops.mdfile-name references
docs/playbooks/secret-manager.md, docs/playbooks/secret-manager-request.mdSOPS file map
docs/06-secrets/README.md, docs/07-access-control/ai-sysadmin-capabilities.md, docs/13-external-services/README.mdfile-map + ownership tables
docs/w3-w4-stack-operations.mdif any W3/W4 rs0 reference moves to mongodb-rs0
docs/disaster-recovery-runbook.md, docs/disaster-recovery.mdrestore-order references to renamed files
ansible/inventory/group_vars/all.yml, ansible/inventory/host_vars/vps-h1.yml, ansible/playbooks/{bms-4,bms-servers}.ymlany hard-coded SOPS filename
workflow consumers: .github/workflows/{credential-smoke-tests,deploy-meta-dispatcher,deploy-p24-auth-worker,nightly-queue-cleanup,wasabi-admin-sync,wasabi-sops-update,self-test,apply-supabase-migrations}.ymlfilename references

Enumeration recipe (run per renamed file at implementation time — do not hand-transcribe):

# Authoritative refs to a given file, EXCLUDING historical logs (see 5.2):
grep -rlE '<oldname>\.env\.sops' \
  --include='*.md' --include='*.yml' --include='*.yaml' --include='*.sh' --include='*.py' . \
  | grep -vE '^\./(docs/20[0-9]{2}-|docs/audits/|CHANGELOG\.md|docs/plans/)'

5.2 MUST NOT rewrite (historical record — leave as-was-true-then)

Per the repo memory rule “recalled/dated docs reflect what was true when written,” do not edit:

  • docs/2026-07/secrets-rotation-log-*.md, docs/audits/2026-07-02-*.md — dated logs.
  • CHANGELOG.md prior entries (add a NEW [Unreleased] line per track instead).
  • docs/plans/*.md (including #3178, #4065, #4556) — prior plans are frozen; this plan references, never edits them.
  • docs/decommissioned-services.md — append-only (Track H adds vps-h1 entries; never rewrites).

Reference-count reality check: monitoring.env.sops appears in 225 files, n8n-bms4.env.sops in 133, bms-servers.env.sops in 75 — the large majority are class 5.2 historical logs. The authoritative surface per file (class 5.1) is ~15–25 files. Do not mass sed across all matches; the enumeration recipe’s exclusion filter is mandatory.


6. Regression risks

  • Decrypt-gap during a rename — mitigated by additive-first §3 (create+deploy+verify before delete) and the workflow-before-file rule.
  • Sync job missing for a new filename — a secrets/*.env.sops push auto-triggers; if no job matches, the consumer silently starves. Mitigation: §3 Step 2 merges the secrets-sync.yml change first; §7 flags the verify as human-action.
  • Recipient loss on dev-only files — if admin-server-access.env.sops gets the 6-recipient default instead of the dev-only isolated rule, root passwords become decryptable by every runner. Mitigation: Track S adds the exclusion + isolated rule in the SAME PR that creates the file; security-audit of the final recipient matrix delegated to sys-security (§7).
  • secrets-sync.yml cross-track conflict — every track edits a distinct job block, but all add to the single target enum line. Mitigation (from #3178): batch enum edits, or land tracks strictly sequentially so each rebases cleanly.
  • Stale vps-h1.env.sops keys with a hidden live consumer — Track H is audit-first; no key is removed until grep-confirmed dead across the fleet.

7. Human-action / delegation flags (deliverable #5)

Every item below is outside dev-coder scope and must be executed by the named role. This plan does not perform any of them.

StepRequired roleWhy human-action
All sops --encrypt / -RekeyOnly writes, every key movesecret-managerdev-coder cannot write/decrypt SOPS
.sops.yaml recipient add/remove + re-keysecret-managerchanges who can decrypt
Update GH Secrets mirrors for renamed files (sync-gh-secrets)secret-managerwrites org secrets
Vercel env re-sync (et-op, brandpilot, art-agency, et-lager) if a consumed key movessecret-managerwrites deployment env
Run secrets-sync.yml -f target=<t> per track + verify consumer pathssys-adminserver-level deploy/verify
Create MongoDB mongodb-rs0 service users if Track J introduces new accountssys-admin (mongosh on rs0 primary)server DB op
Final recipient-matrix review (dev-only files not runner-decryptable)sys-securityaccess-control audit
Track H archival decisions (which vps-h1 keys are dead)sys-admin (fleet grep + confirm)server-level consumer check

Delegation path: on approval, this plan spawns per-track implementation issues (one each for Tracks J, N, S, H) with job_type=secret-manager for the SOPS/recipient work and infra-task (bms-4) for the deploy/verify work, per docs/playbooks/secret-manager-request.md and worker-issue.md Step 0b-ROLE. Tracks M and G are not re-issued — they proceed under 4065.


8. Open questions for /review-plan

  1. administration.env.sops rename? Left as-is (name already means trust tier, not host; 73 refs). Rename to admin-master.env.sops only if the review wants prefix uniformity across Family A. Recommendation: do not rename — low value, high ref-churn.
  2. ovh-api.env.sops is provider-scoped (arguably Family A already). Keep as-is vs. fold under an admin-provider-* convention. Recommendation: keep — provider-scoping is legible.
  3. MongoDB rs0 admin creds — keep in administration.env.sops (trust tier) or move to mongodb-rs0.env.sops (service)? Recommendation: keep admin-tier in administration, put only shared non-admin rs0 creds in mongodb-rs0 (blast-radius separation).
  4. **Track ordering vs. 2835 (fresh-key rebuild): if #2835 executes first, every file is re-keyed anyway — should renames piggy-back on that re-key pass? Flag for human sequencing decision.

9. Deliverable checklist (issue #4729)

  • (1) Concrete old→new file map by role×project — §2 (tables 2.1–2.5).
  • (2) Migration sequencing with zero decrypt-gap — §3 (additive-first + per-track order).
  • (3) Rollback plan — §4.
  • (4) Explicit list of every doc/workflow file to update (+ what NOT to rewrite) — §5.
  • (5) Human-action steps flagged — §7.

This issue is plan-only. Implementation is deferred to per-track secret-manager / infra-task issues spawned on approval. No SOPS file, .sops.yaml, or secrets-sync.yml is modified by this PR.