p24-infra Docs

per-service-credential-isolation-plan

12 min read

Per-Service Credential Isolation & Rotation Migration Plan

Issue: #2416 Status: PLAN (design artifact — not yet executed) Author: worker agent (bms-4) Created: 2026-07-01 Type: Design / migration plan (execution is follow-up infra-task work — see §8)

This is a planning document, not a runbook. It defines the target state, the migration sequence, and the backlog of follow-up execution issues. No credentials are created, rotated, or moved by merging this document — every mutation step is delegated to an infra-task worker with a root shell on the affected server. Secret values never appear here; key names only.

1. Objective (from #2416)

Move the estate from shared credentials (one secret reused by many consumers) to per-exposure-point credentials (one secret per place it can be exposed), starting with MongoDB rs0 and extending the policy to every credential we can auto-rotate.

Three concrete deliverables the issue asks for:

  1. A mongodb-admin worker role — a dedicated MongoDB login used only by AI workers/sessions to administer the database, distinct from the root admin credential that apps currently share.
  2. Per-service MongoDB logins — each service that reaches rs0 gets its own scoped user (e.g. vercel-et-operational-platform-mongodb-w4, …-test-…), so a leak at one consumer never forces a rotation across all of them.
  3. A migration + rotation plan for every auto-rotatable secret, plus an explicit list of the services where auto-rotation is not yet possible and what is needed to enable it (§7) — the backlog the human can work through.

2. Guiding principle — one secret per exposure point

Definition. An exposure point is any distinct location where a credential value comes to rest or transits: a SOPS file, a GH Secret, a Vercel env var, an n8n credential-vault entry, a Docker env file baked into a container, an app config on a specific host.

Policy. Each exposure point SHOULD hold a credential that is used only there. When a value is shared across N exposure points, a compromise or accidental log/chat/git leak at any one of them forces a rotation that must fan out to all N — increasing blast radius and rotation cost, and making “which copy leaked?” unanswerable.

Where sharing is unavoidable (e.g. a provider issues a single account-level key), the credential stays shared but is documented in the “Key cross-file duplications” table of secret-rotation-access-matrix.md and rotated atomically across all copies. The goal of this plan is to shrink that shared set to only the genuinely-unavoidable cases.

This principle already appears as “preferred” in secrets-design-standard.md; #2416 promotes it to an enforced target state with a migration path.

3. Part A — MongoDB rs0 per-service credential model

3.1 Current state (the anti-pattern)

MongoDB rs0 spans bms-2 (PRIMARY), bms-3 (secondary), bms-4 (arbiter — holds no user data). Today there are only two users (see mongodb-credential-rotation.md):

UserRoleUsed byStored as
adminroot on adminp24-status.py, manual ops, Pinbox24 v42-prod (w4) + v32-prod (w3) app containers on bms-1MONGODB_RS0_ADMIN_USER/PASSWORD in monitoring.env.sops, mirrored in bms-servers.env.sops, baked into Docker env files on bms-1
prometheusclusterMonitor on adminmongodb-exporter-bms2/bms3 on bms-4MONGODB_RS0_PROMETHEUS_PASSWORD in n8n-bms4.env.sops

Problems:

  • The root admin credential is a single secret exposed in ≥5 places (2 SOPS files + 2 Pinbox24 container env files + any worker/ops session that uses it). Rotating it is a high-blast-radius operation that has already caused a production outage (#1716 — Pinbox24 w3/w4 down after a SOPS-only rotation missed the container env files; see step 1e of the rotation playbook).
  • Applications authenticate with root — far more privilege than a per-tenant app needs.
  • There is no dedicated login for AI workers/sessions; a worker doing DB maintenance would reuse the same root admin secret, widening its exposure further.
  • prometheus is already correctly isolated (least-privilege clusterMonitor, its own secret) — the model to replicate for everyone else.

3.2 Target state — one user per consumer, least privilege

MongoDB userRole (least privilege)DB scopeConsumer / exposure pointSOPS key namesSOPS file
adminrootadminBreak-glass only — manual ops / recovery. Removed from all app + worker paths.MONGODB_RS0_ADMIN_USER/PASSWORDadministration.env.sops (moved out of monitoring)
sys-admin-aireadWriteAnyDatabase + dbAdminAnyDatabase (NOT userAdmin/root)clustermongodb-admin worker role — AI workers/sessions administering rs0MONGODB_SYS_ADMIN_AI_USER/PASSWORDbms-servers.env.sops
prometheusclusterMonitoradminmongodb-exporter-bms2/bms3 (unchanged)MONGODB_RS0_PROMETHEUS_PASSWORDn8n-bms4.env.sops
svc-pinbox24-w4readWrite on the w4 app DBw4 DB onlyPinbox24 v42-prod container on bms-1PINBOX24_W4_MONGODB_USER/PASSWORDbms-servers.env.sops
svc-pinbox24-w3readWrite on the w3 app DBw3 DB onlyPinbox24 v32-prod container on bms-1PINBOX24_W3_MONGODB_USER/PASSWORDbms-servers.env.sops
svc-etop-prod-w4read (or readWrite if writes are required) on the w4 DBw4 DB onlyet-operational-platform Vercel (prod) — PINBOX24_MONGODB_URI todayVERCEL_ETOP_MONGODB_W4_USER/PASSWORDet-operational-platform.env.sops
svc-etop-test-w4read on the w4 (or a dedicated test) DBtest scope onlyet-operational-platform Vercel (preview/test)VERCEL_ETOP_TEST_MONGODB_W4_USER/PASSWORDet-operational-platform.env.sops

Confirm during execution, do not assume: the exact app database names on rs0 (w3 DB, w4 DB) and whether et-op needs read vs readWrite must be read off the live cluster and the app config before creating users. This plan intentionally leaves those as the one open discovery item; every other field above is fixed.

Why sys-admin-ai is not root: the worker role must be able to create indexes, run maintenance, and read/write app data, but it should not be able to create or drop other users or grant roles — that stays with the break-glass admin. readWriteAnyDatabase + dbAdminAnyDatabase covers worker maintenance without handing workers the ability to escalate the credential estate.

3.3 Naming convention (canonical)

  • MongoDB username: svc-<consumer>-<scope> for services, sys-admin-ai for the worker role, provider defaults (admin, prometheus) kept for the two existing users.
  • SOPS env key: MONGODB_<CONSUMER>_<SCOPE>_USER / MONGODB_<CONSUMER>_<SCOPE>_PASSWORD (uppercase, underscore-separated — matches existing SOPS dotenv style). The issue’s illustrative vercel-et-operational-platform-mongodb-w4-login maps to VERCEL_ETOP_MONGODB_W4_USER here (shortened et-operational-platform → ETOP to keep env keys readable; the mapping is recorded in this table so it is unambiguous).
  • One consumer → one SOPS file → one exposure point. A user’s secret lives in exactly the SOPS file that deploys to the host that consumer runs on (Pinbox24 app users → bms-servers; Vercel app users → et-operational-platform; exporters → n8n-bms4).

3.4 MongoDB migration steps (delegated to an infra-task worker on the PRIMARY)

Each step is a follow-up execution issue (§8). All mongosh writes run on the current PRIMARY (find it per step 0 of mongodb-credential-rotation.md) and replicate to secondaries automatically. Generate passwords with openssl rand -base64 24 | tr -d '@#/?+' to stay URI-safe.

  1. Create the new users on the PRIMARY with the roles in §3.2 (db.createUser(...) per user). Users replicate to bms-3; the bms-4 arbiter holds none. No consumer is touched yet — additive only.
  2. Store each new secret in its designated SOPS file (§3.2) using the Windows LF-only write pattern (see credential-design / sops-windows-crlf.md), canary-decrypt, commit, PR to main → secrets-sync.yml deploys.
  3. Cut over consumers one at a time, verifying health after each before moving on:
    • Pinbox24 v42-prod / v32-prod on bms-1 → update the Docker env URIs to svc-pinbox24-w4/w3, recreate the container (the mandatory step-1e procedure from the rotation playbook, now done once against the dedicated user instead of the shared root), verify api.w4/w3.pinbox24.com return 200.
    • et-operational-platform Vercel → repoint PINBOX24_MONGODB_URI (prod) and the preview env to the svc-etop-* users; redeploy; verify.
    • p24-status.py and any worker maintenance path → switch to sys-admin-ai.
  4. Verify no consumer still authenticates as admin — grep app configs + check db.adminCommand({getLog:"global"}) on the PRIMARY for auth events tied to the admin user.
  5. Rotate the root admin password once more and move its SOPS key to administration.env.sops, now that no app or worker depends on it. It becomes break-glass only, and its future rotations no longer require the Pinbox24 container recreate (step 1e is retired for app users).
  6. Log every new user + rotation in secrets-rotation-log.md and register the new credentials in dev_r_services (§9).

3.5 Rotation of the new MongoDB users

All rs0 users rotate via db.changeUserPassword(...) over SSH to the PRIMARY — the same mechanism already classified Tier-1 autonomous for MONGODB_RS0_PROMETHEUS_PASSWORD in the access matrix. Once the per-service users exist, extend scripts/rotate-credentials.py with one auto-rotate entry per user (each writes only its own SOPS file + restarts only its own consumer). Because each secret now has exactly one exposure point, these rotations lose their cross-consumer fan-out and their outage risk — the core payoff of the isolation work.

4. Part B — one-secret-per-exposure-point migration for shared secrets

The secret-rotation-access-matrix.md “Key cross-file duplications” table already enumerates the shared secrets. Classify each into split (give each consumer its own credential) vs keep-shared (provider issues only one value) and act:

Shared secretCopies todayTargetRationale
MONGODB_RS0_ADMIN_PASSWORDmonitoring + bms-servers SOPS + 2 Pinbox24 container env filesSPLIT → §3 per-service users; admin demoted to break-glassHighest blast radius; already caused #1716
SUPABASE_SERVICE_ROLE_KEY / SUPABASE_SERVICE_KEYmonitoring, n8n-bms4, vps-h1, brandpilot, art-agency + VercelKEEP-SHARED (one project-level key per Supabase project) but track as an unavoidable duplication; rotate atomically via supabase-service-key-rotation.mdSupabase issues one service key per project; splitting needs separate DB roles (larger initiative)
GH_TOKEN / GitHub PATsmonitoring, n8n-bms4, vps-h1SPLIT — issue a distinct fine-grained PAT (or GitHub App installation token) per host/consumerGitHub supports many PATs; per-host tokens shrink blast radius and are already preferred via the GitHub App
ATRAX_* (client id/secret/user/pass)monitoring, n8n-bms4, vps-h1KEEP-SHARED (vendor issues one OAuth client) — document; rotation is Tier-3 manualNo per-consumer credential available from Atrax
MEZMO_INGESTION_KEY (legacy shared)monitoring, vps-h1SPLIT → per-host ingestion keys per mezmo-key-rotation.md §B-legacyMezmo supports per-host keys
GITHUB_TRIGGER_SECRET, N8N_HU_SP_REPORT_SECRET, BRAND_WEBHOOK_SECRETn8n-bms4 + vps-h1KEEP-SHARED (shared HMAC — both ends must match)A shared secret is inherent to an HMAC handshake between two hosts
WASABI_ADMIN_ACCESS_KEY/SECRETadministration, whatsupSPLIT if whatsup can use a scoped IAM user instead of the root admin key; else keep-shared + documentWasabi IAM supports scoped users
N8N_CLOUD_API_KEY, DISCORD_WEBHOOK_URL, CLAUDE_PROXY_SECRET (vestigial)2 files eachReview individually; CLAUDE_PROXY_SECRET can be removed (claude-proxy retired 2026-06-24)Low priority; one is dead

Rule going forward: the “keep-shared” set is the only place a secret may live in more than one exposure point, and each entry there must state why it cannot be split. Any new secret defaults to one exposure point.

5. Part C — auto-rotation coverage snapshot

Current coverage (full detail in secret-rotation-access-matrix.md):

TierCountMeaningAutomation
✅ Tier 1 — autonomous~35No human, no browserscripts/rotate-credentials.py via credential-rotation.yml (Mon 06:00 UTC)
⚠️ Tier 2 — semi-auto~10Needs an unblocking prerequisite (Playwright session, valid admin key, TOTP secret)Scripted once unblocked
❌ Tier 3 — manual~30No API / UI-only / 2FA / root-of-trustHuman obtains value; Claude distributes

The per-service MongoDB users (§3) each become new Tier 1 entries once created.

6. Part C (cont.) — the human-enablement backlog: services we CANNOT auto-rotate yet

This is the list #2416 explicitly asks for — where auto-rotation is blocked and what would unblock it, so the human can drive each toward automation.

Credential / serviceBlockerWhat would make it auto-rotatableCurrent tier
CF_GLOBAL_API_KEY / CLOUDFLARE_GLOBAL_API_KEYGlobal key has no rotation API (UI “Roll” only)Migrate all consumers off the Global Key onto scoped API tokens (which are rollable), then retire the Global Key❌ Tier 3
SUPABASE_ACCESS_TOKEN (sbp_…)Personal access token, browser-only creationSupabase does not offer PAT-creation API; would need a service-account/OAuth-app token model❌ Tier 3
GitHub PATs (GH_TOKEN, GITHUB_PAT_*)No PAT-creation API; needs browser + 2FAComplete the migration to GitHub App installation tokens (already partially done) → removes the PATs entirely⚠️ Tier 2
DISCORD_BOT_TOKEN + Discord webhooksDeveloper Portal UI-only, 2FANone from Discord; keep manual, minimise consumers❌ Tier 3
ATRAX_*Vendor portal has no credential APIRequest API-based credential management from Atrax vendor❌ Tier 3
GITHUB_APP_PRIVATE_KEY_B64Key generation is github.com/settings/apps UI-onlyNone (root of GitHub-App trust); keep manual❌ Tier 3
OVH_* / SYS_* consumer keysOVH consumer key needs browser OAuthOVH supports long-lived consumer keys once authorized — could script the use, not the initial grant❌ Tier 3
GCP_SA_KEY_JSON, Google OAuth clients, Google Ads tokenGCP Console / approval flowsgcloud service-account key create with owner IAM could automate SA keys; OAuth clients stay manual❌ Tier 3
SMTP_PASSWORD, MAILGUN_API_KEYMailgun dashboard UI-onlyMailgun has a sending-key API — could move MAILGUN_API_KEY to Tier 1; SMTP password stays UI❌ Tier 3
SUPABASE_JWT_SECRET, IAM_ENC_KEY, N8N_ENCRYPTION_KEYRotating invalidates sessions / requires full credential re-encryptionDesign a maintenance-window rotation procedure per key; N8N_ENCRYPTION_KEY needs a full vault migration plan❌ Tier 3
AGE_KEY_GHA (SOPS master)Root of trust for all secretsmaster-keys-bootstrap.md procedure exists but is deliberately manual❌ Tier 3
MONGODB_RS0_ADMIN_PASSWORD (today)Shared root baked into app containers → rotation risks outage§3 of this plan — splitting into per-service users removes the blocker⚠️ → ✅ after §3
ClickUp, LinkedIn, Groq, IONOS, Sentry, TelegramPer-provider UI/OAuth constraints (see matrix)Provider-specific; Telegram is already scriptable if the old token is valid (near Tier 1)❌/⚠️

Actionable enablement items (highest leverage first): finish GitHub-App migration (retires 3 PATs), move Wasabi/whatsup onto scoped IAM users, adopt CF scoped tokens to retire the Global Key, and move MAILGUN_API_KEY to its sending-key API. Each should become its own tracked issue.

7. Phased rollout & sequencing

PhaseScopeBlocking?Follow-up issue type
P1MongoDB per-service users §3 (create → cut over → demote root)No (additive, then per-consumer cutover with health gates)infra-task (SSH to rs0 PRIMARY + bms-1 + Vercel)
P2Extend rotate-credentials.py with the new MongoDB usersNodev-issue (Python edit + PR)
P3Split the “SPLIT” secrets in §4 (GH PATs, Mezmo, Wasabi/whatsup)Nomixed infra-task / dev-issue
P4Human-enablement backlog §6 (one issue per provider)Nohuman-action + design

P1 is the highest-value, highest-risk phase — sequence its consumer cutovers with a health check after each, and keep the old shared admin valid until every consumer is confirmed migrated (rollback = repoint back to admin).

8. Follow-up execution issues to create

This document delivers the plan; execution is separate work. Create:

  1. infra-task — “MongoDB rs0: create per-service users + sys-admin-ai worker role” (§3.4 P1).
  2. dev-issue — “rotate-credentials.py: add per-service MongoDB rotation entries” (§3.5 P2).
  3. dev-issue/infra-task — “Split shared secrets to per-exposure-point” (§4 P3).
  4. human-action — one per row in §6 for providers that need vendor-side or account-level changes.

(This worker is a dev-issue worker and cannot SSH to rs0 — the P1 execution is correctly an infra-task. Producing this plan is the deliverable for #2416.)

9. Compliance

When the new credentials are created (P1), register each in Supabase dev_r_services (element_type='credential', criticality, rotation_freq, auto_rotate=true once §3.5 lands, next_due). The mongodb-admin worker role (sys-admin-ai) is an AI-worker credential — if it is used by an AI system that falls under Annex III, also reflect it in dev_r_ai_systems per the EU AI Act obligations (deadline 2026-08-02).

Graph View

Backlinks