Secret Rotation Access Matrix
Last updated: 2026-08-04
Owner: p24-infra
Review trigger: any new credential added, any new provider API discovered, or quarterly (with credential-rotation-policy.md)
Purpose
This document answers one question for every secret in the ecosystem: can Claude rotate it autonomously, or does a human need to act?
Use this matrix when:
- Deciding whether to create a
human-actionGH issue or run the auto-rotator - Onboarding a new secret (check which tier applies)
- Auditing rotation coverage gaps
For execution details, see the per-key playbooks linked in each row.
Worker dispatch — autonomous rotation via secret-manager job_type (#2605)
Tier 1 and Tier 2 keys can be dispatched to the worker queue as job_type=secret-manager.
A bms-4 Claude worker picks up the task and runs infra/agent-prompts/worker-secret-manager.md.
# Dispatch from any Claude session (p24-infra preferred):
curl -sf -X POST "$QUEUE_API_URL/queue-issue" \
-H "Authorization: Bearer $QUEUE_API_KEY" \
-H "Content-Type: application/json" \
-d "{\"issue_number\": ISSUE_NUM, \"repo\": \"radieu/p24-infra\", \"job_type\": \"secret-manager\"}"- The dispatcher routes
secret-manager→ infra role → bms-4 (same asinfra-task). - The worker verifies SOPS age key, reads the issue, checks the tier, and rotates autonomously.
- Tier 3 keys are refused — the worker comments on the issue and sets label
human-action. - Never create a
human-actionissue for a Tier 1 or Tier 2 key — dispatch it instead.
Master-key deliverability re-check (#4400 — 2026-07-30)
Question: now that secret-manager holds MASTER/ADMIN management keys for Supabase, Wasabi and
Mailgun, which of the ~13 rotation issues previously parked as human-action are actually
autonomously deliverable? Verification (owner findings on #4400, 2026-07-20) confirmed all three
master keys authenticate against their provider management API (200). This section records the
durable classification and one worker-reachability correction found while producing it.
Master keys — authentication vs. bms-4 worker reachability
A 200 from the provider proves the key is valid where it was probed. It does not prove the
standard bms-4 secret-manager worker (age key ~/.age/p24-infra-keys.txt) can read it: that worker
can decrypt monitoring, brandpilot, whatsup but not administration or role-secret-manager.
| Provider | Master key (NAME) | Mgmt API (verified) | File the bms-4 worker can decrypt | Worker-reachable? |
|---|---|---|---|---|
| Mailgun | MAILGUN_ADMIN_API_KEY | GET/POST/DELETE /v1/keys → 200 | monitoring.env.sops | ✅ yes |
| Supabase | SUPABASE_MANAGEMENT_TOKEN (≡ ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN) | GET api.supabase.com/v1/projects → 200 | brandpilot.env.sops | ✅ yes |
| Wasabi | WASABI_ADMIN_ACCESS_KEY / WASABI_ADMIN_SECRET_KEY | iam:ListUsers / iam:ListAccessKeys → 200 | secrets/wasabi-admin-worker.env.sops (narrow 2-recipient file, developer + bms-4 worker key — added 5601) | ✅ yes (as of #5555) |
Correction to the 2026-07-20 RUN-NOW recommendation (superseded — see below): the Wasabi family was not run-now deliverable by a standard bms-4 secret-manager worker as of this writing — the only Wasabi admin key it could decrypt was the stale
whatsupcopy. The valid key lived inrole-secret-manager/administration, which this worker’s age key was not a recipient of.Resolved (#5555/#5601): rather than widening
administration.env.sopsitself, a narrowsecrets/wasabi-admin-worker.env.sopsfile was created (mirrors thepinbox24-gitlab.env.sopspattern exactly — 2 recipients only, developer + bms-4 worker key) holding a copy of the currentWASABI_ADMIN_*pair. The bms-4 worker can now decrypt it directly. Scope caveat still applies: the probes above werelist-*/GETonly — a200there does not provecreate/rotatescope. Confirm mint/policy-write scope verify-first at operation time. Root-key rotation is still Wasabi-console-only (no self-rotation API for the root identity) regardless of file reachability — this only unblocks day-to-day IAM read/policy operations with the current key. WheneverWASABI_ADMIN_*is rotated (console step, human-required), bothadministration.env.sopsandwasabi-admin-worker.env.sopsmust be updated atomically — seedocs/policies/credential-isolation-policy.md.
Two triage premises falsified
#2804— Mailgun “no rotation API” → superseded.MAILGUN_ADMIN_API_KEYnow has a working/v1/keysmanagement API (create/rotate/delete → 200, US + EU). Mint-automatable. Caveat: confirm the replacement key carries admin scope before retiring the old one. See the Tier-3 Mailgun rows below — they are annotated to point here.#3366— WASABI_ADMIN “invalid everywhere, regenerate at root console” → false. A valid admin key exists (authenticated live, 16 users); only thewhatsup+ GH-Secret copies are stale. Remedy is a SOPS-sync of the working key, not a human root-console regeneration — but see the reachability caveat above (the worker cannot read the valid source).
Classification — NO-HUMAN-DELIVERABLE vs STILL-HUMAN
mint = master key alone completes it · mint+deploy = master key mints, downstream code/deploy still
required (not fully autonomous) · verdict is per capability, independent of the worker-reachability
gate noted above.
| Issue | Family | Verdict | Automation | Notes |
|---|---|---|---|---|
| #2827 (CLOSED) | Supabase | NO-HUMAN | mint (copy) | ADD SUPABASE_ACCESS_TOKEN — already closed |
| #3088 | Supabase | NO-HUMAN | mint+deploy | Mgmt API mints project key → set CF Worker secret (wrangler) |
| #2828 | Supabase | NO-HUMAN | mint+deploy | Rotate service-role key → SOPS + Vercel; needs-clarification = confirm project ref |
| #4110 | Supabase | NO-HUMAN | mint+deploy | Reset Postgres pw → SOPS + PG consumers + add ::add-mask:: to CI (dev-coder, not human) |
| #2889 | Supabase (umbrella) | PARTIAL | mixed | Supabase members deliverable; OPENAI_API_KEY / GITHUB_TOKEN PAT = STILL-HUMAN → decompose |
| #2824 | et-op dump (umbrella) | PARTIAL | mixed | Supabase/Wasabi/Mailgun members deliverable; OVH/OpenAI/GitHub-PAT = STILL-HUMAN → decompose |
| #3366 | Wasabi | NO-HUMAN* | mint (sync) | *Blocked on worker-reachability of the valid admin key (see correction above) |
| #3350 | Wasabi | NO-HUMAN* | mint+deploy | *Same reachability block; then IAM create+delete → SOPS(monitoring) + GH Secret + non-prod restart |
| #2673 | Wasabi (umbrella) | PARTIAL | mixed | Wasabi core deliverable; CF/OVH/Discord/GITHUB_APP carve-outs STILL-HUMAN → decompose |
| #2714 | Wasabi (already routed) | NO-HUMAN | mint+deploy | Reads W4 keys on bms-4 → et-op SOPS/Vercel; pre-delivery-approved, no master key needed |
| #2804 | Mailgun | NO-HUMAN | mint | /v1/keys supersedes dashboard-only premise; confirm admin scope of new key first |
| #4333 | Mailgun (W3) | SPLIT | mint+deploy / human | Mailgun part mint-able but needs GitLab de-hardcode (**blocked 3775) + W3 prod redeploy → Friday; V32_LOGDNA_KEY (Mezmo) = STILL-HUMAN |
| #3587 | Mailgun (W3) | NO-HUMAN | mint+deploy | Mint-able; fully deliverable only after GitLab de-hardcode (**blocked 3775) + W3 prod redeploy → Friday |
RUN-NOW vs FRIDAY-EVENING-WINDOW split
Rule: anything touching W3/W4 prod containers → Friday window; pure provider + SOPS + non-prod restart → run-now eligible and master key worker-reachable.
- ✅ RUN-NOW (worker can read the master key):
#3088,#2828,#4110,#2804. - ⚠️ BLOCKED on Wasabi-admin worker-reachability (fix #3366 sync first):
#3366,#3350. - 🗓️ FRIDAY window + BLOCKED on
#3775(GitLab de-hardcode):#4333,#3587. - 🧩 DECOMPOSE (umbrellas — deliverable members can proceed once split):
#2889,#2824,#2673. - ↪️ Already re-routed:
#2714(infra-task,pre-delivery-approved). - 🔴 STILL-HUMAN (no mgmt-API path):
V32_LOGDNA_KEY(Mezmo console),#3775GITLAB_ADMIN_PAT(GitLab console), and the OpenAI / GitHub-PAT / OVH / CF / Discord members of the umbrellas.
Each RUN-NOW item is a separate secret-manager rotation job, one at a time, verify-first per the standing W3/W4 rotation model. This issue (#4400) performed classification only — no rotation.
Tiers
| Tier | Icon | Meaning |
|---|---|---|
| Fully autonomous | ✅ | Claude can rotate without any human interaction. No browser, no 2FA, no console login. |
| Semi-autonomous | ⚠️ | Rotation is possible without a human, but requires one of: Playwright MCP browser session, TOTP secret present in SOPS, or the admin/root secret for that provider to be valid. The rotation itself is scripted once unblocked. |
| Manual only | ❌ | Requires a human to log into a UI, complete 2FA, or take an action that has no API equivalent. Claude can distribute the new value after the human obtains it, but cannot generate it alone. |
Distribution chain (after any rotation)
Every new credential value must propagate through all applicable destinations before the old value is revoked:
secrets/<file>.env.sops— update + canary decrypt + commit + PR tomain- GH Secret (
gh secret set KEY_NAME --repo radieu/p24-infra) — if used by CI/CD secrets-sync.ymlauto-deployment (fires on merge tomain) → vps-i1/opt/p24-infra/monitoring/.envand bms-4/opt/p24-infra/bms-4/.env- Vercel env vars — if used by et-operational-platform, brandpilot, or any Vercel project
- n8n credential vault — if an n8n workflow reads this credential directly (not via env)
- Container recreate (not
docker restart) on affected server(s) — see the warning below - Revoke old value at the provider
Step 6 must RECREATE the container, never
docker restart.docker restartreuses the environment baked into a container at creation time and does not re-read the on-disk env file — so a restarted container keeps the OLD credential even aftersecrets-sync.ymlupdated the file. Onlydocker-compose up -d --force-recreate(after the old container is stopped/removed) picks up the new value. This matters most for the Pinbox24 bms-1 MongoDB consumers: rotatingw4_apprequires recreatingv42-prod,mailgun-v42-prod, ands3-v2-v42-prod; rotatingw3_apprequires recreatingv32-prod. Skipping the recreate ofmailgun-v42-prodcaused the 2026-07-10 silent mailgun outage — mailgun-mongodb-stale-credential-hang.md. Verify the recreate took with the sha256 env-hash comparison in that playbook (never prints the secret).
Matrix
✅ Tier 1 — Fully Autonomous (no human required)
| Secret | SOPS file(s) | Provider / system | How Claude rotates | Playbook |
|---|---|---|---|---|
GRAFANA_ADMIN_PASSWORD | monitoring | Grafana (self-hosted) | Random string → bcrypt hash → Caddyfile + grafana cli admin reset-admin-password → container restart. Mandatory manual step: grafana cli reset must run after every rotation; env-var update alone does not update the SQLite DB. | credential-rotation-180d.md §GRAFANA_ADMIN_PASSWORD |
SUPABASE_GRAFANA_PASSWORD | monitoring | Supabase (Postgres role) | ALTER ROLE grafana_readonly PASSWORD '<new>' via Supabase Management API or direct psql → Grafana datasource restart | credential-rotation-180d.md §SUPABASE_GRAFANA_PASSWORD |
CLOUDFLARE_TOKEN_ZINTEGROWANA | monitoring | Cloudflare API | PUT /client/v4/user/tokens/{id}/value with CF_GLOBAL_API_KEY (User:API Tokens:Edit scope) → GH Secret update | cloudflare-credential-rotation.md §2 |
CF_EDIT_ALL_ZONES_API_TOKEN | monitoring | Cloudflare API | Same as above — rolled via CF_GLOBAL_API_KEY | cloudflare-credential-rotation.md §3 |
CF_API_TOKEN | monitoring | Cloudflare API | POST /client/v4/user/tokens with CLOUDFLARE_GLOBAL_API_KEY + CLOUDFLARE_EMAIL (X-Auth-Key/X-Auth-Email). Lists existing tokens to preserve policies; falls back to Workers Scripts Write + Zone DNS Write. Deletes old token after GH Secret + SOPS updated. Prerequisite: CLOUDFLARE_GLOBAL_API_KEY and CLOUDFLARE_EMAIL must be valid in env (injected from GH Secrets by credential-rotation.yml). | cloudflare-credential-rotation.md §4 |
EMAIL_SENDER_API_KEY | monitoring | Cloudflare Workers | CF Worker secret rotation via Wrangler API or wrangler secret put in CI | credential-rotation-180d.md §EMAIL_SENDER_API_KEY |
BMS4_N8N_API_KEY | n8n-bms4 | n8n REST API | POST /api/v1/credentials on bms-4 n8n → delete old key → GH Secret update | n8n/n8n-bms4-api-key-rotation.md, credential-rotation-180d.md §8 |
GitLab runner auth tokens on bms-1 (glrt-…, not SOPS-tracked — live only in /etc/gitlab-runner/config.toml) | — (uses administration.env.sops → GITLAB_ADMIN_PAT, api scope) | GitLab REST API | Corrected 2026-08-01 (#4829) — was wrongly filed human-action on a stale “UI-only” premise. POST /api/v4/user/runners creates + returns the token, DELETE /api/v4/runners/:id revokes; multi-project runners need extra attach/detach calls. Register the new token on bms-1 via gitlab-runner register, verify via gitlab-runner verify (never list — it prints tokens), clean config.toml, then delete the old runner objects. | gitlab-runner-bms1-reregister.md |
N8N_BMS4_API_KEY | monitoring | n8n REST API | Same as BMS4_N8N_API_KEY (alias in monitoring SOPS) | n8n/n8n-bms4-api-key-rotation.md |
P24_INFRA_WASABI_ACCESS_KEY + P24_INFRA_WASABI_SECRET_KEY | monitoring | Wasabi IAM API | boto3 iam.wasabisys.com: create_access_key (p24-infra user) → update SOPS → deploy → delete_access_key (old). Requires WASABI_ADMIN_ACCESS_KEY to be valid. | wasabi-key-rotation.md, credential-rotation-180d.md §10 |
WASABI_ACCESS_KEY + WASABI_SECRET_KEY | monitoring | Wasabi IAM API | Alias keys — updated in same SOPS edit as P24_INFRA_WASABI_* | wasabi-key-rotation.md |
wasabi_art_agency_access_key + wasabi_art_agency_secret_key | art-agency | Wasabi IAM API | boto3 create_access_key (art-agency IAM user) → SOPS update | wasabi-key-rotation.md |
MYSQL_PASSWORD | monitoring | MySQL (Traccar DB) | ALTER USER 'traccar'@'%' IDENTIFIED BY '<new>' via SSH → container restart | credential-rotation-180d.md §MYSQL_PASSWORD |
TRACCAR_PASSWORD | monitoring | Traccar REST API | POST /api/session (needs TRACCAR_ADMIN_EMAIL) → PUT /api/users/{id} → SOPS update. Unblocked 2026-08-02 — both former blockers cleared in 9f3676f (#5099/#5111): TRACCAR_ADMIN_EMAIL is now in SOPS, and MYSQL_PASSWORD is split out so this key no longer doubles as the MySQL traccar user password via backup-common.sh. First autonomous rotation ran under #3352. Still run the Step 0 pre-flight before any rotation, and re-run Probe 2 afterwards — it is the only check that catches a dual-consumer regression. | traccar-admin-key-rotation.md §Rotation — TRACCAR_PASSWORD |
TRACCAR_FORWARD_TOKEN | monitoring | CF Worker + Traccar | CF Worker traccar-gw secret update + Traccar env update | traccar-admin-key-rotation.md |
TRACCAR_GW_ADMIN_KEY | monitoring | CF Worker | CF Worker traccar-gw ADMIN_KEY secret update | traccar-admin-key-rotation.md |
REDIS_PASSWORD | n8n-bms4 | Redis (self-hosted bms-4) | Random hex → bms-4 .env + SOPS update → docker compose restart redis + n8n services (BullMQ reconnects) | credential-rotation-180d.md §REDIS_PASSWORD |
PDF_SERVICE_API_KEY | monitoring | Internal (pdf-service) | Random hex → vps-i1 .env + SOPS → pdf-service recreate | credential-rotation-180d.md §PDF_SERVICE_API_KEY |
GF_RENDERING_RENDERER_TOKEN | monitoring | Internal (Grafana renderer) | Random hex → vps-i1 .env + SOPS → grafana + grafana-image-renderer recreate | credential-rotation-180d.md §GF_RENDERING_RENDERER_TOKEN |
GITHUB_TRIGGER_SECRET | n8n-bms4, vps-h1 | Internal (webhook HMAC) | Random hex → bms-4 + vps-h1 .env + both SOPS files + n8n services restart. Note: GH Secrets cannot store keys starting with GITHUB_ | credential-rotation-180d.md §GITHUB_TRIGGER_SECRET |
CRON_SECRET | n8n-bms4 | Vercel (et-op-platform) | Random hex → Vercel env vars for et-operational-platform + SOPS | credential-rotation-180d.md §CRON_SECRET |
CLAUDE_SESSION_MANAGER_KEY | n8n-bms4 | Internal (session-manager) | Random hex → bms-4 .env + SOPS → claude-session-manager restart | credential-rotation-180d.md §CLAUDE_SESSION_MANAGER_KEY |
N8N_HU_SP_REPORT_SECRET | n8n-bms4, vps-h1 | Internal (n8n webhook) | Random hex → bms-4 + vps-h1 .env + both SOPS files + n8n restart | credential-rotation-180d.md §N8N_HU_SP_REPORT_SECRET |
N8N_DB_PASSWORD | n8n-bms4 | PostgreSQL (bms-4) | ALTER USER n8n PASSWORD '<new>' via SSH → bms-4 .env + SOPS → n8n restart | n8n/n8n-db-password-rotation.md |
RADEK_BRAND_RESEND_API_KEY | monitoring | Resend API | DELETE /api-keys/{id} + POST /api-keys via Resend REST API | resend-api-key-rotation.md |
RESEND_API_KEY | brandpilot | Resend API | Same as above (separate key, same account) | resend-api-key-rotation.md |
MEZMO_INGESTION_KEY_* | monitoring, vps-h1, bms-servers | Mezmo API | POST /v1/config/keys?type=ingestion → rename key → update SOPS → restart logdna-agent on affected host | mezmo-key-rotation.md |
VPS_SSH_PRIVATE_KEY (claude-admin) | GH Secret only | Internal (SSH ed25519) | ssh-keygen locally → append new pub to authorized_keys on vps-i1 + vps-h1 → GH Secret update → remove old key | credential-rotation-180d.md §11, ssh-key-rotation.md |
BRAND_WEBHOOK_SECRET | n8n-bms4, vps-h1 | Internal (HMAC) | Random hex → both SOPS files + env files on both VPSes | Internal, see SOPS edit pattern |
N8N_GPS_SYNC_SECRET | n8n-bms4 | Internal (n8n validator) | Random hex → n8n credential vault (via n8n REST API) + Vercel env + SOPS. Note: N8N_GPS_SYNC_WEBHOOK_URL is a stored credential in n8n vault — requires n8n REST API call to update | credential-rotation-180d.md §N8N_GPS_SYNC_SECRET |
MONGODB_RS0_PROMETHEUS_PASSWORD | n8n-bms4 | MongoDB rs0 | db.changeUserPassword('prometheus', '<new>') on PRIMARY via SSH → bms-4 .env + SOPS | mongodb-credential-rotation.md |
MONGODB_RS0_ADMIN_PASSWORD | monitoring, bms-servers | MongoDB rs0 | db.changeUserPassword('admin', '<new>') on PRIMARY via SSH → multiple SOPS files → all consumers (Pinbox24 bms-1 containers must also be updated — see critical note in playbook) | mongodb-credential-rotation.md |
PINBOX24_MONGODB_URI (et_oper user) | et-operational-platform | MongoDB rs0 (w4_db, read-only consumer, authSource=admin) | ⚠️ Pre-check is mandatory before rotating (added #5396). First test the current SOPS URI against live rs0 — et-op#1363 was a false-premise ROTATE request against a live-working credential; a documented row/playbook + this pre-check would have caught it up front. Only if the current value genuinely fails auth: db.getSiblingDB("admin").updateUser("et_oper", {pwd: "<new URI-safe hex>"}) on the rs0 PRIMARY (bms-2 145.239.133.104 / bms-3 51.68.155.224) using MONGODB_RS0_ADMIN_USER + MONGODB_RS0_ADMIN_PASSWORD from secrets/administration.env.sops (canonical — bms-servers.env.sops holds the password but no admin-user key, so it is insufficient on its own; phase A finding) → rebuild the URI with a URI-safe hex password (no percent-encoding needed) → update PINBOX24_MONGODB_URI in secrets/et-operational-platform.env.sops → distribute to Vercel only (this SOPS file has no bms-4 deploy step): both et-op projects, PROD prj_ziLl911… + STAGING prj_lli8aw…, via secrets-sync.yml. Proven end-to-end twice (2026-07-09 Phase A/B, secrets-rotation-log.md). Worker-reachability caveat: the canonical admin creds live in administration.env.sops, which the bms-4 secret-manager worker cannot decrypt — a worker-run rotation must source them from administration.env.sops recipients or escalate. | w4-mongodb-credential-rotation.md §et-operational-platform (et_oper) |
NEXCON_API_KEY | monitoring | Nexcon | Nexcon REST API key rotation | nexcon-api-key-rotation.md |
TELEGRAM_BOT_TOKEN | n8n-bms4 | Telegram BotFather | Generate new token via @BotFather on Telegram (does NOT require Google login or 2FA) — fully scriptable via Telegram Bot API if the old token is still valid | telegram-bot-token-rotation.md |
TELEGRAM_P24_CLAUDE_BOT_API_KEY | n8n-bms4 | Telegram BotFather | Same as above | telegram-bot-token-rotation.md |
SUPABASE_SERVICE_ROLE_KEY / SUPABASE_SERVICE_KEY | monitoring, n8n-bms4, vps-h1, brandpilot, art-agency | Supabase Management API | POST /v1/projects/{ref}/api-keys → update all SOPS files + Vercel + n8n credentials + GH Secret. Requires SUPABASE_ACCESS_TOKEN to be valid. Note: Management API response masks the key value — must capture immediately from response. ⚠️ Gate may abort mid-rotation (#4626, #5319): the Management API can mint an HTTP 200, well-formed sb_secret_... key that then returns 401 Invalid API key from the project data API (control-plane/data-plane desync). The Step 2.5 liveness gate makes this fail-safe (deletes the dead key before distribution) but does not guarantee success — rotation can abort and require a human to check org/project linkage, so it is not unconditionally hands-off. bms-4 worker reachability: the only worker-decryptable management-token copy is SUPABASE_MANAGEMENT_TOKEN in brandpilot.env.sops (not role-secret-manager / administration) — this is the sole worker-reachable path for Supabase project-key rotation today. | supabase-service-key-rotation.md |
SUPABASE_ANON_KEY / NEXT_PUBLIC_SUPABASE_ANON_KEY | monitoring, n8n-bms4, brandpilot | Supabase Management API | Same as above but for anon key — including the same ⚠️ 5319 caveats: the Step 2.5 liveness gate may abort mid-rotation, and the sole bms-4-worker-reachable management token is SUPABASE_MANAGEMENT_TOKEN in brandpilot.env.sops. | supabase-service-key-rotation.md |
INSPECTION_WEBHOOK_SECRET | n8n-bms4 | Internal (webhook HMAC) | Random hex → bms-4 .env + SOPS → n8n credential + consumer update | Internal |
V32_REDIS_PASSWORD (OVH Valkey kr40258-001) | pinbox24-w3 | OVH hosting/privateDatabase | POST /hosting/privateDatabase/kr40258-001/user/default/changePassword via OVH Python SDK. Max 30 chars. Requires valid OVH consumer key in monitoring SOPS with this route granted. Update SOPS + v32-prod env files on bms-1 + recreate container. | valkey_rotate_v2.py (session scratchpad). #2415 |
V42_REDIS_PASSWORD (OVH Valkey kr40258-001) | pinbox24-w4 | OVH hosting/privateDatabase | Same as above for v42-prod. Update SOPS + v42-prod env files on bms-1 + recreate container. | valkey_rotate_v2.py (session scratchpad). #2415 |
OVH_INFRA_CLIENT_ID / OVH_INFRA_CLIENT_SECRET (OVH OAuth2 IAM service account, main realm eu.api.ovh.com) | ovh-api | OVH API (server mgmt) | Migrated to Tier 1 — 2026-08-05, round 6 (#5410), live-verified. OVHcloud OAuth2 CLIENT_CREDENTIALS service account — pure-API, no browser. Rotation = delete + recreate the client via its own Bearer token + re-attach IAM policy: scripts/rotate/ovh-api-credentials.sh (default invocation = mint-only, SKIP_CLEANUP=1). Ran on bms-4 against main@651a8984 (PR #5494, review-iteration-2 HEAD) with a clean SOPS (OVH_INFRA_CLIENT_ID/SECRET absent — classic AUTH_MODE selected as expected): mint clientId=EU.7b9eafea413fb063 → IAM policy attach hard-ok (no WARN, no 409/400 — the run-unique, dot-sanitized POLICY_NAME from PR 5489 worked as designed) → verify GET /dedicated/server → 200 → dispatched to sops-sync-receiver.yml (run 30969182799, conclusion=success) → live on main commit a3026658. Independently re-verified in the same session, not just the script’s exit code: (1) GET /v2/iam/policy (signed separately with the admin bootstrap credential, not the script’s own token) shows policy p24-infra-rotate-2026-08-05-EU-7b9eafea413fb063-policy (id 904f7aac-0bae-4f84-a5aa-f9f6e7404085) granting dedicatedServer:apiovh:* on urn:v1:eu:resource:dedicatedServer:* to identity urn:v1:eu:identity:credential:kr40258-ovh/oauth2-EU.7b9eafea413fb063 — genuinely attached, unlike round 4’s silent 409. (2) A fresh OAuth2 Bearer token minted directly from OVH_INFRA_CLIENT_ID/SECRET as read from SOPS (not the script’s cached token) passed both GET /dedicated/server → 200 (lists all 3 dedicated servers) and the stronger GET /dedicated/server/{name}/boot → 200 (per ovh-api-operations.md §3 guidance that a route closer to actual usage is a better smoke test than a generic list call). Legacy OVH_INFRA_APP_KEY/APP_SECRET/CONSUMER_KEY triple is untouched (SKIP_CLEANUP=1, the default — this run was mint-only by design) and remains the fallback credential; its revocation is a deliberate, separate follow-up via CLEANUP_TARGET=legacy SKIP_CLEANUP=0 bash rotate/ovh-api-credentials.sh, not done in this pass. History of rounds 1-5 (halts, the wrong-Application-delete bug, the /me verify-endpoint bug, the round-4 silent IAM 409, the round-5 policy-name dot-format bug) preserved in ovh-api-operations.md §5a for context. | ovh-api-operations.md §5a, scripts/rotate/ovh-api-credentials.sh |
CLAUDE_PROXY_SECRET | n8n-bms4, vps-h1 | Internal (deprecated) | Random hex → both SOPS files. Note: claude-proxy was removed 2026-06-24; this key is vestigial | Internal |
WASABI_PROCESSOR_ACCESS_KEY_ID + WASABI_PROCESSOR_SECRET_ACCESS_KEY | whatsup — not yet present, ADD pending (#2128) | Wasabi IAM API | boto3 iam.wasabisys.com: create IAM user wa-db-processor-reader, attach a read-only inline policy scoped to the exact upload-bucket ARN (s3:GetObject, s3:ListBucket, s3:GetBucketLocation only — never a broader Resource: "*"), create_access_key. Blocked as of 2026-08-01: the only WASABI_ADMIN_ACCESS_KEY/WASABI_ADMIN_SECRET_KEY copy the standard bms-4 secret-manager worker can decrypt is in whatsup.env.sops, which is the stale copy per the Master-key deliverability re-check above — the valid key lives in role-secret-manager/administration only. Do a list-users verify-first call before attempting create-user; if it 401/403s, this is blocked on the #3366 SOPS-sync, not on this issue. Also check list-users for an existing wa-db-processor-reader first — Wasabi’s 2-key-per-user limit (#2650) means a second create fails if one already exists. Bucket name/ARN and endpoint must be read from the real upload bucket configuration, never guessed. | wasabi-key-rotation.md |
MAILGUN_ADMIN_API_KEY | monitoring, role-secret-manager | Mailgun (account-level admin key) | Moved from Tier 3 (#4400, 2026-07-30 — this row previously sat in the Tier 3 table with a note explaining it no longer belonged there; it’s now filed correctly). POST /v1/keys create/rotate/delete → 200 on both api.mailgun.net (US) and api.eu.mailgun.net (EU). Auth: existing valid Admin/Developer-role key, HTTP Basic api:<key>. Reachable by the bms-4 worker via monitoring.env.sops. Caveat: confirm the new key carries admin scope before retiring the old one. | POST /v1/keys (account-level) → capture value from response (shown once) → verify → DELETE /v1/keys/{id} old key. Update both monitoring.env.sops AND role-secret-manager.env.sops. |
MAILGUN_API_KEY (domain sending key) | monitoring | Mailgun | Reclassified 2026-08-04 — previous premise (“dashboard-only Sending Key”) was never checked against the same /v1/keys endpoint used for the admin key above; it is the same endpoint, differentiated by payload. | POST /v1/keys with kind=domain, role=sending, domain_name=<domain> using MAILGUN_ADMIN_API_KEY for auth — mints a key scoped to /messages + /messages.mime for that domain only. Same US/EU host split as above. Rate limit ~50 req/s, not a practical constraint. |
SMTP_PASSWORD (Mailgun EU, monitoring/Alertmanager) | monitoring | Mailgun EU | Reclassified 2026-08-04 — /v3/domains/{domain}/credentials was assumed dashboard-only but has a full CRUD API. | PUT /v3/domains/{domain_name}/credentials/{login} (existing login) with a caller-supplied password value, authenticated with MAILGUN_ADMIN_API_KEY (Basic api:<key>) — no dashboard step. POST to the same collection creates a new login if needed; DELETE removes one. Confirmed identical path/method on api.mailgun.net and api.eu.mailgun.net (only the dashboard host differs, app.eu.mailgun.com vs app.mailgun.com — the API host does not gate by dashboard region). |
V32_MAILGUN_PASSWORD / other per-domain Mailgun SMTP or API-key values in pinbox24-w3.env.sops / pinbox24-w4.env.sops | pinbox24-w3, pinbox24-w4 | Mailgun | Same reclassification as the two rows above — these are the same Mailgun API surface, just different domains/SOPS files. Not yet exercised live for these specific domains — verify-first before treating as unconditionally hands-off (see caveat in mailgun-mongodb-stale-credential-hang.md about downstream container recreate). | Same /v1/keys (sending key) or /v3/domains/{domain}/credentials/{login} (SMTP) calls as above, scoped to the W3/W4 domain |
MAILGUN_AUTH_TOKEN | pinbox24-backends | Internal (mailgun-prod microservice, bms-1) | Not actually a Mailgun-provider credential — self-issued. This is our own mailgun-prod container’s bearer token (added 2026-07-08), verified against our own service and a GitLab CI variable (AUTH_TOKEN in pinbox24/p24-ms-mailgun), not against any Mailgun dashboard. No provider portal involved at all. | Generate random token → update 3 locations: secrets/pinbox24-backends.env.sops, live mailgun-prod container env on bms-1 (recreate, not restart), GitLab CI variable AUTH_TOKEN in pinbox24/p24-ms-mailgun (via GITLAB_ADMIN_PAT / pinbox24-gitlab.env.sops) |
AI_GATEWAY_API_KEY | et-operational-platform | Vercel AI Gateway (REST API) | Added 2026-08-03 (#1305, et-op#1305) — classified Tier 1, NOT dashboard-only: POST https://api.vercel.com/v1/api-keys?teamId=<team> with body {"purpose":"ai-gateway","name":…} using a Vercel access token (VERCEL_ADMIN_TOKEN in monitoring.env.sops); response returns apiKeyString (shown once — capture immediately) + id. Validate the new key against GET https://ai-gateway.vercel.sh/v1/models (expect 200) before persisting. Revoke old via DELETE /v1/api-keys/{id}?teamId=<team>. Team team_RuXLTWx5DXK6n0HUKoOUTGY3 (devp24coms-projects), et-op project prj_ziLl911FOYLAeukQujL4NjxR4eWy. Distribute to et-op Vercel env (development + preview — Vercel-hosted prod uses OIDC, no key). Replaces ANTHROPIC_API_KEY as et-op’s only AI-provider credential (drop tracked in p24-infra#2889). | secret-manager.md §Distribution chain (no per-key playbook yet) |
DISCORD_WEBHOOK_URL / P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL | n8n-bms4, vps-h1 / monitoring, et-operational-platform (per consumer — see §Key cross-file duplications) | Discord Bot REST API | Reclassified Tier 1 2026-08-04 (#2889/#2666), superseding an earlier same-day Tier 2 classification. First correction claimed Tier 2 via Playwright (login + TOTP); that was itself wrong — a live test the same day found Playwright’s login hits an unsolvable hCaptcha challenge (Discord/Cloudflare bot detection, not a selector bug — see docs/playbooks/discord-provisioning.md status note). The REST API path (POST/GET/DELETE on /channels/{id}/webhooks, DELETE /webhooks/{id}) needs only DISCORD_BOT_TOKEN + the bot holding Manage Webhooks on the target channel — proven live end-to-end (create + delete) against the real p24-infra guild the same day. No login, no TOTP, no CAPTCHA risk — genuinely Tier 1 once the one-time per-channel permission grant is done (a human action, but a one-time server-config step, not part of each rotation). Debugging note: every ad-hoc REST call against this API must send a User-Agent header — Discord/Cloudflare returns JSON error 40333 (“internal network error”) and silently blocks the request otherwise; this cost significant time to trace before finding it in Discord’s own documented error-code table. Autonomous: discord-provisioning.py --action create-webhook / --action delete-webhook — tries REST automatically, falls back to the (currently hCaptcha-blocked) Playwright path only if DISCORD_BOT_TOKEN is absent or the REST call 403s. Full flow: docs/playbooks/discord-provisioning.md §Step 2b. | discord-provisioning.md §Step 2b, discord-credential-rotation.md |
Deploy-key autonomy via GitHub App (GITHUB_APP_ID / GITHUB_APP_PRIVATE_KEY_B64):
The GitHub App (GITHUB_APP_ID = 2109526, installation 142989418) generates short-lived installation tokens (1h TTL) via RS256 JWT signed with GITHUB_APP_PRIVATE_KEY_B64. No PAT or browser required. This enables autonomous rotation for:
- Creating / deleting deploy keys on any repo (
POST /repos/{owner}/{repo}/keys) - Opening issues, PRs, comments, triggering dispatches
- Any operation within the App’s permission scope
To rotate the GitHub App private key itself: go to github.com/settings/apps/2109526 → Private keys → Generate new key (browser required — see ❌ Tier 3 below for GITHUB_APP_PRIVATE_KEY_B64).
Tier 1 via App — bms-4 worker consumers (#4068, plan #4065 Wave A): day-to-day bms-4 worker
git/secret operations no longer read the GH_TOKEN / GITHUB_PAT_* PATs directly. They mint a
1h installation token on demand via bin/gh-app-token.sh (wraps scripts/gh-app-token):
- git clone/push over HTTPS →
bms-4/git-credential-gh-token.sh(App-token-first,GH_TOKENfallback). - post-rotation
gh secret set→scripts/lib/gh_app_secret_set.sh(scopes asecrets:writetoken). bin/gh-app-token.sh --checkvalidates creds without printing the token.
§2.4 permission audit (2026-07-20, App 2109526 / installation 142989418 / account radieu,
repository_selection: all): originally granted issues:write, secrets:write, contents:write, metadata:read, pull_requests:write, administration:write. actions:write and workflows:write were
gaps at this point — see the 2026-08-09 update below for how both were closed. #4068’s own code
consumers (git→contents, gh secret set→secrets) were fully covered even before that.
2026-08-09 update (#5841, three attempts — third one succeeded): workflows: write was added
to the App manifest to unblock auto-fix-gh-actions.yml (PR #5927, attempt 1), but the first live
dispatch 422’d at token-mint with a blanket “permissions requested are not granted” error. A
disposable probe workflow (test/5841-app-token-permission-probe, PR #5940) isolated the cause by
requesting permission subsets individually: contents:write + issues:write alone succeeds; adding
either workflows:write or actions:write alone failed. Root cause: editing the App’s
manifest does not retroactively grant a new permission to an already-installed instance — the
installing account must separately accept a pending permission-upgrade prompt at
github.com/settings/installations/142989418. actions:write had the added problem of never having
been requested in the manifest at all. Attempt 1 was reverted (auto-fix-gh-actions.yml is
dispatched by the live gh-actions-monitor n8n automation on every GH Actions failure, not dormant).
Both gaps were then closed and independently verified: actions: write was added to the App
manifest, and — the step attempt 1 was missing — the pending permission-upgrade prompt at
github.com/settings/installations/142989418 was accepted. bin/gh-app-token.sh --check --permissions <perm> on bms-4 confirmed all four of contents:write, workflows:write,
actions:write, issues:write individually PASS against the live installation. The migration was
then re-applied (attempt 3) and verified working end-to-end via a real workflow_dispatch run of
auto-fix-gh-actions.yml. Current granted set: issues:write, secrets:write, contents:write, metadata:read, pull_requests:write, administration:write, workflows:write, actions:write — no
outstanding App-permission gaps as of 2026-08-09. Full three-attempt writeup:
docs/playbooks/gh-pat-rotation.md §0.5. Lesson for future permission-gap diagnosis: a manifest
edit alone is not proof a permission is live on a given installation — always verify at the
installation level (bin/gh-app-token.sh --check or the PR #5940 probe pattern), and if the
manifest was just edited, check for a pending acceptance prompt at
github.com/settings/installations/<id> before assuming the grant took effect.
Playbook: docs/playbooks/n8n/github-app-token-n8n.md, docs/playbooks/github-pat-rotation.md §App-first
General/canonical doc — docs/playbooks/github-app-token-integration.md (added 2026-08-09,
#5924 follow-up): the three references above (n8n Code node, bms-4 bash worker, GH Actions
composite step) are each scoped to one specific runtime. The new doc generalizes the same
JWT→installation-token pattern for application server code — a Next.js/Vercel API route or any
other Node/TS backend that wants to mint its own installation token inline instead of depending on
a long-lived classic PAT. It includes a Node/TypeScript reference implementation (zero-dependency
crypto-based, plus an @octokit/auth-app alternative), a migration checklist for an existing
classic-PAT consumer, and the et-operational-platform commit.ts case (#5924) as a worked example
of the gap this fills.
⚠️ Tier 2 — Semi-Autonomous (automation unblocked by a prerequisite)
These credentials can be rotated without a human present once a prerequisite condition is met. The prerequisite is checked first; if it fails, fall back to Tier 3.
| Secret | SOPS file(s) | Provider | Prerequisite | How (once unblocked) | Playbook |
|---|---|---|---|---|---|
GH_TOKEN, GITHUB_PAT_ALL_WRITES, GITHUB_PAT_P24_INFRA | monitoring, n8n-bms4-gh, vps-h1 | GitHub PATs | Unblocked 2026-08-01 (#4069): login credentials GITHUB_PASSWORD (added #4977) and GITHUB_TOTP_SECRET both present and valid in secrets/n8n-bms4.env.sops; Playwright + Chromium + oathtool provisioned on bms-4 (playwright_enabled: true, ansible/inventory/host_vars/bms-4.yml) | Autonomous worker flow: node scripts/rotate/github-pat.js (per PAT) → verify against api.github.com/user → update n8n-bms4-gh.env.sops (+ monitoring/vps-h1 GH_TOKEN copies as applicable) + GH Secrets. Mechanically build-validated (TOTP generation, --dry-run login-page navigation) in #4069; live E2E run against a real PAT still pending — see #4065 §3.3 (disposable TEST_PAT_ROTATION_4065) before treating this row as fully proven. | github-pat-rotation.md §Option A, infra/agent-prompts/worker-secret-manager.md §Tier 2 — Playwright rotation pattern (GitHub PAT) |
GITHUB_TOTP_SECRET | n8n-bms4 | GitHub (2FA seed) | None — this key is itself the prerequisite for the row above | No self-rotation path; this is a base32 TOTP seed obtained once from github.com/settings/security → 2FA → reconfigure authenticator → reveal seed (Tier 3, human-only). Safety note: grants anyone holding SOPS+age keys the ability to bypass 2FA on the radieu GitHub account for the PAT UI — same trust boundary as GITHUB_APP_PRIVATE_KEY_B64 in the same file. Treat like a root credential; rotate it (get a new seed) if exposure is suspected. | docs/plans/plan-4065-github-pat-autonomous-rotation.md §3.2.2 |
| (deprecation note, #4068) | — | — | — | bms-4 worker consumers of these PATs are migrating to Tier-1-via-App (bin/gh-app-token.sh, see the App section above). The PATs still need Tier-2 rotation until removed from SOPS; removal is gated on rollout + #4180 + telegram-bot resolution. Note the bms-4 PATs live in n8n-bms4-gh.env.sops (not n8n-bms4.env.sops). | — |
OPENAI_ADMIN_KEY | monitoring | OpenAI | radieu@gmail.com Google session active in Playwright browser profile | Playwright: platform.openai.com/api-keys → delete old → Create new | playwright-rotation-template.md |
OPENAI_API_KEY (et-op project key) | et-op Vercel + secrets/et-operational-platform.env.sops | OpenAI | Autonomous — CORRECTED 2026-08-04, prior “blocked”/“permanently impossible” wording was wrong. POST /organization/projects/{project_id}/api_keys genuinely has no create method (confirmed twice: 403 on 2026-06-30 runs 28462756890, 404 on 2026-08-04 run #30893243746) — that finding stands. But it was the wrong resource: POST /organization/projects/{project_id}/service_accounts (Default project proj_YnpqoGhWyCbwpeTUcahfY5sK) works with the existing OPENAI_ADMIN_KEY and returns a usable key in one call — the same pattern already proven for WAP_OPENAI_KEY_MINI (docs/playbooks/openai-key-management.md §4), just never previously applied to this key. Successfully used to rotate this key 2026-08-04 (issue #2889, docs/secrets-rotation-log.md same date) — the pre-rotation key tested dead (401) beforehand, so et-op’s OpenAI features were likely already down. | Autonomous: POST /organization/projects/proj_YnpqoGhWyCbwpeTUcahfY5sK/service_accounts {"name": "et-op-production-<date>"} with OPENAI_ADMIN_KEY → svc.api_key.value is the new key (returned once) → update et-op Vercel prj_ziLl911FOYLAeukQujL4NjxR4eWy (team team_RuXLTWx5DXK6n0HUKoOUTGY3) + secrets/et-operational-platform.env.sops via scripts/sops-set.ps1 → delete the superseded service account (DELETE .../service_accounts/{id}) once confirmed unused. Full walkthrough: docs/playbooks/openai-key-management.md §4. | openai-key-management.md §4, static-api-key-incident-rotation.md |
HSTGR_N8N_MCP_TOKEN | monitoring (.env.local) | n8n (Hostinger) | n8n UI is reachable (no additional auth beyond n8n credentials) | n8n Settings → API → Regenerate key | hstgr-n8n-mcp-token-rotation.md |
VPS_ROOT_SSH_KEY | GH Secret only | SSH ed25519 | All servers are reachable with current root key | ssh-keygen → append new pub to all servers’ authorized_keys → verify → swap local id_ed25519 → remove old pub → GH Secret update | credential-rotation-180d.md §6, ssh-key-rotation.md |
IONOS_API_TOKEN_PREFIX + IONOS_API_TOKEN_ENCRYPTION | administration | IONOS API | IONOS account session active OR existing API token valid (self-rotation not confirmed) | IONOS API: POST /v1/um/users/{id}/tokens (requires existing token with sufficient scope) | ionos-api-token-rotation.md |
P24_INFRA_MEZMO_SERVICE_KEY | monitoring | Mezmo API | Mezmo POST /v1/config/keys returns 200 (was returning 400 in 2026-06-15 — verify first) | Mezmo API: create new service key, delete old | mezmo-key-rotation.md |
WASABI_ADMIN_ACCESS_KEY + WASABI_ADMIN_SECRET_KEY | administration, whatsup | Wasabi (root account) | Wasabi root account console login (Tier 3) needed if key is invalid; if key is valid, IAM self-rotation is not available for root keys — must be done in console | See Tier 3 below | wasabi-key-rotation.md §Human-action |
ANTHROPIC_API_KEY | n8n-bms4 | Anthropic Console | radieu@gmail.com Google session active in Playwright profile | Playwright: console.anthropic.com/settings/keys → delete old → Create new | anthropic-api-key-rotation.md |
DISCORD_BOT_TOKEN | monitoring | Discord | Corrected 2026-08-04 — previously listed in Tier 3 below as “no API for bot token rotation,” but its own dedicated playbook already documents this as Playwright-automatable (Developer Portal Bot → Reset Token is a UI flow, not an API gap — same automation class as the webhook row above). Same DISCORD_TOTP_SECRET prerequisite + human-supervised foreground session. | Playwright Option A (MCP session) or Option C (scripts/rotate/discord-bot-token.js on the bms-4 heavy worker, job_type: "heavy") — both documented end-to-end including SOPS + GH Secret write, n8n credential update, and container restart. Full flow: docs/playbooks/rotate-discord-bot-token.md. Manual fallback (Option B) if the browser session/TOTP prerequisite is unavailable. | rotate-discord-bot-token.md |
bare_metal_server_*_root_password + HOSTINGER_ROOT_PASSWORD | bms-servers | Linux passwd | Server reachable via current root SSH key | `ssh root@ | chpasswd’` — random hex passphrase, update bms-servers SOPS |
❌ Tier 3 — Manual Only (human required, no automation path)
These secrets have no API for key creation/rotation, require a UI that cannot be browser-automated due to 2FA, or are root-level credentials that bootstrap all automation.
Before filing a classic-GitHub-PAT consumer here: check
docs/playbooks/github-app-token-integration.md first — App
2109526 may already cover the permission and repo, turning what looks like a recurring
human-rotation burden into a one-time integration (#5924).
| Secret | SOPS file(s) | Provider | Why manual | What the human must do | Claude’s role after human action | Playbook |
|---|---|---|---|---|---|---|
CF_GLOBAL_API_KEY / CLOUDFLARE_GLOBAL_API_KEY | monitoring, administration | Cloudflare | Global API Key has no rotation API — UI only via “Roll” button | dash.cloudflare.com → Profile → API Tokens → Global API Key → Roll. Copy new value. | Update SOPS (monitoring + administration), GH Secrets, then trigger auto-rotation of dependent CF tokens | cloudflare-credential-rotation.md |
SUPABASE_ACCESS_TOKEN (sbp_...) | administration (+ role-scoped copy ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN in role-secret-manager) — not monitoring (#4585) | Supabase | Personal access token — browser-only creation at app.supabase.com/account/tokens | Create new token at Supabase dashboard; copy immediately | Update administration.env.sops + role-secret-manager.env.sops + GH Secret SUPABASE_ACCESS_TOKEN; validate against Management API. Do not add it to monitoring.env.sops — that file is deployed to vps-i1 and read via env_file: .env by 4 containers (#2620, PR #4700 rejected). Write path currently broken — see #4626 | supabase-service-key-rotation.md |
discord_radieu_password | monitoring | Discord | User account password — no API. Deliberately kept manual even though the other Discord keys are Tier 2: every automated Discord action logs in with this password; scripting its own rotation risks a mid-flow failure locking out the one credential everything else depends on. | Discord account settings | Update SOPS | discord-credential-rotation.md |
BMS4_N8N_ADMIN_PASSWORD | n8n-bms4 | n8n (self-hosted bms-4) web UI | n8n web login password for radieu@gmail.com — no REST API to change it; requires n8n Settings UI or a bcrypt UPDATE on the DB. Corrected 2026-08-11 (#6088): the live key is BMS4_N8N_ADMIN_PASSWORD (confirmed live-login test, HTTP 200), not BMS4_N8N_RADIEU_PASSWORD — that key name was superseded 2026-08-01 (#4918) and this row still cited it for over a week. BMS4_N8N_RADIEU_PASSWORD remains in SOPS as dead data (confirmed 401 on login), not consumed by anything. | Log into bms-4 n8n → Settings → Change password (or bcrypt-hash locally + UPDATE "user" in bms-4-n8n-postgres-1) | Update n8n-bms4 SOPS + live bms-4 .env (consumed by scripts/rotate/n8n-bms4-api-key.ps1 Playwright login) | n8n/n8n-bms4-api-key-rotation.md |
ATRAX_AUTH_STRING / ATRAX_USERNAME / ATRAX_PASSWORD / ATRAX_CLIENT_ID / ATRAX_CLIENT_SECRET | monitoring, n8n-bms4 | Atrax (fleet system) | Atrax credential management is UI-only in their web portal (no public API). Corrected 2026-08-05 (#1713): these keys were removed from vps-h1.env.sops back on 2026-07-05 (commit 7fb1f7be, post-WAHA-decommission cleanup — “no vps-h1 service consumes them”); this row and the playbook still said 3 files for a month. Live consumer is bms-4/docker-compose.yml (n8n main + 3 workers). | Log into Atrax admin portal → change credentials or request new OAuth client | Update the 2 SOPS files (monitoring, n8n-bms4) where these appear; restart consumers | atrax-credential-rotation.md |
GITHUB_APP_PRIVATE_KEY_B64 | n8n-bms4 | GitHub App | Private key generation requires github.com/settings/apps/{id} UI | github.com/settings/apps/2109526 → Private keys → Generate new → download PEM | Base64-encode PEM → update SOPS → merge → secrets-sync.yml deploys to bms-4 | n8n/github-app-token-n8n.md |
OVH_APPLICATION_KEY / OVH_APPLICATION_SECRET / OVH_APPLICATION_CONSUMER_KEY / OVH_ENDPOINT | monitoring | OVH API (server mgmt — p24-infra-claude, admin scope ovh-role-admin) | Classic AK/AS/CK — consumer key needs browser OAuth to rotate the CK itself. #5410: the main realm (eu.api.ovh.com) does support OAuth2 IAM service accounts (live-confirmed 200), so this credential could migrate to Tier 1 the same way as OVH_INFRA_* — but today it is the admin bootstrap CK that mints the OAuth2 clients, so it is deliberately kept as the classic root credential. Rotating the CK stays browser-OAuth (Tier 3) until/unless it too is migrated. Upgraded 2026-08-04 (#5410, Option A): re-minted via request_consumerkey() with the full legacy route set plus the two write routes the migration needs (POST /me/api/oauth2/client, POST /v2/iam/policy//iam/policy) — one human browser-authorization, live-confirmed both routes granted (18 total rules). This CK no longer needs a second browser trip for this specific OAuth2-mint action; rotating the CK itself is still Tier 3 (browser OAuth, unchanged). | Create app at api.ovh.com/createApp/; POST to /1.0/auth/credential (Claude can do this); authorize the validationUrl in a browser | Write AK+AS+CK to monitoring SOPS; no container restart (keys read by the Claude agent at runtime) | ovh-api-operations.md §7 |
OVH_APP_KEY / OVH_APP_SECRET / OVH_CONSUMER_KEY | monitoring | OVH API (cost-exporter) | Classic AK/AS/CK, browser OAuth to rotate the CK. #5410: main realm supports OAuth2 (confirmed) — migratable to Tier 1 as a follow-up once cost-exporter is updated to accept an OAuth2 Bearer token; not migrated in #5410 (scoped to OVH_INFRA_*). | Create new app + token at OVH API portal; authorize via browser | Update SOPS; restart cost-exporter | ovh-api-operations.md §7 |
SYS_APP_KEY / SYS_APP_SECRET / SYS_CONSUMER_KEY (+ split-file SYS_INFRA_*) | monitoring, ovh-api | SoYouStart/Kimsufi API (api.soyoustart.com) | Permanently Tier 3 (#5410, live-confirmed). The SoYouStart realm has no /me/api/oauth2/client endpoint — a valid, authenticated CK still gets 404 "Got an invalid (or empty) URL". OAuth2 IAM service accounts are an OVHcloud-realm feature SoYouStart does not expose, so the browser-CK model is the only path until OVH ships it there. Health note: SYS_INFRA_APP_KEY was found invalid (“This application key is invalid”) during the #5410 probe — needs a secret-manager rotation independent of this migration. | Create new app + token at the SoYouStart API portal; authorize via browser | Update SOPS; restart cost-exporter | ovh-api-operations.md §7 |
GCP_SA_KEY_JSON | administration | Google Cloud | Service account key creation requires console.cloud.google.com UI or gcloud with owner IAM | GCP Console → IAM → Service Accounts → key → Add Key | Update SOPS administration | google-oauth-credential-rotation.md |
GITLAB_ADMIN_PAT | administration + pinbox24-gitlab (#4796, atomic — same value in both files) | GitLab | PAT creation requires GitLab UI with 2FA | GitLab → User Settings → Access Tokens → Create | Update SOPS administration AND pinbox24-gitlab.env.sops (narrow 2-recipient copy — developer + bms-4 worker key — created so the bms-4 infra-task worker can decrypt it; administration.env.sops is developer-only and never reaches any server) | — |
GITLAB_SECRET_MANAGER_PAT | role-secret-manager | GitLab | ⚠️ admin-equivalent until remediated (#3504) — currently shares its value with GITLAB_ADMIN_PAT, not a separately-scoped token. GitLab SaaS blocks token creation via API (only rotation works, per feedback_gitlab_saas_token_creation_unavailable), so minting a real api-scope-only replacement requires GitLab UI with 2FA. Rotating either GITLAB_ADMIN_PAT or GITLAB_SECRET_MANAGER_PAT today silently breaks the other until this is de-aliased — check #3504 before rotating GitLab PATs. | GitLab → User Settings → Access Tokens → Create: name p24-secrets-manager-gitlab, scope api only, explicit expiry (see #3504 for the full verification checklist) | Update secrets/role-secret-manager.env.sops; confirm the two PATs hash to different values post-rotation | secret-manager.md |
GH_PAT_ADMIN | administration | GitHub | PAT with admin scope — GitHub UI + 2FA required; no API equivalent | github.com/settings/tokens → regenerate. Note: For most automation (deploy keys, issues, PRs) use the GitHub App instead — only fall back to GH_PAT_ADMIN for operations outside the App’s scope. | Update SOPS administration | github-pat-rotation.md |
LINKEDIN_ACCESS_TOKEN / LINKEDIN_CLIENT_ID / LINKEDIN_CLIENT_SECRET | n8n-bms4 | LinkedIn OAuth | OAuth 2.0 flow requires browser authorization on linkedin.com; token lifetime ~60 days | Complete OAuth flow via browser; copy new access_token from callback | Update n8n-bms4 SOPS + n8n credential vault | linkedin-credential-rotation.md |
GOOGLE_CLIENT_ID / GOOGLE_CLIENT_SECRET | n8n-bms4 | Google OAuth | OAuth client management requires console.cloud.google.com UI | GCP Console → APIs & Services → Credentials → reset secret | Update SOPS | google-oauth-credential-rotation.md |
GOOGLE_OAUTH_WEB_CLIENT_ID / GOOGLE_OAUTH_WEB_CLIENT_SECRET | whatsup | Google OAuth | Same as above | Same as above | Update whatsup SOPS | google-oauth-credential-rotation.md |
GOOGLE_ADS_DEVELOPER_TOKEN / GOOGLE_ADS_CLIENT_ID / GOOGLE_ADS_CLIENT_SECRET / GOOGLE_ADS_REFRESH_TOKEN | art-agency | Google Ads / GCP | Google Ads Developer Token: manual approval process; OAuth client: GCP console | Google Ads Console + GCP Console | Update art-agency SOPS | google-oauth-credential-rotation.md |
PINBOX24_SMTP_PASSWORD | bms-servers | SMTP (unknown provider) | SMTP credential from Pinbox24 external provider | Provider dashboard | Update bms-servers SOPS | mailgun-credential-rotation.md |
SUPABASE_JWT_SECRET / IAM_ENC_KEY | whatsup | Internal crypto key | Not rotatable via API — changing JWT secret invalidates all existing sessions immediately | Generate new random 32-byte key; plan a maintenance window (all sessions invalidated) | Update whatsup SOPS; restart all consumers that sign/verify JWTs | encryption-key-rotation.md |
WEBHOOK_SECRET | whatsup | Internal HMAC | Can be rotated (random hex) but requires coordinated update between producer and consumer during a maintenance window | Generate new value | Update whatsup SOPS; restart consumer | encryption-key-rotation.md |
N8N_ENCRYPTION_KEY | n8n-bms4 | n8n (internal) | Rotating the n8n encryption key requires decrypting + re-encrypting ALL stored credentials in n8n’s credential vault — no API. Effectively a full credential migration. | Do not rotate without a full n8n credential migration plan | This is a major operation — document in a separate issue | encryption-key-rotation.md |
AGE_KEY_GHA | GH Secret only | age (SOPS) | Rotating the age master key requires age-keygen + sops updatekeys on all SOPS files + updating the GH Secret. It is the root of trust for all other secrets. | Run age-keygen, re-encrypt all SOPS files with new recipient, update GH Secret | Full procedure in master-keys-bootstrap.md | master-keys-bootstrap.md |
clickup_clickup_api_key | art-agency | ClickUp | ClickUp API token management is UI-only | app.clickup.com → Settings → Apps → API Token → Regenerate | Update art-agency SOPS | clickup-credential-rotation.md |
P24_LOGIN / P24_MD5 / P24_AUTOMATION_KEY | n8n-bms4 | Pinbox24 internal | Pinbox24 app credentials — managed via Pinbox24 admin panel | Pinbox24 admin panel | Update n8n-bms4 SOPS | — |
pinbox24_com_power_user_password | n8n-bms4 | Pinbox24 (app user) | App user password — Pinbox24 admin | Pinbox24 admin panel | Update n8n-bms4 SOPS | — |
GROQ_API_KEY (grog_api_key) | — (.env.local only) | Groq Console | console.groq.com has no public key management API | Groq Console UI → API Keys → Create | Update SOPS when added | groq-credential-rotation.md |
ANTHROPIC_ADMIN_API_KEY (sk-ant-admin…) | monitoring — not yet present, ADD pending (#4970) | Anthropic (Claude Console, org-admin) | Verified against Anthropic’s Admin API docs 2026-08-01: there is no create endpoint — /v1/organizations/api_keys is list/get/update only, and the docs state plainly that “new API keys can only be created through the Claude Console for security reasons.” Provisioning additionally requires the admin org role, and this org’s Console login is Google SSO (radieu@gmail.com). No bootstrap exists either: zero ANTHROPIC_* keys remain in any SOPS file after the 2026-07-06 revoke-only operation (#2970). The Tier 2 ANTHROPIC_API_KEY Playwright row does not transfer — different Console surface, higher privilege, and the Playwright prerequisite is provisioned on bms-4 only. The org:admin OAuth alternative is also human (interactive browser login, short-lived token; WIF setup itself needs an org:admin token, which Admin API keys cannot obtain). | platform.claude.com → Settings → Admin keys → Create Admin API key (name p24-infra-cost-exporter) → copy immediately (shown once) | Add to monitoring.env.sops (append, not sed-replace) → PR → merge auto-triggers secrets-sync.yml, which writes vps-i1 /opt/p24-infra/monitoring/.env and force-recreates cost-exporter. No GH Secret / Vercel / n8n surface. Verify GET /v1/organizations/cost_report → 200 (status code only) and anthropic_token_spend_usd present on 127.0.0.1:9210/metrics. Deactivating an old key IS API-automatable (POST /v1/organizations/api_keys/{id} {"status":"inactive"}) — so rotation is human-mint + autonomous-revoke, verify-first. | anthropic-admin-api-key.md |
GOOGLE_DRIVE_CLIENT_ID / GOOGLE_DRIVE_CLIENT_SECRET | whatsup — not yet present, ADD pending (#2128, human-action #5028) | Google Cloud Console | Same constraint as the existing GOOGLE_CLIENT_ID / GOOGLE_OAUTH_WEB_CLIENT_ID rows above: OAuth client management requires console.cloud.google.com UI. This is a Desktop app client (p24-wa-a processor gdrive), distinct from the Web client already in this file (GOOGLE_OAUTH_WEB_CLIENT_ID / GOOGLE_OAUTH_WEB_CLIENT_SECRET, created for Supabase Auth in #2077) — gcloud cannot mint installed-app client IDs. | GCP Console → APIs & Services → Credentials → Create Credentials → OAuth 2.0 Client ID → type Desktop app, name p24-wa-a processor gdrive (same project as #2077) | Add both to whatsup.env.sops; unblocks human-action #5029 (part [C] below) and part [D] finalize | google-oauth-credential-rotation.md |
GOOGLE_DRIVE_TOKEN_PATH (token.json for ecotrans.automation@gmail.com) | whatsup — not yet present, ADD pending (#2128, human-action #5029) | Google Drive (installed-app OAuth, appDataFolder scope) | One-time interactive consent flow — no service-account or headless path for a personal Gmail account’s appDataFolder, and no browser-automatable credentials for ecotrans.automation@gmail.com exist in SOPS. Depends on the row above (part [A]) being done first. | Run the installed-app OAuth setup flow (see 5029) in a browser signed in as ecotrans.automation@gmail.com; copy the resulting token.json to a permanent path on bms-4 (never git) | Store the path (not the token contents) in whatsup.env.sops as GOOGLE_DRIVE_TOKEN_PATH | — |
SENTRY_AUTH_TOKEN | monitoring | Sentry (sentry.io) | Re-verified 2026-08-04 — no change. Classic User Auth Tokens (sntryu_...) have no creation endpoint. Newer “Organization Auth Tokens” (sntrys_...) exist and would suit our CI-only use case, but are minted as a side effect of creating an “Internal Integration” — and that creation step is UI-only too (POST /api/0/sentry-apps/ is undocumented/unsupported and itself requires an existing valid token to call, so it doesn’t break the circularity). No CI-bootstrap token type or roadmap signal found. | sentry.io/settings/auth-tokens/ → Create Token — or, better, do this once: Settings → Developer Settings → Custom Integrations → Create New Internal Integration → yields a reusable sntrys_ org token that then never needs dashboard interaction again for its lifetime (only the one-time integration creation is human) | Update monitoring.env.sops + GH Secret + Vercel env (et-operational-platform) | sentry-token-rotation.md |
MONGODB_URI (et-lager) | et-lager | MongoDB Atlas (cluster nextjs-db-dev.ncgd33w.mongodb.net, DB user radieu) | Atlas M0 free-tier DB-user password has no rotation API — Atlas dashboard UI only (mirrors the MONGODB_URI|3|mongodb-atlas|manual note in docs/sops-templates/et-lager.keys). Cluster is outside rs0 (approved isolation exception, see credential-isolation-policy.md). Originally rotated 2026-07-05 (GitLab MR !1, #2703) when the hardcoded URI was moved to process.env.MONGODB_URI; that rotation is backfilled in dev_r_rotation_log (#5754). | MongoDB Atlas dashboard → Database Access → edit user radieu → change/autogenerate password | Rebuild the mongodb+srv:// URI → update MONGODB_URI in secrets/et-lager.env.sops → merge (auto-triggers secrets-sync.yml sync-et-lager → Vercel et-lager project env) → redeploy et-lager to pick up the new value | et-lager-operations.md §Credential Rotation |
V42_PM2_PUBLIC_KEY / V42_PM2_SECRET_KEY | pinbox24-w4 (corrected 2026-08-06, #2731 verification — these keys now live in pinbox24-w4.env.sops, not pinbox24-backends.env.sops; migrated at some point after the 2026-07-08 log entries that originally added them to the latter, as part of the W3/W4 credential-isolation split) | PM2 Plus / pm2.io (bms-1 monitoring agent) | Rotation WON’T-FIX (#2537, closed 2026-07-02) — re-checked 2026-08-04, conclusion unchanged but refined. No in-place rotate endpoint exists. New finding: @pm2/js-api (keymetrics/pm2-io-js-api, actively maintained) exposes an account-level client.bucket.create (POST /api/bucket/create_classic) that CAN mint a new bucket/key pair without touching the dashboard UI — but it needs a one-time human-obtained OAuth refresh token to authenticate the API client, and there’s still no delete-old-bucket-in-place step (create-new + retire-old, same shape as today, just scriptable once bootstrapped). Not pursued: these are read-only monitoring-identity tokens (no process control, no data exposure — risk already accepted in #2537), so the OAuth-token-custody overhead isn’t judged worth it. pm2.io is actively developed (not deprecated), so “migrate off it” isn’t forced by product-maintenance status either. | app.pm2.io → Account Settings → delete server bucket → re-add (mints new pair) | Update pinbox24-w4.env.sops; restart PM2 agent | — |
Key cross-file duplications
The following secrets exist in multiple SOPS files and must be updated in all of them atomically:
| Secret | SOPS files | Note |
|---|---|---|
SUPABASE_SERVICE_KEY / SUPABASE_SERVICE_ROLE_KEY | monitoring, n8n-bms4, vps-h1, brandpilot, art-agency | After rotation, also update GH Secret + Vercel + n8n vault (6 credentials) |
GH_TOKEN | monitoring, n8n-bms4, vps-h1 | May be same or separate tokens depending on current config |
ATRAX_CLIENT_ID / ATRAX_CLIENT_SECRET / ATRAX_USERNAME / ATRAX_PASSWORD | monitoring, n8n-bms4 | Both files must be updated together (not vps-h1 — removed there 2026-07-05, #1713 correction) |
MEZMO_INGESTION_KEY (legacy shared) | monitoring, vps-h1 | Replace with per-host keys per mezmo-key-rotation.md §B-legacy |
DISCORD_WEBHOOK_URL | n8n-bms4, vps-h1 | Webhook URLs do not expire but if regenerated, update both |
WASABI_ADMIN_ACCESS_KEY / WASABI_ADMIN_SECRET_KEY | administration, whatsup | Same root account key — must be updated in both files |
GITHUB_TRIGGER_SECRET | n8n-bms4, vps-h1 | Update both SOPS files + both VPS .env files |
N8N_HU_SP_REPORT_SECRET | n8n-bms4, vps-h1 | Update both files |
BRAND_WEBHOOK_SECRET | n8n-bms4, vps-h1 | Update both files |
N8N_CLOUD_API_KEY | monitoring, n8n-bms4 | Same n8n.cloud API key in both files |
CLAUDE_PROXY_SECRET | n8n-bms4, vps-h1 | Vestigial (claude-proxy removed 2026-06-24); update if rotated |
Rotation automation status summary
| Count | Tier | Description |
|---|---|---|
| ~41 | ✅ Autonomous | Fully scriptable via rotate-credentials.py (GH Actions) or direct CLI/API calls |
| ~10 | ⚠️ Semi-auto | Automation exists but requires a unblocking step (Playwright session, valid admin key) |
| ~28 | ❌ Manual | Human must interact with a UI; Claude handles SOPS update + distribution afterward |
2026-08-04 Tier-3 portal-rotation review (Mailgun/PM2/Sentry/OVH, issue #4851): reclassified all Mailgun credentials (MAILGUN_ADMIN_API_KEY, MAILGUN_API_KEY, SMTP_PASSWORD, MAILGUN_AUTH_TOKEN) from Tier 3 → Tier 1 — /v1/keys and /v3/domains/{domain}/credentials cover both key types, and MAILGUN_AUTH_TOKEN was never a provider credential at all. OVH/SYS consumer keys stay Tier 3, but #5410 (merged same day) live-confirmed the main OVH realm supports a zero-browser OAuth2 migration path (OVH_INFRA_CLIENT_ID/SECRET, pending first-run mint) while the SoYouStart/Kimsufi realm does not — SYS_* is permanently Tier 3 until OVH ships IAM there. Sentry and PM2 Plus were re-verified and confirmed to have no full automation path — they stay Tier 3, now with citations.
Auto-rotation schedule: credential-rotation.yml runs every Monday 06:00 UTC. Force all: gh workflow run credential-rotation.yml --repo radieu/p24-infra -f force_all=true -f dry_run=false
Decision flow — which tier to use
New rotation triggered
│
├─ Is secret in Tier 1 (✅)?
│ └─ YES → Run rotate-credentials.py (GH Actions) or per-key autonomous script
│ No human notification needed
│
├─ Is secret in Tier 2 (⚠️)?
│ ├─ Check prerequisite (TOTP secret? Playwright session? Admin key valid?)
│ │ ├─ PREREQUISITE MET → proceed as if Tier 1 (Playwright script or admin API)
│ │ └─ PREREQUISITE NOT MET → escalate to Tier 3 for that key
│ └─ Document which prereq was missing
│
└─ Secret is Tier 3 (❌)
├─ Create GH issue: label "human-action" (+ "critical" if exposure)
│ Title: "sec: rotate KEY_NAME — [exposure|scheduled] YYYY-MM-DD"
├─ Human rotates at provider dashboard
├─ Human stores new value: $env:NEW_VALUE = "paste-here" (never paste in chat)
└─ Claude reads $env:NEW_VALUE → updates SOPS → PR → distributes → rotates old key
Rotation-issue label routing (which label means what)
Every rotation issue filed by automation or a worker carries a label that encodes its
severity and de-duplication thread. scripts/rotate-credentials.py:find_open_issue()
de-duplicates against the first label, so each class stays on its own thread.
| Label | Filed by | Meaning | Severity |
|---|---|---|---|
human-action (+ critical / patch) | Tier-3 manual routing; auto-rotation failure; ADR-004 fail-closed lock-unverified; ADR-004 lock-lost mid-rotation abort | A human must act — rotate manually, or reconcile a partial/failed rotation | high (critical/high → critical; medium → patch) |
rotation-deferred | rotate-credentials.py scheduled auto_due loop only (ADR 004 / #5988) | The scheduled rotation was skipped this cycle because another session already holds the per-secret advisory lock (clean 409). The mutual-exclusion mechanism working as intended — not a failure. Retries automatically next run. | low — audit/observe only, no action unless it keeps deferring week after week |
Rules:
- Never hand-file a
rotation-deferredissue — it is emitted only by the automated cron’s lock-degradation policy (outcome 2 of ADR 004 §Layer 1). Arotation-deferredissue that never clears across several weekly runs means the other holder is stuck — investigate the concurrent rotation of that secret, do not just close it. lock-unverified(Supabase unreachable past the retry window → fail-closed) andlock-lost(holds_lock failed before a live-write step) are genuine failures at the same severity as any other rotation failure — they reuse thehuman-actionlabel deliberately, so they surface in the same actionable queue.
Audit trail — after any rotation
Desktop Claude sessions:
| YYYY-MM-DD HH:MM UTC | #ISSUE | KEY_NAME | reason: exposure/scheduled/preventive | rotator | confirmed in: SOPS / GH Secret / vps live |Append to docs/secrets-rotation-log.md.
VPS worker agents: Write to dev_r_rotation_log Supabase table. Desktop sessions summarise when confirmed.
Related playbooks
| Topic | Playbook |
|---|---|
| Full rotation policy + SLAs | docs/playbooks/credential-rotation-policy.md |
| Auto-rotated keys (180d/90d cycle) | docs/playbooks/credential-rotation-180d.md |
| Per-key manual steps | docs/playbooks/credential-rotation-manual-static.md |
| Exposure incident response | docs/playbooks/static-api-key-incident-rotation.md |
| SOPS file editing (Windows) | docs/playbooks/sops-edit-operations.md, docs/playbooks/sops-windows-crlf.md |
| GitHub App tokens in n8n | docs/playbooks/n8n/github-app-token-n8n.md |
| GitHub PAT rotation | docs/playbooks/github-pat-rotation.md |
| Wasabi IAM key rotation | docs/playbooks/wasabi-key-rotation.md |
| MongoDB credential rotation | docs/playbooks/mongodb-credential-rotation.md |
| Pinbox24 w3/w4 app MongoDB rotation (recreate all bms-1 consumers) | docs/playbooks/w3-mongodb-credential-rotation.md, docs/playbooks/w4-mongodb-credential-rotation.md |
| Silent stale-credential hang (why recreate ≠ restart) | docs/playbooks/mailgun-mongodb-stale-credential-hang.md |
| Master age key bootstrap | docs/playbooks/master-keys-bootstrap.md |
| Detecting SOPS ↔ GH Secret drift | docs/playbooks/gh-secret-sops-drift.md |