Secret Manager — Master Playbook

Status: Single source of truth for all p24-infra secret operations.
Role: Spawn a /role-secret-manager session for any secret add, rotate, retrieve, or distribute task.
Last updated: 2026-07-01


What the secret-manager session does

The secret-manager is the only session that touches SOPS files, GH Secrets, or credential values. No other session (dev, sys-admin, content, etc.) is authorised to edit secrets/*.env.sops.

Scope of the role:

  • Add / update / rotate secrets in SOPS
  • Distribute new values: SOPS → GH Secrets → live servers → Vercel → n8n
  • Check rotation tiers and create human-action issues when a human must act
  • Log every rotation to Supabase dev_r_rotation_log via scripts/rotation-log-entry.py (the single log; auto-falls-back to docs/rotation-log-fallback.md only when the API is unreachable — never hand-edit either markdown file)
  • Respond to exposure incidents (1-hour SLA)

When to spawn a secret-manager session

Any session (dev, infra-task, brandpilot, et-op, etc.) that needs to:

  • Add a new API key / token / password to the ecosystem
  • Rotate an existing credential (scheduled, exposure, or preventive)
  • Retrieve which SOPS file holds a specific key
  • Distribute a value to servers, Vercel, or n8n after a human has rotated it
  • Audit what credentials exist and their rotation status

Pattern: The requesting session creates a GH issue describing the operation, then either: a) Spawns a /role-secret-manager inline session, or b) Opens a new Claude Code session in C:\code_2026\p24-infra and runs /role-secret-manager

The secret-manager session handles everything from that point — the requesting session does NOT touch credentials.


Startup sequence (secret-manager session)

  1. Acquire env lock — MANDATORY before any SOPS write (protects against concurrent sessions):
    . C:\code_2026\p24-infra\scripts\env-lock.ps1
    Acquire-EnvLock -Resource "sops:<target-file-stem>" -Operation "secret-manager-<task-description>"
    # Example: Acquire-EnvLock -Resource "sops:monitoring" -Operation "credential-rotation-grafana"
    If blocked: another session holds the lock — see docs/playbooks/env-locking.md §What to do when a lock is held.

0b. Linux workers — pre-flight concurrent-write guard, MANDATORY before any raw SOPS write (env lock above doesn’t cover Linux worker checkouts, #5242): bash scripts/check-worktree-write-safety.sh secrets/<file>.env.sops || exit 1 See docs/playbooks/concurrent-sops-write-worktree-race.md (related 5225). Full procedure: infra/agent-prompts/worker-secret-manager.md §General pattern.

  1. Verify context: git remote -v must point to radieu/p24-infra
  2. Verify SOPS key: $env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
  3. Canary: sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null — must exit 0
  4. Read docs/priorities.md for active P0/P1 items
  5. If rotation task: read docs/playbooks/secret-rotation-access-matrix.md to determine tier

At session end — after canary decrypt confirms SOPS file intact:

sops --decrypt --input-type dotenv --output-type dotenv secrets\<file>.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — investigate before releasing lock" }
Release-EnvLock -Resource "sops:<target-file-stem>"

Do NOT load everything at session start. Read only what the current task requires.

I need to…Load this
Know if I can rotate autonomouslysecret-rotation-access-matrix.md — Tier check FIRST
Respond to suspected exposurestatic-api-key-incident-rotation.md — START THERE, skip everything else
Edit a SOPS fileAcquire lock first (env-locking.md) → sops-edit-operations.mdsops-windows-crlf.md only on errors
Acquire/release env lock before SOPS writeenv-locking.md
Rotate a specific credentialAccess matrix → find key → follow “playbook” column
Know the SLA / rotation policycredential-rotation-policy.md
Decide which SOPS file a new credential belongs indocs/policies/credential-isolation-policy.md — MANDATORY before adding any key
Add a brand new credential§Adding new credential to the ecosystem below
Know which SOPS file holds KEY_NAME§SOPS file map below
Verify distribution after rotation§Distribution chain below
Scheduled / 180-day rotationcredential-rotation-180d.md
Manual (UI-only) rotation procedurescredential-rotation-manual-static.md
Log a rotation (Supabase dev_r_rotation_log)§Rotation log entry (mandatory) below
Create fresh SOPS file / bootstrap new envsops-environment-bootstrap.md

SOPS file map — where every secret lives

KeysSOPS fileWho can decryptDeployed to
Monitoring, Grafana, Wasabi p24-infra, CF tokens, SMTP, Traccar, PDF service, Supabase Grafanasecrets/monitoring.env.sopsdev + CI + runnersvps-i1 /opt/p24-infra/monitoring/.env
n8n, Redis, MongoDB prometheus user, GitHub App, Discord, Telegram, RabbitMQ, n8n DBsecrets/n8n-bms4.env.sopsdev + CI + runnersbms-4 /opt/p24-infra/bms-4/.env
n8n GitHub PAT (GH_TOKEN, GITHUB_PAT_ALL_WRITES, GITHUB_PAT_P24_INFRA) — migrated from n8n-bms4secrets/n8n-bms4-gh.env.sopsdev + CI + runnersbms-4 /opt/p24-infra/bms-4/n8n-bms4-gh.env
Traefik, SSL, vps-h1 services (WAHA decommissioned 2026-07-02)secrets/vps-h1.env.sopsdev + CI + runnersvps-h1 /root/.env
BMS root passwordssecrets/bms-servers.env.sopsdev + CI + runnersNot deployed — operator decrypt on demand
MongoDB rs0 admin + prometheus credentials (feat/#2728)secrets/mongodb-bms.env.sopsdev + CI + runnersbms-4 /opt/p24-infra/bms-4/mongodb-bms.env via secrets-sync.yml
Art Agency (Supabase, PayPal, KDP, Drive, ClickUp)secrets/art-agency.env.sopsdev + CI + runnersArt-Agency/.env.local (manual sync)
BrandPilot (Resend, Wasabi, Anthropic, Supabase)secrets/brandpilot.env.sopsdev + CI + runnersVercel prj_brandpilot via secrets-sync.yml
et-operational-platform (Supabase, n8n, GPS, PINBOX24, Atrax, etc.)secrets/et-operational-platform.env.sopsdev + CI + runnersVercel prj_ziLl911FOYLAeukQujL4NjxR4eWy via secrets-sync.yml
p24-wa-a / whatsup (Google OAuth, Supabase JWT, Wasabi IAM, WEBHOOK_SECRET)secrets/whatsup.env.sopsdev + CI + runnersApp env (manual sync)
et-lager warehouse (MONGODB_URI)secrets/et-lager.env.sopsdev + CI + runnersVercel et-lager project via secrets-sync.yml sync-et-lager
Cloudflare API tokens (DNS + Workers deploy) — split from monitoring.env.sops (#3178 Track B, PR-E, 2026-08-02)secrets/cloudflare.env.sopsdev + CI + runners (6-recipient rule; no narrowing yet)NOT YET DEPLOYED — no secrets-sync.yml job exists (path excluded from auto-trigger); values are duplicates of the still-live copies in monitoring.env.sops until PR-D/PR-G land
Traccar + MySQL credentials — split from monitoring.env.sops (#3178 Track B, PR-E, 2026-08-02)secrets/traccar.env.sopsdev + CI + runnersNOT YET DEPLOYED (same caveats). docs/sops-templates/traccar.keys notes a pre-existing gap: MYSQL_ROOT_PASSWORD is currently empty in monitoring.env.sops
Worker queue credentials (Supabase queue read/write, GH_TOKEN, Discord error webhooks, Mezmo) — split from monitoring.env.sops (#3178 Track B, PR-E, 2026-08-02); VPS_SSH_PRIVATE_KEY deliberately deferred, see file headersecrets/worker-queue.env.sopsdev + CI + runnersNOT YET DEPLOYED (same caveats). Intended targets: vps-h1/bms-4/vps-i1/dev-laptop worker .env files, see docs/plans/monitoring-env-sops-reorganisation.md §1
Pinbox24 W3 backend credentials (V32_* — ~34 keys: MongoDB, JWT, S3/Wasabi, OneSignal, Twilio, jabber, mailgun)secrets/pinbox24-w3.env.sopsdev + CIbms-1 v32-prod + s3-v32-prod via secrets-sync.yml sync-pinbox24-w3
Pinbox24 W4 backend credentials (V42_* — ~65 keys: MongoDB, JWT, Redis, S3/Wasabi, PM2, mailgun, PayU)secrets/pinbox24-w4.env.sopsdev + CIbms-1 v42-prod + s3-v42-prod + s3-v2-v42-prod via secrets-sync.yml sync-pinbox24-w4
Pinbox24 mailgun-prod service (4 keys: MAILGUN_MONGODB_URL, MAILGUN_AUTH_TOKEN, MAILGUN_S3_SERVER_URL, MAILGUN_S3_V2_SERVER_URL)secrets/pinbox24-backends.env.sopsdev + CIbms-1 mailgun-prod container via secrets-sync.yml sync-pinbox24-backends
GCP service account JSON (GCP_SA_KEY_JSON)secrets/gcp.env.sopsdev + CI + runnersServices using GCP APIs
Rotation script transient credentials (AGE_KEY_SOPS_SYNC_RECEIVER)secrets/rotate-secret.env.sopsdev onlyInjected by rotate scripts at runtime
IONOS API, Wasabi admin IAM, CF Global Key, GH_PAT_ADMIN, GITLAB_ADMIN_PAT (canonical copy), GCP SA key, MongoDB rs0 adminsecrets/administration.env.sopsdeveloper local ONLY (no CI, no runners)Not deployed — operator decrypt on demand
GITLAB_ADMIN_PAT — narrow copy, 2 recipients only (#4796)secrets/pinbox24-gitlab.env.sopsdeveloper + bms-4 infra-task worker key ONLY (custom exact-anchored .sops.yaml rule — not CI, not vps-i1/vps-h1/dev-laptop)Not deployed by secrets-sync.yml (not a listed target job) — delivered to bms-4 for free by the existing git pull origin main inside sync-bms-4 (runs on every push touching any non-excluded secrets/*.env.sops, including this one); worker decrypts on demand from /opt/p24-infra/secrets/pinbox24-gitlab.env.sops. Same token value as administration.env.sops — rotate both atomically.
ANTHROPIC_ADMIN_API_KEY — narrow copy, 2 recipients only, isolated from monitoring.env.sops’s env_file: .env blast radius into alertmanager/mezmo-exporter/caddy (#5697/#5685)secrets/anthropic-admin.env.sopsdev + CI (AGE_KEY_GHA) ONLY — same custom exact-anchored .sops.yaml rule pattern as pinbox24-gitlab.env.sopsvps-i1 /opt/p24-infra/monitoring/.env.anthropic-admin (0600 root:root) via secrets-sync.yml’s dedicated sync-anthropic-admin job — recreates cost-exporter only, never the full monitoring stack. See anthropic-admin-api-key.md.
SOPS health canary — test keys only (SOPS_CANARY, SOPS_CANARY_TIMESTAMP)secrets/.canary.env.sopsdev onlyscripts/remediation/test-sops-canary.sh

Lookup order for any secret: SOPS file → on-server .env.bak (may be stale) → .env.local (emergency only)

Role / worker credentials

Each Claude worker role gets its own SOPS file with scoped credentials — loaded by /role-<name> skill at session start.

RoleSOPS fileKeys
sys-adminsecrets/role-sys-admin.env.sopsROLE_SYS_ADMIN_GITHUB_CLASSIC_PAT, ROLE_GIT_SIGNING_KEY
secret-managersecrets/role-secret-manager.env.sopsROLE_SECRET_MANAGER_GITHUB_PAT, ROLE_SECRET_MANAGER_VERCEL_TOKEN, ROLE_GIT_SIGNING_KEY, ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN, MAILGUN_ADMIN_API_KEY, OPENAI_ADMIN_KEY, SENTRY_AUTH_TOKEN
dev-codersecrets/role-dev-coder.env.sopsROLE_DEV_CODER_GITHUB_PAT, ROLE_GIT_SIGNING_KEY
dev-reviewersecrets/role-dev-reviewer.env.sopsROLE_DEV_REVIEWER_GITHUB_PAT, ROLE_GIT_SIGNING_KEY
dev-testersecrets/role-dev-tester.env.sopsROLE_DEV_TESTER_GITHUB_PAT, ROLE_GIT_SIGNING_KEY

Provider master-key coverage (which master key rotates which provider)

Three provider management/admin keys let the secret-manager mint or rotate downstream keys via API (verified #4400, 2026-07-30 — all authenticate 200). Reachability matters: a standard bms-4 secret-manager worker (age key ~/.age/p24-infra-keys.txt) can decrypt monitoring, brandpilot, whatsup — but not administration or role-secret-manager.

ProviderMaster keyMints/rotatesWorker-reachable copyWorker-reachable?
MailgunMAILGUN_ADMIN_API_KEY (/v1/keys API)Mailgun sending + admin keysmonitoring.env.sops
SupabaseSUPABASE_MANAGEMENT_TOKENROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN (Management API)SUPABASE_SERVICE_ROLE_KEY, anon key, DB password, project keysbrandpilot.env.sops
WasabiWASABI_ADMIN_ACCESS_KEY / _SECRET_KEY (IAM API)all p24 Wasabi IAM keysvalid copy only in role-secret-manager / administration — the whatsup copy is stale (#3366)⚠️ no — sync valid key to a worker-readable file first

Full classification + RUN-NOW/FRIDAY split: secret-rotation-access-matrix.md §Master-key deliverability re-check.


Windows SOPS operations — safe patterns

Canonical reference: sops-windows-patterns.md — single source of truth for write safety and write method comparison. Patterns below are preserved for operational self-containment.

Canary (always run before git add)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\<file>.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }

Preferred: sops-set.ps1 (canonical tool)

scripts/sops-set.ps1 (issue #3269) wraps the exact decrypt → edit → encrypt → canary → atomic-move sequence documented in the fallback patterns below via scripts/lib/sops-common.psm1’s Update-SopsKey / Update-SopsKeys. Use it for every add/rotate instead of hand-running the raw pattern — it is unit-tested (Pester, scripts/tests-ps1/sops-common.Tests.ps1) and never echoes a secret value.

This is not optional guidance — a .claude/hooks/pre-bash-safety* guard (issue #3545) blocks a direct sops --encrypt ... .env.sops call that doesn’t go through sops-set.ps1 (or Update-SopsKey/Update-SopsKeys/Write-SopsFileContent directly). Hand-rolled encrypt round trips skip the tested safety net and are exactly how #3275’s recipient-widening bug got introduced in the first place.

Single key (value via -Value or $env:NEW_VALUE):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:NEW_VALUE = "paste-here"   # human sets this, never in chat
.\scripts\sops-set.ps1 -SopsFile secrets\<file>.env.sops -Key KEY_NAME
$env:NEW_VALUE = ""

Batch — multiple keys in one file, one decrypt/encrypt round trip:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
.\scripts\sops-set.ps1 -SopsFile secrets\<file>.env.sops -Pairs '{"KEY_A":"'+$env:VALUE_A+'","KEY_B":"'+$env:VALUE_B+'"}'

Preview without writing:

.\scripts\sops-set.ps1 -SopsFile secrets\<file>.env.sops -Key KEY_NAME -Value x -DryRun

Run from the repo root with a relative -SopsFile path (e.g. secrets\<file>.env.sops) — Update-SopsKey/Update-SopsKeys build their temp-file paths by joining the current location with $SopsFile, so an absolute path will fail. Exits non-zero on any failure; only key names and OK/FAIL are ever printed, never values.

secrets/administration.env.sops is now safe to write via sops-set.ps1 (fixed by issue #3275 / PR referencing it). Update-SopsKey/Update-SopsKeys detect exact-filename -anchored .sops.yaml rules (Test-SopsSingleRecipientRule in sops-common.psm1) — e.g. path_regex: (^|[/\\])secrets[/\\]administration\.env\.sops$, restricted to a single developer-only age recipient — and route those writes through an in-place-safe path instead of the generic *-edit-tmp.env.sops / *-enc-tmp.env.sops temp-file naming. The plaintext temp file is staged under a throwaway directory OUTSIDE secrets/ whose path still ends in secrets/administration.env.sops (sops resolves creation_rules from the positional input path passed to --encrypt, not from --output — verified empirically while building the fix), so it correctly resolves to the single-recipient rule while plaintext never touches the real secrets/ directory. Every write (single- recipient or general) is now also verified post-write: the resulting file’s embedded sops_age__list_N__map_recipient= count is compared against the matched rule’s expected recipient count, and any mismatch triggers an automatic rollback to a pre-write backup plus a thrown error — the real file is never left in a widened or corrupted state. The manual fallback pattern below is kept for reference / emergencies (e.g. sops or age-keygen unavailable) but is no longer required for administration.env.sops.

Preferred: sops-invoke.ps1 (canonical read/USE path — #5298, ADR 003)

sops-set.ps1 is the enforced path to put a value into SOPS. Its counterpart scripts/sops-invoke.ps1 (issue #5298) is the preferred path to use a value from SOPS in a command — replacing the ad-hoc $v = (sops -d ... | Select-String '^KEY=')... ; cmd $v one-off that was behind every historical leak (#2040/#3545/#3714/#5223). The broker decrypts inside its own process, exposes each value only as an env var for the wrapped command’s lifetime, and redacts every fetched value (raw + URL-encoded + base64 + JSON-escaped) out of the command’s merged stdout/stderr before returning it. The value is never printed and never returned. See docs/adr/003-sops-secret-access-broker.md.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# The JSON maps ENV-VAR-NAME -> SOPS-KEY-NAME; the command references $env:GL_TOKEN.
.\scripts\sops-invoke.ps1 -SopsFile secrets\administration.env.sops `
  -Keys '{"GL_TOKEN":"GITLAB_ADMIN_PAT"}' `
  -Command 'curl -s -H "PRIVATE-TOKEN: $env:GL_TOKEN" https://gitlab.com/api/v4/user'

The broker also underpins the Playwright carve-out for UI-only credential flows (scripts/secret-broker/Invoke-PlaywrightWithSecret.ps1) — gitlab-token-playwright-broker.md.

Bash/Linux workers: a bash broker sibling is not yet built (ADR 003 §Known gap), so on Linux workers the documented safe-extraction pattern (VAR=$(sops -d … | grep '^KEY=' | cut -d= -f2-); cmd "$VAR"; unset VAR) remains the correct fallback until it lands.

Fallback — manual pattern (only if the script can’t be used at all)

Add a new key to an existing SOPS file

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$SOPS_FILE = "$PWD\secrets\<file>.env.sops"
$TEMP_PLAIN = "$PWD\secrets\<file>-edit-tmp.env.sops"   # must match path_regex
$TEMP_ENC   = "$PWD\secrets\<file>-enc-tmp.env.sops"    # encrypted staging file
 
# 1. Decrypt to variable (never print)
$plain = sops --decrypt --input-type dotenv --output-type dotenv $SOPS_FILE
if ($LASTEXITCODE -ne 0) { throw "Decrypt failed" }
 
# 2. Append new key in memory (value from $env:NEW_VALUE — never hardcoded or printed)
$content = ($plain -join "`n") + "`n" + "NEW_KEY=$env:NEW_VALUE" + "`n"
$plain = @()  # clear plaintext from memory
 
# 3. Write plaintext to a TEMP file — never to the production SOPS path
[System.IO.File]::WriteAllText($TEMP_PLAIN, $content, [System.Text.UTF8Encoding]::new($false))
$content = ""
 
# 4. Encrypt to a second temp — SOPS writes binary, no PowerShell string pipeline
sops --encrypt --input-type dotenv --output-type dotenv --output $TEMP_ENC $TEMP_PLAIN
if ($LASTEXITCODE -ne 0) {
  @($TEMP_PLAIN, $TEMP_ENC) | Where-Object { $_ -and (Test-Path $_) } | ForEach-Object { Remove-Item $_ -Force -ErrorAction SilentlyContinue }
  throw "Encrypt failed"
}
Remove-Item $TEMP_PLAIN -Force -ErrorAction SilentlyContinue  # plaintext gone
 
# 5. Canary on encrypted temp — production file untouched until this passes
sops --decrypt --input-type dotenv --output-type dotenv $TEMP_ENC | Out-Null
if ($LASTEXITCODE -ne 0) {
  Remove-Item $TEMP_ENC -Force -ErrorAction SilentlyContinue
  throw "SOPS corrupt on temp — NOT overwriting production file"
}
 
# 6. Atomic move — production file replaced only after canary passes
Move-Item $TEMP_ENC $SOPS_FILE -Force
 
# 7. Clear sensitive vars
$env:NEW_VALUE = ""
Write-Host "Done"

Update an existing key value

Same pattern as above, but step 2 uses -replace instead of append:

$content = ($plain -join "`n") + "`n"
$content = $content -replace "(?m)^KEY_NAME=.*$", "KEY_NAME=$env:NEW_VALUE"

Check if a key exists / verify its identity (MANDATORY — never hand-roll this)

Never write (... | Where-Object {...}) -ne $null or -eq $null against decrypted SOPS output. When the left side of -ne/-eq is a collection (which Where-Object always returns), PowerShell applies the operator elementwise and returns the matching elements themselves — the raw KEY=value secret line — not a boolean. This is exactly how WASABI_ADMIN_ACCESS_KEY/WASABI_ADMIN_SECRET_KEY were exposed in full on 2026-07-09 (issue #3545): an agent’s presence-check one-liner echoed the real values to tool output. A .claude/hooks/pre-bash-safety* guard now blocks this pattern outright — use the safe, tested helpers from scripts/lib/sops-common.psm1 instead:

Import-Module .\scripts\lib\sops-common.psm1 -Force
 
# Boolean presence only:
Test-SopsKeyExists -SopsFile secrets\<file>.env.sops -KeyName KEY_NAME
# -> $true / $false, never prints the value
 
# Presence + length + a one-way SHA-256 fingerprint (first 8 hex chars) — useful for
# "is this the same value as last time" / "did this differ across two SOPS files"
# (e.g. the #3426 MongoDB-admin-credential-drift investigation) WITHOUT ever exposing
# a character of the actual value, not even a truncated fragment (a `head -c N` on a
# credential file was separately blocked this same session for exactly that reason —
# partial output still counts as exposure):
Get-SopsKeyFingerprint -SopsFile secrets\<file>.env.sops -KeyName KEY_NAME
# -> @{ Exists = $true; Length = 42; Fingerprint = "a3f9c1e2" }

List key names in a SOPS file (safe — no values)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\<file>.env.sops |
  ForEach-Object { $_.Split("=")[0] }

Read a single key value (for use in a command — never print)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:THE_SECRET = (sops --decrypt --input-type dotenv --output-type dotenv secrets\<file>.env.sops |
  Where-Object { $_ -match "^KEY_NAME=" }).Split("=",2)[1]
some-command --token $env:THE_SECRET
$env:THE_SECRET = ""

NEVER use >, Out-File, or Set-Content for SOPS files — they add CRLF + BOM. Always use [System.IO.File]::WriteAllText($path, $content, [System.Text.UTF8Encoding]::new($false)).

administration.env.sops is developer-only. Its .sops.yaml path_regex restricts it to one recipient. The in-place encrypt pattern works correctly because the file path matches the first creation_rule. Do NOT use a temp file with a different name in secrets/ for administration — it would match the second rule (5 recipients) and encrypt with CI + runner keys. Always encrypt administration in-place.


Pitfall — Select-String against a hand-off file can silently return MULTIPLE matches (#5835)

When extracting a value from a human hand-off file (.env.local) with Select-String -Pattern "^KEY=", always check .Count (or inspect .LineNumber) before trusting the result — never assume exactly one match. If the key appears on more than one line (a stale/corrupted leftover line plus a freshly-appended correct one, or any other duplicate), the returned object is an array of MatchInfo, and this is where it gets dangerous:

  • $line.Line on that array auto-enumerates the property, silently returning a string array, not a scalar.
  • A later -match / -replace / string-method call against that array can return wrong or leaked data instead of erroring — e.g. $array -match 'pattern' (elementwise, like the documented -ne $null bug in this same file’s “Check if a key exists” section) returns the full matching line(s) themselves, not a boolean. In #5835 this printed a corrupted hand-off line’s full content (including a real token) straight into an agent’s tool output.
  • Depending on which element “wins” implicitly, a naive .Split("=",2)[1] extraction can also pick up the wrong line’s value without any error at all — in #5835 this caused a rotated GH_TOKEN to be distributed to 3 SOPS files + a GH Secret using a corrupted/garbage value, which then broke every consumer (gh auth login --with-token → 401) until caught and corrected in a follow-up PR.

Safe pattern:

$matches = Select-String -Path $path -Pattern "^KEY_NAME="
if ($matches.Count -ne 1) {
  Write-Host "Expected exactly 1 match for KEY_NAME, found $($matches.Count) -- inspect LineNumbers before proceeding"
  $matches | ForEach-Object { Write-Host "LineNumber=$($_.LineNumber) LineLength=$($_.Line.Length)" }  # metadata only, never content
  # Resolve manually: pick the correct LineNumber explicitly, e.g.
  # $chosen = ($matches | Where-Object { $_.LineNumber -eq 39 }).Line
  exit 1
}
$env:NEW_VALUE = $matches.Line.Split("=",2)[1]

Ground-truth verify any extracted credential (HTTP status / auth check only, never printed) before writing it to SOPS or a GH Secret — this is what caught the #5835 mis-extraction on the very next distribution step, before the wrong value could do more damage than one failed secrets-sync.yml job.

How to receive a new credential from a human

The human should NOT paste the value into chat. Instead:

# Human runs this in their terminal — value never goes to chat
$env:NEW_VALUE = "paste-here"

Then the secret-manager session uses $env:NEW_VALUE in the SOPS add/update pattern above. After the SOPS operation: $env:NEW_VALUE = "" to clear.


Distribution chain — after any SOPS update

Every changed credential must propagate to ALL applicable destinations before the old value is revoked:

  1. SOPS commitgit add secrets/<file>.env.sopsgit commit → PR to main
  2. GH Secret (if used by CI/CD):
    $env:VAL = (sops ... | Where-Object { $_ -match "^KEY=" }).Split("=",2)[1]
    gh secret set KEY_NAME --body "$env:VAL" --repo radieu/p24-infra
    $env:VAL = ""
  3. secrets-sync.yml fires on merge to dev/main → deploys .env to vps-i1 and bms-4
  4. Vercel — if used by et-operational-platform, brandpilot, or any Vercel project:
    vercel env add KEY_NAME production --yes  # prompts for value securely
  5. n8n credential vault — if an n8n workflow reads this key directly (not via env): POST /api/v1/credentials on bms-4 n8n via REST
  6. Container restart on affected server(s) after .env is deployed:
    ssh root@<ip> "cd /opt/p24-infra/... && docker compose up -d --no-deps <service>"
  7. Revoke old value at provider — only after all destinations confirmed live

Rotation tier decision tree

Before starting any rotation, read docs/playbooks/secret-rotation-access-matrix.md:

New rotation triggered
│
├─ Tier 1 (✅ fully autonomous)?
│   └─ YES → run `scripts/rotate-credentials.py` or per-key API script
│
├─ Tier 2 (⚠️ semi-autonomous)?
│   ├─ Prerequisite met (TOTP? Playwright session? Admin key valid)?
│   │   ├─ YES → proceed as Tier 1 via Playwright or admin API
│   │   └─ NO  → escalate to Tier 3 for this key
│
└─ Tier 3 (❌ manual only)?
    ├─ Create GH issue: label "human-action" (+ "critical" if exposure)
    │   Title: "sec: rotate KEY_NAME — [exposure|scheduled] YYYY-MM-DD"
    ├─ Human rotates at provider dashboard → sets $env:NEW_VALUE in terminal
    └─ secret-manager session: SOPS update → distribution → old key revoked

Exposure SLA: 1 hour. Any credential value visible in chat, logs, or git must be rotated within 1 hour.


Adding a new credential to the ecosystem

  1. Choose the SOPS file based on the scope (see file map above)
  2. Check rotation tier — add to docs/playbooks/secret-rotation-access-matrix.md
  3. Add to SOPS using scripts/sops-set.ps1 (§“Preferred: sops-set.ps1 (canonical tool)” above) — safe for all SOPS files including administration.env.sops (issue #3275)
  4. Add to GH Secrets if used in CI/CD
  5. Reference in config (docker-compose.yml or app config) via env var — never hardcode
  6. Register in Supabase dev_r_services table (compliance_workbook='yes')
  7. Add rotation schedule to dev_r_services.next_due and rotation_type
  8. Log the bootstrap to Supabase dev_r_rotation_log via scripts/rotation-log-entry.py (never hand-edit any markdown log file):
    ROW_ID=$(python3 scripts/rotation-log-entry.py open --secret KEY_NAME \
        --reason "bootstrap — new credential" --reason-category manual --rotator secret-manager)
    python3 scripts/rotation-log-entry.py close "$ROW_ID" --sops-commit "$(git rev-parse HEAD)"
    Writes to dev_r_rotation_log; if the API is unreachable the script auto-falls-back to docs/rotation-log-fallback.md (returning a fallback:<uuid> id), drained back on the next successful API call.

Bootstrap & Audit — kompletny zestaw narzędzi (2026-07-05)

Nowe narzędzia do tworzenia środowisk SOPS od zera i weryfikacji kompletności.

Szybki start — sprawdź co brakuje w SOPS

# Sprawdź WSZYSTKIE środowiska (monitoring, n8n-bms4, et-lager, et-op, brandpilot, art-agency)
.\scripts\bootstrap\validate-sops-env.ps1 -Env all
 
# Lub jedno środowisko
.\scripts\bootstrap\validate-sops-env.ps1 -Env monitoring

Wynik: lista brakujących kluczy z [Tier1]/[Tier2]/[Tier3] i notką. Tier 3 = wymaga human-action.

Uzupełnij brakujące klucze

# 1. Ustaw Tier 3 klucze ręcznie (wartości TYLKO w terminalu — nie w chacie):
$env:VALUE_SUPABASE_SERVICE_ROLE_KEY = "sb_secret_..."
$env:VALUE_GH_TOKEN = "ghp_..."
# itd. — pełna lista P0/P1/P2 w docs/playbooks/sops-audit-report.md
 
# 2. Uruchom bootstrap tylko dla brakujących kluczy:
.\scripts\bootstrap\new-sops-env.ps1 -Env monitoring -OnlyMissing
 
# 3. Tylko Tier 1 (autonomous) — bez interakcji:
.\scripts\bootstrap\new-sops-env.ps1 -Env monitoring -OnlyMissing -TierFilter 1
 
# 4. Dry run — podgląd bez zmian:
.\scripts\bootstrap\new-sops-env.ps1 -Env monitoring -DryRun

Tier-routing w bootstrap script

TierCo robi script
1Generuje autonomicznie (random hex, bcrypt, API call)
2Drukuje instrukcję jak uruchomić skrypt Playwright; jeśli $env:VALUE_KEY ustawiony — użyje go
3Czeka na $env:VALUE_KEY ustawiony przez człowieka; bez niego pomija
0Pomija (nie-secret, wartość statyczna)

Szablony kluczy (machine-readable)

docs/sops-templates/<env>.keys — format: KEY_NAME|tier|provider|rotation_script|notes

PlikŚrodowiskoStatus
monitoring.keysmonitoring.env.sops✅ kompletny (~90 kluczy)
n8n-bms4.keysn8n-bms4.env.sops✅ kompletny (~60 kluczy)
et-operational-platform.keyset-operational-platform.env.sops✅ kompletny (27 kluczy)
brandpilot.keysbrandpilot.env.sops✅ kompletny (13 kluczy)
art-agency.keysart-agency.env.sops✅ kompletny (26 kluczy)
et-lager.keyset-lager.env.sops⚠️ TODO — uzupełnij z et-lager repo .env.example

Aby dodać nowe środowisko do walidatora: stwórz docs/sops-templates/<env>.keys według wzoru z istniejących plików.

Audit — aktualny stan SOPS

docs/playbooks/sops-audit-report.md — snapshot z 2026-07-05: lista P0/P1/P2 broken/stale kluczy, dostępne admin keys umożliwiające autonomiczną rotację, skonsolidowane $env:VALUE_* do ustawienia.

Regeneruj po każdej sesji rotacji: uruchom validate-sops-env.ps1 -Env all i zaktualizuj plik.

Kiedy używać bootstrap vs. standardowej rotacji

SytuacjaNarzędzie
Sprawdź co brakuje w istniejącym plikuvalidate-sops-env.ps1
Uzupełnij brakujące klucze (patch)new-sops-env.ps1 -OnlyMissing
Nowe środowisko od zeranew-sops-env.ps1 (bez -OnlyMissing)
Jeden klucz — update wartościscripts\sops-set.ps1 -Key ... -Value ... (§Preferred: sops-set.ps1 above)
Rotacja autonomiczna Tier 1/2credential-rotation.yml (cron)
Rotacja po ekspozycjistatic-api-key-incident-rotation.md

Rotation log entry (mandatory)

Supabase dev_r_rotation_log is the single rotation log for everyone (desktop sessions and workers alike). Every rotation — successful, failed, or escalated — must be recorded there via scripts/rotation-log-entry.py, never by hand-editing a markdown file. Open the pending entry before touching the key so a traceable record survives even an interrupted session:

# 1. Open (prints the row id — a uuid, or "fallback:<uuid>" if the API was down):
ROW_ID=$(python3 scripts/rotation-log-entry.py open --secret KEY_NAME \
    --reason "<free text / #issue>" \
    [--reason-category scheduled|exposure|preventive|manual] [--rotator secret-manager])
 
# 2. Close as completed after distribution:
python3 scripts/rotation-log-entry.py close "$ROW_ID" \
    --sops-commit "$(git rev-parse HEAD)" [--gh-secret-name KEY_NAME] --verify-result "<outcome>"
 
# or mark it failed:
python3 scripts/rotation-log-entry.py fail "$ROW_ID" --error "<what went wrong>"

reason is now free text; the optional reason_category (scheduled/exposure/preventive/manual) is a coarse label used for reporting only. Status values: pending / completed / failed / human-action.

Live-write rotations use acquire/check, NOT plain open (MANDATORY — ADR 004)

open above is fail-open logging. It does not stop a second session rotating the same secret at the same time. For a live-write rotation — one that pairs a secrets/*.env.sops write with a non-idempotent live server-side write (MongoDB changeUserPassword/updateUser, a Wasabi/S3 IAM key mint, any provider “regenerate” that mints a value in place) — you MUST hold the fail-closed per-(repo, secret) advisory lock instead. Two sessions each mint a different value and silently diverge (live server vs SOPS/Vercel) — the #5925 race ADR 004 exists to stop. Replace open with acquire as the first step (before touching the credential), and re-check with check immediately before the live write:

# 1. Acquire — this REPLACES `open` (it opens the pending row itself). Fail-CLOSED exit codes:
#    0 = acquired ($LOCK_ID is the pending row id) · 3 = denied (another session holds it — STOP,
#    reconcile, do NOT blind-overwrite) · 4 = could-not-verify Supabase (fail-closed, do NOT proceed).
LOCK_ID=$(python3 scripts/rotation-log-entry.py acquire --secret KEY_NAME \
    --repo p24-infra --reason "<free text / #issue>" --rotation-type manual) \
    || { echo "acquire denied/unverified — STOP, do NOT rotate"; exit 1; }
 
# 2. Layer-2 re-check immediately before the non-idempotent live write — lock still held?
python3 scripts/rotation-log-entry.py check "$LOCK_ID" \
    || { echo "lock lost — abort the live write and reconcile"; exit 1; }
 
# 3. Release with the SAME close/fail verbs on $LOCK_ID after distribution (there is no separate
#    release verb — closing/failing the pending row frees the lock for the next rotation).
python3 scripts/rotation-log-entry.py close "$LOCK_ID" --sops-commit "$(git rev-parse HEAD)"
  • --repo is the short name the lock is keyed on (p24-infra, or the requester’s repo such as et-operational-platform for a cross-repo secret like PINBOX24_MONGODB_URI) — matched exactly.
  • SOPS-only edits keep using plain open. Adding a key, editing a non-live config value, a distribution-only sync, or a gap-audit is git-mergeable and does NOT acquire the lock (ADR 004 scope). The lock is as narrow as the race: same (repo, secret), live server-side write only.
  • On the Windows dev machine use python and test $LASTEXITCODE. Full CLI reference (all verbs, exit codes, fallback semantics): secret-manager-rotation-log.md.

Offline fallback: if the Supabase API is unreachable, the script writes the entry to docs/rotation-log-fallback.md instead and returns a fallback:<uuid> id that close/fail correlate against the same file entry. On the next successful API call the pending fallback entries are drained automatically (a daily reimport-check cron is the backstop). If your fallback write must travel in a commit to reach the cron, stage it with the rotation’s SOPS commit — see docs/playbooks/secret-manager-rotation-log.md. Never hand-edit docs/rotation-log-fallback.md.


Rotation automation

Auto-rotation runs Monday 06:00 UTC via credential-rotation.yml:

gh workflow run credential-rotation.yml --repo radieu/p24-infra
# Force all + live run:
gh workflow run credential-rotation.yml --repo radieu/p24-infra -f force_all=true -f dry_run=false

Safe-handling rules (absolute — no exceptions)

  • Never display a secret value in chat. Reference KEY_NAME only.
  • Never hardcode values in docker-compose, scripts, or n8n node parameters.
  • Never use >, Out-File, Set-Content for SOPS files — CRLF/BOM corruption.
  • Never commit plaintext .env — only encrypted secrets/*.env.sops in git.
  • Never use sops --set for dotenv files — it does not work (JSON path only).
  • If a secret value appears in chat: stop immediately, reference KEY NAME only, rotate within 1h. Follow docs/playbooks/static-api-key-incident-rotation.md.

TopicPlaybook
Access tier for every credentialdocs/playbooks/secret-rotation-access-matrix.md
Rotation SLAs + GH issue formatdocs/playbooks/credential-rotation-policy.md
Auto-rotated keys (scheduled cycle)docs/playbooks/credential-rotation-180d.md
Playwright browser automation patterndocs/playbooks/playwright-rotation-template.md
SOPS CRLF/BOM recoverydocs/playbooks/sops-windows-crlf.md
Exposure incident responsedocs/playbooks/static-api-key-incident-rotation.md
Master age key bootstrapdocs/playbooks/master-keys-bootstrap.md
GitHub PAT rotationdocs/playbooks/github-pat-rotation.md
Wasabi IAM key rotationdocs/playbooks/wasabi-key-rotation.md
MongoDB credential rotationdocs/playbooks/mongodb-credential-rotation.md
GH Secret ↔ SOPS drift detectiondocs/playbooks/gh-secret-sops-drift.md
Per-key rotation detailsdocs/playbooks/credential-rotation-180d.md
Bootstrap fresh SOPS env / fresh keys workflowdocs/playbooks/sops-environment-bootstrap.md

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="<resource>",
    result="success",  # "success" | "failed" | "skipped"
    detail="Secret manager operation — see step detail for specific resource and environment",
    env="local",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', '<resource>', 'success', 'Secret manager operation — see step detail for specific resource and environment', 'local')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''