p24-infra Docs

master-keys-bootstrap

4 min read

Playbook: Master Keys Bootstrap & Recovery

Owner: p24-infra · Updated: 2026-06-27

Master keys are the credentials that enable ALL other rotation and deployment operations. If a master key is lost or compromised, multiple dependent rotations fail silently. This playbook documents what each master key controls, where it lives, and how to recover.

Master Key Inventory

Master KeyWhere storedControlsRotation frequencyRecovery if lost
Developer age key~/.age/p24-infra-keys.txt + KeePass backupALL SOPS files (incl. administration.env.sops)Never rotate — add new recipient insteadRestore from KeePass; if lost: re-key all 7 SOPS files with sops-set.ps1 -RekeyOnly (not sops updatekeys — #4601)
AGE_KEY_GHAGH Secret radieu/p24-infraMON+N8N+VH1+BMS+ART+BP in CI (NOT administration)Manually on suspected exposureGenerate new age key: age-keygen; re-key all 6 non-admin SOPS files with sops-set.ps1 -RekeyOnly; update GH Secret
GH_PAT_ADMINsecrets/administration.env.sops + GH Secret GH_PATgh secret set in credential-rotation.yml (write+secrets scope)Human-action (browser, PAT page)GitHub Settings → Developer Settings → Personal access tokens → create new with repo + secrets:write scope
SUPABASE_ACCESS_TOKENsecrets/administration.env.sops + GH Secret (+ role-scoped ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN in secrets/role-secret-manager.env.sops for bms-4 workers) — not monitoring.env.sops (#2620, #4585)Supabase Management API (schema, keys). ⚠️ Mint/write path broken — #4626Human-action (dashboard)Supabase dashboard → Account → Access Tokens → create new
SUPABASE_SERVICE_KEYsecrets/monitoring.env.sops + GH SecretAll Supabase DB queries by scripts/exportersHuman-action (sb_secret masked in API)Supabase dashboard → Settings → API → create new secret key
WASABI_ADMIN_ACCESS/SECRET_KEYsecrets/administration.env.sops + GH SecretsWasabi IAM (create/delete keys for rotation)Human-action (Wasabi Console)Wasabi Console → Access Keys → create new; update SOPS + GH Secrets
CF_GLOBAL_API_KEYsecrets/administration.env.sopsCloudflare scoped-token self-rotationHuman-action (CF dashboard)CF dashboard → My Profile → API Tokens → API Keys
OPENAI_ADMIN_KEYGH Secret onlyOpenAI monitoring token rotationHuman-action (OpenAI Console)OpenAI Platform → Settings → API Keys → create new
VPS_SSH_PRIVATE_KEYGH Secretsecrets-sync.yml SSH deploy to vps-i1Rotate when key exposedGenerate new ed25519 key; add pubkey to claude-admin authorized_keys; update GH Secret

dev_r_services entries for master keys

These master keys need rows in dev_r_services with rotation_freq and auto_rotate=false:

INSERT INTO dev_r_services (service_name, element_type, criticality, rotation_freq,
  auto_rotate, sops_file, status, compliance_workbook, compliance_notes)
VALUES
  ('AGE_KEY_GHA','credential','critical','365d',false,'none — GH Secret only','active','yes',
   'Master key for CI SOPS access; rotate only if exposed'),
  ('WASABI_ADMIN_ACCESS_KEY','credential','critical','365d',false,
   'secrets/administration.env.sops','active','yes','IAM admin key for Wasabi key rotation'),
  ('CF_GLOBAL_API_KEY','credential','high','365d',false,
   'secrets/administration.env.sops','active','yes','Global CF key for token self-rotation'),
  ('OPENAI_ADMIN_KEY','credential','medium','365d',false,'none — GH Secret only','active','yes',
   'Admin key for OpenAI monitoring token rotation');

Rotation: GH_PAT after sync-github-secrets.ps1 SYNC_MAP reduction

After Faza 4 (SYNC_MAP reduced to local-only keys), GH_PAT GH Secret is no longer auto-synced from .env.local. To rotate:

  1. Create new PAT at GitHub Settings → Developer Settings → Personal access tokens (Classic) Scopes needed: repo (full) + write:packages (optional) — NOT admin:org
  2. Decrypt administration.env.sops, update GH_PAT_ADMIN value, re-encrypt (canary)
  3. Push SOPS commit to dev → merge
  4. Manually set GH Secret: gh secret set GH_PAT --body "<new_token>" --repo radieu/p24-infra
  5. Run credential-rotation.yml dry-run to verify new token works

AGE_KEY_GHA rotation procedure

  1. Generate new age key: age-keygen -o /tmp/new-age-key.txt
  2. Get new public key: age-keygen -y /tmp/new-age-key.txt
  3. For each of the 6 non-admin SOPS files (NOT administration.env.sops): 
    $env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# -RekeyOnly re-encrypts under whatever recipients .sops.yaml now lists, changing no values.
# NOT `sops updatekeys` — broken on dotenv *.env.sops in SOPS 3.9.1, no flag fixes it (#4601).
.\scripts\sops-set.ps1 -SopsFile secrets\monitoring.env.sops -RekeyOnly
# -RekeyOnly already canary-decrypts and verifies the recipient count before it commits the
# write (and rolls back on failure); this is an independent re-check:
sops -d --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
  4. Update GH Secret: gh secret set AGE_KEY_GHA --body (Get-Content /tmp/new-age-key.txt -Raw) --repo radieu/p24-infra
  5. Delete old private key from /tmp; update KeePass with new key

Emergency: developer age key lost

If ~/.age/p24-infra-keys.txt is lost and not in KeePass:

  1. ALL SOPS files become unreadable locally (CI still works via AGE_KEY_GHA)
  2. Generate new developer age key: age-keygen -o ~/.age/p24-infra-keys.txt
  3. For each SOPS file, decrypt with CI key (AGE_KEY_GHA) → re-encrypt with new dev key: 
    # On vps-i1 (has AGE_KEY_GHA loaded in runner):
sops -d secrets/monitoring.env.sops > /tmp/mon.plain
# Then locally: sops --encrypt-in-place secrets/monitoring.env.sops
  4. Update administration.env.sops: only the new dev key can re-encrypt it
  5. Commit all updated SOPS files to a branch; merge to dev

Graph View

Backlinks