Playbook: Manual Static Key Rotation

Trigger conditions:

Security event — key suspected or confirmed compromised (LLM session exposure, unauthorized API call, leaked in logs). Follow docs/playbooks/static-api-key-incident-rotation.md first to triage scope, then return here for per-key steps.
Scheduled hygiene — credential-rotation.yml (Monday 06:00 UTC) opens a human-action GitHub issue when next_due < today for any key listed below. Execute those steps and close the issue.

These keys do not auto-rotate — there is no script that can roll them programmatically without human involvement in a browser dashboard. Rotation frequency guidance is listed per key.

Rotation summary

Credential	GH Secret(s)	SOPS file	Next due	Dashboard
`GH_TOKEN`	`GH_TOKEN`, `GH_PAT`	none	2027-05-08	github.com/settings/tokens
`VERCEL_TOKEN`	`VERCEL_TOKEN`	none	2027-05-08	vercel.com/account/tokens
`SENTRY_AUTH_TOKEN`	`SENTRY_AUTH_TOKEN`	none	2027-05-08	sentry.io/settings/auth-tokens
`ANTHROPIC_API_KEY`	`ANTHROPIC_API_KEY`	none	2027-05-08	console.anthropic.com → API Keys
`SUPABASE_SERVICE_KEY`	`SUPABASE_SERVICE_KEY`	`secrets/monitoring.env.sops`	2026-12-21	supabase.com/dashboard → API Keys
`SUPABASE_ACCESS_TOKEN`	`SUPABASE_ACCESS_TOKEN`	none	2026-12-21	supabase.com/dashboard/account/tokens

1. GH_TOKEN (+ GH_PAT)

What it is: GitHub Personal Access Token (classic or fine-grained). GH_TOKEN is used by CI workflows for cross-repo API calls and runner registration. GH_PAT is used by health-check.yml to check runner status. Both can be the same token or separate tokens with different scopes.

Location: GH Secrets GH_TOKEN and GH_PAT · .env.local on dev workstation
Not in SOPS — CI-only credential; never deployed to a VPS .env file.
Frequency: 365 days (or immediately on suspected exposure)

Rotation steps

Step 1 — open GitHub token settings:
  https://github.com/settings/tokens

Step 2 — create a new token (classic PAT):
  Name: p24-infra-cicd-<year>
  Expiration: No expiration (or 1 year — calendar reminder needed if expiry set)
  Scopes needed:
    repo            (full repo access — write, webhooks, secrets)
    workflow        (update GitHub Actions workflows)
    admin:org       (read:org is sufficient for project board access)
    read:user       (user email, for gh api /user calls)
  → Click "Generate token" → copy value immediately (shown only once)

Step 3 — update GH Secrets (both aliases in one pass):
  gh secret set GH_TOKEN  --body "<new_token>" --repo radieu/p24-infra
  gh secret set GH_PAT    --body "<new_token>" --repo radieu/p24-infra

Step 4 — update .env.local on dev workstation:
  GH_TOKEN=<new_token>
  GH_PAT=<new_token>

Step 5 — verify the new token works:
  gh api /user -H "Authorization: Bearer <new_token>" | Select-String login
  # Expected: "login": "radieu" (or your GitHub username)

Step 6 — trigger a test workflow run to confirm CI still works:
  gh workflow run health-check.yml --repo radieu/p24-infra
  gh run list --repo radieu/p24-infra --workflow health-check.yml --limit 1

Step 7 — revoke the old token:
  https://github.com/settings/tokens → find old token by name → Delete

Step 8 — update dev_r_services:
  UPDATE dev_r_services
  SET last_rotated = '<date>', next_due = '<date + 365 days>'
  WHERE service_name IN ('GH_TOKEN', 'GH_PAT');

Log entry:

| <date> | GH_TOKEN GH_PAT | scheduled | radieu | yes |

Escalation: If CI fails after rotation, check that both GH_TOKEN and GH_PAT were updated (they may be different secrets in some workflows). Run gh secret list --repo radieu/p24-infra to confirm both are set.

2. VERCEL_TOKEN

What it is: Vercel API token used by CI for deployments to the et-operational-platform Vercel project, and by the cost-exporter to query Vercel billing data.

Location: GH Secret VERCEL_TOKEN · .env.local on dev workstation
Not in SOPS — not deployed to any VPS.
Frequency: 365 days (or immediately on suspected exposure)

Rotation steps

Step 1 — open Vercel token settings:
  https://vercel.com/account/tokens

Step 2 — create a new token:
  Name: p24-infra-cicd-<year>
  Scope: Full Account (or limit to specific team if applicable)
  Expiration: No Expiration
  → Click "Create" → copy token value (shown only once)

Step 3 — update GH Secret:
  gh secret set VERCEL_TOKEN --body "<new_token>" --repo radieu/p24-infra

Step 4 — update .env.local on dev workstation:
  VERCEL_TOKEN=<new_token>

Step 5 — verify the new token works:
  Invoke-RestMethod "https://api.vercel.com/v2/user" `
    -Headers @{ Authorization = "Bearer <new_token>" } |
    Select-Object -ExpandProperty user | Select-Object username
  # Expected: your Vercel username

Step 6 — verify cost-exporter picks up the new token:
  # After secrets-sync.yml deploys (or manual push):
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart cost-exporter"
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose logs --tail=20 cost-exporter"
  # Look for: "Vercel billing fetched successfully" or no authentication errors

Step 7 — verify CI deployment still works:
  gh workflow run deploy.yml --repo radieu/et-operational-platform 2>/dev/null || true
  # Or trigger a trivial commit to et-operational-platform to kick off a deploy

Step 8 — revoke old token at:
  https://vercel.com/account/tokens → find old token by name → Delete

Step 9 — update dev_r_services:
  UPDATE dev_r_services
  SET last_rotated = '<date>', next_due = '<date + 365 days>'
  WHERE service_name = 'VERCEL_TOKEN';

Log entry:

| <date> | VERCEL_TOKEN | scheduled | radieu | yes |

Escalation: If Vercel deployments fail with 401, confirm the GH Secret was updated (propagation can take ~1 minute). Check the workflow run logs with gh run view --log <run_id> --repo radieu/et-operational-platform.

3. SENTRY_AUTH_TOKEN

What it is: Sentry authentication token used to upload source maps and track deployments in Sentry.io. Used by CI in the et-operational-platform build pipeline.

Location: GH Secret SENTRY_AUTH_TOKEN · secrets/monitoring.env.sops · .env.local on dev workstation
Frequency: 180 days (or immediately on suspected exposure)

Rotation steps

Step 1 — open Sentry auth token settings:
  https://sentry.io/settings/auth-tokens/

Step 2 — create a new token:
  Name: p24-infra-cicd-<year>
  Scopes needed (minimum):
    project:releases   (create and update releases)
    org:read           (read org info for CLI)
  → Click "Create Token" → copy value (shown only once)

Step 3 — update GH Secret:
  gh secret set SENTRY_AUTH_TOKEN --body "<new_token>" --repo radieu/p24-infra

Step 4 — update .env.local on dev workstation:
  SENTRY_AUTH_TOKEN=<new_token>

Step 5 — verify the new token works:
  Invoke-RestMethod "https://sentry.io/api/0/projects/" `
    -Headers @{ Authorization = "Bearer <new_token>" } |
    Select-Object -First 1 | Select-Object slug
  # Expected: your Sentry project slug(s)

Step 6 — trigger a CI build to confirm source maps upload:
  # After a real deployment, check Sentry → Releases for a new entry
  # No 401 errors in the release upload step

Step 7 — revoke old token:
  https://sentry.io/settings/auth-tokens/ → find old token → Revoke

Step 8 — update dev_r_services:
  UPDATE dev_r_services
  SET last_rotated = '<date>', next_due = '<date + 365 days>'
  WHERE service_name = 'SENTRY_AUTH_TOKEN';

Log entry:

| <date> | SENTRY_AUTH_TOKEN | scheduled | radieu | yes |

4. ANTHROPIC_API_KEY

What it is: Anthropic API key used by the audit-engine on vps-i1 (direct API calls — the claude-proxy is not deployed on vps-i1). Also stored as a GH Secret for potential CI use. VPS agents (vps-i1, bms-4) use OAuth subscription auth — NOT this key.

Important: ANTHROPIC_API_KEY is not the credential that keeps Claude Code running on VPSes. The VPS agents authenticate via OAuth (/home/claude-runner/.claude/.credentials.json). Rotating this key has no effect on VPS agent sessions. See docs/playbooks/claude-oauth-reauth.md for VPS agent re-auth.

Location: GH Secret ANTHROPIC_API_KEY · .env.local on dev workstation · monitoring/.env on vps-i1 (consumed by audit-engine)
Not in SOPS — treated as a CI-only secret; synced to vps-i1 via secrets-sync.yml.
Frequency: 365 days (or immediately on suspected exposure)

Rotation steps

Step 1 — open Anthropic console:
  https://console.anthropic.com → API Keys

Step 2 — create a new key:
  Name: p24-infra-audit-engine-<year>
  → Click "Create Key" → copy value (shown only once; starts with sk-ant-api03-)

Step 3 — update GH Secret:
  gh secret set ANTHROPIC_API_KEY --body "<new_key>" --repo radieu/p24-infra

Step 4 — update .env.local on dev workstation:
  ANTHROPIC_API_KEY=<new_key>

Step 5 — deploy to vps-i1 via secrets-sync.yml:
  # Push commit and merge to dev → secrets-sync.yml runs automatically
  # Or manually update (if urgent):
  ssh root@217.154.82.162 "sed -i 's/ANTHROPIC_API_KEY=.*/ANTHROPIC_API_KEY=<new_key>/' /opt/p24-infra/monitoring/.env"

Step 6 — restart audit-engine to pick up the new key:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart audit-engine 2>/dev/null || true"
  # Note: audit-engine may not be a running container (it is code-only as of 2026-06-23)
  # If not running, no restart needed — key will be read at next launch

Step 7 — verify the new key is valid:
  # From PowerShell (uses Invoke-RestMethod to avoid Python SSL issues):
  Invoke-RestMethod "https://api.anthropic.com/v1/models" `
    -Headers @{ "x-api-key" = "<new_key>"; "anthropic-version" = "2023-06-01" } |
    Select-Object -ExpandProperty data | Select-Object -First 1 id
  # Expected: a model id like "claude-sonnet-4-..."

Step 8 — revoke old key:
  https://console.anthropic.com → API Keys → find old key → Delete

Step 9 — update dev_r_services:
  UPDATE dev_r_services
  SET last_rotated = '<date>', next_due = '<date + 365 days>'
  WHERE service_name = 'ANTHROPIC_API_KEY';

Log entry:

| <date> | ANTHROPIC_API_KEY | scheduled | radieu | yes |

Escalation: If you cannot access console.anthropic.com, the key cannot be rotated autonomously. File a human-action GitHub issue: "🔴 CRITICAL: ANTHROPIC_API_KEY rotation blocked — console.anthropic.com inaccessible".

5. SUPABASE_SERVICE_KEY

What it is: Supabase service_role key — grants full database access, bypasses RLS. Used by all backend services that write to Supabase (audit-engine, n8n workflows, scripts).

CRITICAL LIMITATION — sb_secret_ keys are dashboard-only: The Supabase Management API (POST /v1/projects/{ref}/api-keys) returns the new api_key value with U+00B7 masking dots even on creation. This is a confirmed Supabase limitation (verified 2026-06-21 via byte-level check). The secret key value is ONLY visible in the browser dashboard UI at the moment of creation. It cannot be retrieved via API after creation.

Procedure for handling the value:

Copy the value from the Supabase dashboard UI.

Store it immediately in .env.local as SUPABASE_SERVICE_KEY_NEW=<value>.

Read from .env.local into SOPS and GH Secrets (never paste in chat).

Delete SUPABASE_SERVICE_KEY_NEW from .env.local after SOPS is updated.

Location: GH Secret SUPABASE_SERVICE_KEY · secrets/monitoring.env.sops · /opt/p24-infra/monitoring/.env on vps-i1
Frequency: 180 days (higher risk key — it bypasses RLS)

Rotation steps

Step 1 — open Supabase dashboard API keys:
  https://supabase.com/dashboard/project/mwkqmgadqnkkihjdeqsi/settings/api

Step 2 — create a new secret key:
  Click "Add new key" (or similar — UI may vary)
  Type: Secret key (service_role replacement)
  Name: service-role-<year>-<month>
  → The new key value is displayed ONCE — copy it immediately.
  → Paste into .env.local on dev workstation:
     SUPABASE_SERVICE_KEY_NEW=<new_key>

Step 3 — update SOPS file with new key:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Read new key from .env.local (not from chat/clipboard directly):
$newKey = (Get-Content "d:\code_2026\p24-infra\.env.local" |
    Where-Object { $_ -match '^SUPABASE_SERVICE_KEY_NEW=' }) -replace '^SUPABASE_SERVICE_KEY_NEW=', ''
 
# Decrypt SOPS to temp file (must stay inside secrets/ for path_regex match):
$plain = sops --decrypt --input-type dotenv --output-type dotenv `
    "C:\code_2026\p24-infra\secrets\monitoring.env.sops"
$updated = $plain | ForEach-Object {
    if ($_ -match '^SUPABASE_SERVICE_KEY=') { "SUPABASE_SERVICE_KEY=$newKey" } else { $_ }
}
[System.IO.File]::WriteAllText(
    "C:\code_2026\p24-infra\secrets\mon-edit.env.sops",
    ($updated -join "`n") + "`n",
    [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv `
    "C:\code_2026\p24-infra\secrets\mon-edit.env.sops"
[System.IO.File]::WriteAllText(
    "C:\code_2026\p24-infra\secrets\monitoring.env.sops",
    ($enc -join "`n") + "`n",
    [System.Text.UTF8Encoding]::new($false))
Remove-Item "C:\code_2026\p24-infra\secrets\mon-edit.env.sops" -Force
Write-Host "SOPS updated. Verify: sops -d secrets/monitoring.env.sops | grep SUPABASE_SERVICE_KEY"

Step 4 — update GH Secret (read from .env.local, not from chat):
  # PowerShell:
  $newKey = (Get-Content "d:\code_2026\p24-infra\.env.local" |
      Where-Object { $_ -match '^SUPABASE_SERVICE_KEY_NEW=' }) -replace '^SUPABASE_SERVICE_KEY_NEW=', ''
  gh secret set SUPABASE_SERVICE_KEY --body $newKey --repo radieu/p24-infra

Step 5 — update .env.local: rename NEW → active:
  # Remove SUPABASE_SERVICE_KEY_NEW= line, add/replace SUPABASE_SERVICE_KEY= line
  # (Edit .env.local manually — do not let the value appear in chat output)

Step 6 — deploy to vps-i1 via secrets-sync.yml:
  git add secrets/monitoring.env.sops
  git commit -m "chore: rotate SUPABASE_SERVICE_KEY"
  git push && gh pr create ... (or merge existing PR)
  # secrets-sync.yml auto-runs on merge to dev/main

  # OR if urgent — manual push:
  ssh root@217.154.82.162 "sed -i 's/SUPABASE_SERVICE_KEY=.*/SUPABASE_SERVICE_KEY=<new_key>/' \
    /opt/p24-infra/monitoring/.env"

Step 7 — restart consumers that hold the key in memory:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && \
    docker compose restart queue-exporter pg-stats-exporter backup-exporter"
  # audit-engine (if running): docker compose restart audit-engine

Step 8 — verify the new key works:
  # From dev workstation (PowerShell):
  $newKey = (Get-Content "d:\code_2026\p24-infra\.env.local" |
      Where-Object { $_ -match '^SUPABASE_SERVICE_KEY=' }) -replace '^SUPABASE_SERVICE_KEY=', ''
  Invoke-RestMethod `
    "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/dev_r_services?limit=1" `
    -Headers @{
        apikey        = $newKey
        Authorization = "Bearer $newKey"
    } | ConvertTo-Json -Depth 2
  # Expected: JSON with one dev_r_services row (not a 401 error)

Step 9 — revoke old key:
  https://supabase.com/dashboard/project/mwkqmgadqnkkihjdeqsi/settings/api
  → Find old key by name → Delete / Revoke

Step 10 — update dev_r_services:
  UPDATE dev_r_services
  SET last_rotated = '<date>', next_due = '<date + 180 days>'
  WHERE service_name = 'SUPABASE_SERVICE_KEY';

Log entry:

| <date> | SUPABASE_SERVICE_KEY | scheduled | radieu | yes |

Escalation: If exporters fail after restart (401 on Supabase calls), confirm the new key was correctly written to /opt/p24-infra/monitoring/.env — check with ssh root@217.154.82.162 "grep SUPABASE_SERVICE_KEY /opt/p24-infra/monitoring/.env | cut -c1-40". If SOPS update is blocked (age key unavailable), store the new value in .env.local as SUPABASE_SERVICE_KEY_NEW= and update SOPS at the next session.

6. SUPABASE_ACCESS_TOKEN

What it is: Supabase Management API token — allows managing projects, creating branches, running migrations, and generating TypeScript types via the Supabase management REST API. Used by the mcp__claude_ai_Supabase__* MCP tools in Claude sessions. (audit-engine is not deployed — see CLAUDE.md.)

Location (corrected #4585 — the previous “Not in SOPS” claim was wrong):

Copy	Where	Notes
`SUPABASE_ACCESS_TOKEN`	`secrets/administration.env.sops`	canonical; excluded from `secrets-sync.yml`, so never deployed to any server
`ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN`	`secrets/role-secret-manager.env.sops`	worker-accessible copy; same underlying value (#3825)
GH Secret `SUPABASE_ACCESS_TOKEN`	`radieu/p24-infra`	pending replacement by the CF Worker proxy — see #3835

Never in secrets/monitoring.env.sops — moved out 2026-07-05 (560fb17e, #2075 least-privilege), last in-stack consumer removed by 9abcbaa (#2620). Adding it there pushes an org-level Management PAT into the process env of alertmanager/caddy/loki/mezmo-exporter, which all use env_file: .env. PR #4700 was rejected for exactly this.

⚠️ Mint/write path is currently broken — see #4626. This token authenticates and reads fine, but keys it mints return 401 Invalid API key against the project data API. A successful GET is not proof that a rotation will succeed.

Frequency: 180 days (management-plane access — can alter schema and keys)

Rotation steps

Step 1 — open Supabase account tokens:
  https://supabase.com/dashboard/account/tokens

Step 2 — generate a new token:
  Name: p24-infra-management-<year>
  → Click "Generate token" → copy value immediately (shown only once; starts with sbp_...)

Step 3 — update GH Secret:
  gh secret set SUPABASE_ACCESS_TOKEN --body "<new_token>" --repo radieu/p24-infra

Step 3a — update BOTH SOPS copies (secret-manager role only; see supabase-access-token-rotation.md
          for the exact SOPS write pattern + canary decrypt). Missing either one leaves a stale
          token that fails with 401 later:
  secrets/administration.env.sops       → SUPABASE_ACCESS_TOKEN
  secrets/role-secret-manager.env.sops  → ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN
  # Do NOT add this key to secrets/monitoring.env.sops (#2075/#2620/#4585).

Step 4 — update .env.local on dev workstation:
  SUPABASE_ACCESS_TOKEN=<new_token>
  # Also update SUPABASE_ACCESS_TOKEN in any Claude MCP config that reads from .env.local

Step 5 — verify the new token works:
  Invoke-RestMethod "https://api.supabase.com/v1/projects" `
    -Headers @{ Authorization = "Bearer <new_token>" } |
    Where-Object { $_.id -eq "mwkqmgadqnkkihjdeqsi" } |
    Select-Object id, name, status
  # Expected: project "mwkqmgadqnkkihjdeqsi" with status "ACTIVE_HEALTHY"

Step 6 — revoke old token:
  https://supabase.com/dashboard/account/tokens
  → Find old token by name → Revoke

Step 7 — update dev_r_services:
  UPDATE dev_r_services
  SET last_rotated = '<date>', next_due = '<date + 180 days>'
  WHERE service_name = 'SUPABASE_ACCESS_TOKEN';

Log entry:

| <date> | SUPABASE_ACCESS_TOKEN | scheduled | radieu | yes |

Note on Claude MCP: The Supabase MCP tool (mcp__claude_ai_Supabase__*) reads SUPABASE_ACCESS_TOKEN from the MCP server config, not from .env.local. After rotating, restart the Claude Code session to force the MCP server to re-read the token from the updated config location.

After any rotation — standard checklist

[ ] New value generated at provider dashboard (never via management API for sb_secret_ keys)
[ ] GH Secret updated: gh secret set KEY_NAME --body "<value>" --repo radieu/p24-infra
[ ] SOPS file updated (if applicable): secrets/*.env.sops committed and pushed
[ ] .env.local on dev workstation updated
[ ] Value deployed to VPS .env via secrets-sync.yml or SSH (if applicable)
[ ] Affected service(s) restarted and healthy
[ ] Old key/token revoked at the issuing service dashboard
[ ] last_rotated + next_due updated in dev_r_services
[ ] Row appended to docs/secrets-rotation-log.md

docs/playbooks/static-api-key-incident-rotation.md — emergency triage when a key is suspected compromised
docs/playbooks/credential-rotation-180d.md — auto-rotated keys (Grafana, WAHA, Wasabi, SSH, SMTP)
docs/secrets-management.md — SOPS+age setup, key lookup order, add/rotate procedures
docs/secrets-rotation-log.md — append a row after every rotation

p24-infra Docs

Explorer

credential-rotation-manual-static

Playbook: Manual Static Key Rotation

Rotation summary

1. GH_TOKEN (+ GH_PAT)

Rotation steps

2. VERCEL_TOKEN

Rotation steps

3. SENTRY_AUTH_TOKEN

Rotation steps

4. ANTHROPIC_API_KEY

Rotation steps

5. SUPABASE_SERVICE_KEY

Rotation steps

6. SUPABASE_ACCESS_TOKEN

Rotation steps

After any rotation — standard checklist

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

credential-rotation-manual-static

Playbook: Manual Static Key Rotation

Rotation summary

1. GH_TOKEN (+ GH_PAT)

Rotation steps

2. VERCEL_TOKEN

Rotation steps

3. SENTRY_AUTH_TOKEN

Rotation steps

4. ANTHROPIC_API_KEY

Rotation steps

5. SUPABASE_SERVICE_KEY

Rotation steps

6. SUPABASE_ACCESS_TOKEN

Rotation steps

After any rotation — standard checklist

Related playbooks

Graph View

Table of Contents

Backlinks