p24-infra Docs

wasabi-iam-rotator

7 min read

Wasabi IAM Rotator — Operational Runbook

Service: infra-src/wasabi-iam-rotator/ Type: Vercel TypeScript serverless function Purpose: Rotate the p24-infra Wasabi IAM user’s access key on demand

Canonical rotation path (#2407). For the p24-infra IAM user, prefer this HTTP endpoint over any local boto3 script. Rotation runs server-side on Vercel; the admin key never leaves SOPS/Vercel and nothing sensitive materializes in the local session, so it sidesteps the command-safety classifier that blocked the #2401 local rotation attempt. Invoke it classifier-safe (token via env, check status code only) — see sops-rotation-classifier-patterns.md Task 3.

Scope gap. This function is hard-scoped to the p24-infra user + Vercel/vps-i1 consumers. It does not rotate other Wasabi keys (e.g. pinbox24Public* in /root/s3v2-prod/*.env on bms-1, from #2401). Those need the classifier-safe local pattern (Task 2) until the endpoint is generalized to accept an iam_username + pluggable consumer.

What this service does

POST /api/rotate (Bearer auth):

  1. Lists current access keys for the p24-infra Wasabi IAM user via iam.wasabisys.com
  2. Creates a new IAM access key
  3. Updates four Vercel env vars with the new key:
    • WASABI_ACCESS_KEY (alias)
    • WASABI_SECRET_KEY (alias)
    • P24_INFRA_WASABI_ACCESS_KEY (primary)
    • P24_INFRA_WASABI_SECRET_KEY (primary)
  4. Triggers secrets-sync.yml workflow dispatch on dev branch (deploys new key to vps-i1 monitoring stack)
  5. Deletes the old access key
  6. Returns {"rotated": true, ...} — new secret key is NOT in the response body, only stored in Vercel

Required Vercel environment variables

These must be set in Vercel project settings before the first deploy. See human-action GH issue for step-by-step.

VariableSourceRequired
WASABI_ADMIN_ACCESS_KEYsecrets/administration.env.sopsYes
WASABI_ADMIN_SECRET_KEYsecrets/administration.env.sopsYes
ROTATOR_API_KEYGenerate: openssl rand -hex 32, store in secrets/administration.env.sopsYes
VERCEL_TOKENFrom secrets/monitoring.env.sops (key: VERCEL_TOKEN)Yes
VERCEL_PROJECT_IDVercel project ID of the deployed functionYes
VERCEL_TEAM_IDVercel team ID (if using a team)No
GH_ROTATION_TOKENGitHub fine-grained PAT with Actions: write scope on radieu/p24-infraYes
WASABI_IAM_USERNAMEDefault: p24-infra — override only if rotating a different userNo

How to get WASABI_ADMIN_* keys (Windows dev machine)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Check key names only (never print values):
sops -d --input-type dotenv --output-type dotenv secrets\administration.env.sops | Select-String "^WASABI_ADMIN"

Deploy

vercel --cwd infra-src/wasabi-iam-rotator deploy --prod

Note the deployed URL — it is needed for calling the endpoint and for VERCEL_PROJECT_ID.

How to trigger rotation

Acquire the rotation lock FIRST — MANDATORY (ADR 004). POST /api/rotate makes the service mint a new Wasabi key in place (a non-idempotent live write), so per ADR 004 hold the per-secret advisory lock before firing it. Two sessions triggering a rotate at once each mint a different key and silently diverge (live Wasabi IAM + Vercel vs SOPS) — the #5925 race.

Run from the repo root so the repo-relative script path resolves on the Windows dev machine (the absolute /opt/p24-infra/... path does not exist there):

# Acquire before triggering. Exit 3 = another session holds it (STOP, reconcile, do NOT trigger);
# exit 4 = Supabase unreachable (fail-closed, do NOT proceed). $LOCK_ID replaces an `open` id.
$env:LOCK_ID = python scripts/rotation-log-entry.py acquire --secret WASABI_ACCESS_KEY --repo p24-infra --reason "#<issue>" --rotation-type auto
if ($LASTEXITCODE -ne 0) { throw "acquire denied/unverified — STOP, reconcile, do NOT trigger a rotate" }
 
# Layer-2 re-check immediately before firing the trigger — lock still held?
python scripts/rotation-log-entry.py check $env:LOCK_ID
if ($LASTEXITCODE -ne 0) { throw "lock lost — abort the rotate and reconcile" }
 
curl -X POST https://<deployed-url>/api/rotate `
  -H "Authorization: Bearer $env:ROTATOR_API_KEY"

On a Linux worker run the equivalent bash: LOCK_ID=$(python3 scripts/rotation-log-entry.py acquire … ) || exit 1, then python3 scripts/rotation-log-entry.py check "$LOCK_ID" || exit 1.

Release the lock after the SOPS update below is committed: rotation-log-entry.py close "$LOCK_ID" (or fail "$LOCK_ID" --error "…") — never leave it pending (it blocks the next rotation and trips the 2 h stalled-rotation alert).

Expected response:

{
  "rotated": true,
  "user": "p24-infra",
  "old_key_id": "ABCDEF...",
  "new_key_id": "XYZ123...",
  "next_steps": [
    "New key deployed to Vercel env vars",
    "secrets-sync triggered — VPS will receive new key on next deployment",
    "Update secrets/monitoring.env.sops manually: sops edit pattern with $env:WASABI_ACCESS_KEY from Vercel dashboard"
  ]
}

What happens after rotation

StepAutomatic?Details
New Wasabi key createdYesVia iam.wasabisys.com
Vercel env vars updatedYesAll 4 aliases updated atomically via Vercel API
secrets-sync.yml triggeredYesDispatched to dev branch — deploys to vps-i1
vps-i1 monitoring stack restartsYes (via secrets-sync)Containers pick up new key on next docker compose up -d
Old Wasabi key deletedYesHappens after new key is stored and VPS trigger is sent
secrets/monitoring.env.sops updatedNo — manual stepSee SOPS update section below

SOPS update (manual step — required after rotation)

The function cannot update SOPS files at runtime (no age key access from Vercel). After a successful rotation:

  1. Open Vercel dashboard → project → Settings → Environment Variables
  2. Note the new values for WASABI_ACCESS_KEY and WASABI_SECRET_KEY
  3. Update secrets/monitoring.env.sops using the standard SOPS edit pattern (Windows):
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
 
# 1. Decrypt to temp file (LF-only, no BOM)
$plain = sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring-edit.env.sops",
  ($plain -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
 
# 2. Edit the file: update WASABI_ACCESS_KEY, WASABI_SECRET_KEY,
#    P24_INFRA_WASABI_ACCESS_KEY, P24_INFRA_WASABI_SECRET_KEY
#    (use your editor or PowerShell string replacement -- never echo values in chat)
 
# 3. Re-encrypt
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\monitoring-edit.env.sops
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring.env.sops", ($enc -join "`n") + "`n",
  [System.Text.UTF8Encoding]::new($false))
 
# 4. Canary check before git add
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit. See docs/playbooks/sops-windows-crlf.md" }
Write-Host "Canary OK"
 
# 5. Remove temp plaintext
[System.IO.File]::Delete("$PWD\secrets\monitoring-edit.env.sops")
 
# 6. Commit
git add secrets/monitoring.env.sops
git commit -m "chore: rotate Wasabi p24-infra IAM key"
  1. Also update GH Secrets (for CI/CD parity):
# Read new values safely first (never print them):
$env:NEW_ACCESS = (sops -d --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "^P24_INFRA_WASABI_ACCESS_KEY=").ToString().Split("=",2)[1]
$env:NEW_SECRET = (sops -d --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "^P24_INFRA_WASABI_SECRET_KEY=").ToString().Split("=",2)[1]
gh secret set P24_INFRA_WASABI_ACCESS_KEY --body "$env:NEW_ACCESS" --repo radieu/p24-infra
gh secret set P24_INFRA_WASABI_SECRET_KEY --body "$env:NEW_SECRET" --repo radieu/p24-infra
gh secret set WASABI_ACCESS_KEY --body "$env:NEW_ACCESS" --repo radieu/p24-infra
gh secret set WASABI_SECRET_KEY --body "$env:NEW_SECRET" --repo radieu/p24-infra
$env:NEW_ACCESS = ""; $env:NEW_SECRET = ""
  1. Add a row to docs/secrets-rotation-log.md with “Confirmed in sync” = yes only when all consumers are updated.

Verification after rotation

# Test that the new key works (replace with new key values from Vercel dashboard)
aws s3 ls s3://p24-infra \
  --endpoint-url https://s3.eu-central-2.wasabisys.com \
  --region eu-central-2

Expected: bucket listing without errors.

Also check vps-i1 monitoring stack logs (wait ~2 minutes after rotation for secrets-sync to complete):

ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose logs --tail=20 thanos-sidecar"

Expected: no authentication errors.

Trigger: when to rotate

  • Routine rotation: every 90 days (see docs/secrets-rotation-log.md for last rotation date)
  • Emergency: immediately on suspected credential exposure — follow docs/playbooks/static-api-key-incident-rotation.md

Escalation path

  1. Rotation API returns 500 — check Vercel function logs; most likely WASABI_ADMIN_* keys are expired or wrong
  2. Vercel env vars not updated — VERCEL_TOKEN or VERCEL_PROJECT_ID may be wrong; update manually in Vercel dashboard
  3. secrets-sync not triggered — GH_ROTATION_TOKEN expired; rotate it and update Vercel env vars; run workflow dispatch manually: 
    gh workflow run secrets-sync.yml --repo radieu/p24-infra --ref dev
  4. Old key already deleted but VPS still fails — the new key may not have propagated yet; wait for secrets-sync or restart containers manually: 
    ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose up -d"

Prevention

  • Store ROTATOR_API_KEY in secrets/administration.env.sops — never in code or chat
  • Never return the new secret key in API responses — it is stored in Vercel env vars only
  • Complete the SOPS update within the same session as the rotation; do not defer it
  • Record every rotation in docs/secrets-rotation-log.md

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="WASABI_IAM_KEY",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — Wasabi IAM key pair rotated via wasabi-iam-rotator and SOPS updated",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'WASABI_IAM_KEY', 'success', 'Scheduled rotation — Wasabi IAM key pair rotated via wasabi-iam-rotator and SOPS updated', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks