p24-infra Docs

wasabi-key-rotation

7 min read

Wasabi Key Rotation Playbook

When to use

  • Alert: BackupExporterDown + error InvalidAccessKeyId in backup-exporter logs
  • Alert: ReportNotGenerated / WeeklyReportNotGenerated — Wasabi upload step fails
  • Wasabi key returns InvalidAccessKeyId (key deleted in console) or SignatureDoesNotMatch (secret mismatch in SOPS)
  • Routine rotation (every 90 days per credential-rotation-policy.md)

Windows dev machine — ALWAYS add --no-verify-ssl when testing Wasabi IAM

Known Windows issue (2026-07-08): The AWS CLI on Windows uses Python’s ssl module, which cannot verify Wasabi IAM’s TLS certificate (missing issuer in the Windows Python cert store). Without --no-verify-ssl, the CLI returns exit code 255 with an SSL error that can be mistaken for InvalidAccessKeyId. Always use --no-verify-ssl for all Wasabi IAM and S3 endpoint tests on the Windows dev machine.

# Correct — add --no-verify-ssl on Windows
aws iam list-users --endpoint-url https://iam.wasabisys.com --no-verify-ssl
# exit 0 + user list = key valid
# exit non-0 with InvalidAccessKeyId = key truly invalid

This is the same class of issue as feedback_cloudflare_ssl_windows.md (Python urllib SSL failures for CF API). On Linux workers (vps-i1, bms-4), --no-verify-ssl is not needed — the system cert store is complete.

Primary path — wasabi-iam-rotator (fully automated)

# Get ROTATOR_API_KEY from monitoring.env.sops
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$mon = sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops
$env:R_KEY = ($mon | Select-String "^ROTATOR_API_KEY=").ToString().Split("=",2)[1].Trim()
 
$resp = Invoke-RestMethod -Method POST -Uri "https://wasabi-iam-rotator.vercel.app/api/rotate" `
    -Headers @{ Authorization = "Bearer $env:R_KEY" }
$env:R_KEY = ""
$resp | ConvertTo-Json
# Expect: rotated=true, sops_update_workflow.triggered=true

What happens automatically:

  1. New IAM key created for Wasabi user p24-infra
  2. Key staged in Vercel rotator project (WASABI_ACCESS_KEY encrypted env var)
  3. wasabi-sops-update.yml GH workflow triggered → reads staged key → updates monitoring.env.sops → commits to dev
  4. secrets-sync.yml fires on commit → deploys new key to vps-i1 → docker compose restart

Monitor: GH Actions → wasabi-sops-update + sync-vps-i1 should be green in ~5 min.

If rotator fails with SignatureDoesNotMatch — WASABI_ADMIN key in Vercel is stale. Fix:

gh workflow run wasabi-admin-sync.yml --repo radieu/p24-infra --ref dev

Then retry the rotation.

If sops_update_workflow.triggered=false — use new_secret_key from the response and follow §Autonomous rotation below.

Key inventory

Env varSOPS fileIAM userUsed by
P24_INFRA_WASABI_ACCESS_KEY + P24_INFRA_WASABI_SECRET_KEYsecrets/monitoring.env.sopsp24-infrabackup-exporter, report-scheduler, Thanos sidecar
WASABI_ACCESS_KEY + WASABI_SECRET_KEYsecrets/monitoring.env.sopsp24-infraaliases — same value, docker-compose compat
WASABI_ADMIN_ACCESS_KEY + WASABI_ADMIN_SECRET_KEYsecrets/administration.env.sops + .env.localroot/admin accountIAM management, boto3 self-rotation

⚠️ Bucket naming hazard — 3 “test-*” buckets hold real Pinbox24 W4 production data

Before any bucket-level Wasabi operation (delete, lifecycle policy, cost-cleanup pass, list-buckets audit), read w3-w4-stack-operations.md — the warning right after its permission-matrix table. Short version: test-replicated-to-us-bucket, test-us-bucket-for-replication-testing, and p24-was-us-east-1 look disposable by name but hold ~215,000 real Pinbox24 W4 files (confirmed 2026-08-04). Do not delete or deprioritize them based on the name alone. Rename tracked in p24-infra#2709, deferred as of 2026-08-05.

Autonomous rotation (Claude can do this without human)

Prerequisite: WASABI_ADMIN_ACCESS_KEY in .env.local is valid (verify first with SignatureDoesNotMatch vs InvalidAccessKeyId).

If the admin key is valid, Claude can rotate all other keys autonomously via boto3.

Acquire the rotation lock FIRST — MANDATORY (ADR 004). iam.create_access_key (step 2 below) is a non-idempotent live write — it mints a new key in place — so per ADR 004 hold the per-secret advisory lock before it. Two sessions rotating this key at once each mint a different key and silently diverge (live Wasabi IAM vs SOPS/vps-i1) — the #5925 race. A SOPS-only edit would not need this; a live key mint does.

# 0. Acquire the lock before minting. Exit 3 = another session holds it (STOP, reconcile, do NOT
#    mint); exit 4 = DB unreachable (fail-closed, do NOT proceed). $LOCK_ID replaces an `open` id.
$env:LOCK_ID = python scripts/rotation-log-entry.py acquire --secret WASABI_ACCESS_KEY --repo p24-infra --reason "#<issue>"
if ($LASTEXITCODE -ne 0) { throw "acquire denied/unverified — STOP, reconcile, do NOT rotate WASABI key" }
 
# 1. Extract admin credentials safely (never print)
$env:WAKI = (Get-Content ".env.local" | Select-String "^WASABI_ADMIN_ACCESS_KEY=").ToString().Split("=",2)[1].Trim()
$env:WAKS = (Get-Content ".env.local" | Select-String "^WASABI_ADMIN_SECRET_KEY=").ToString().Split("=",2)[1].Trim()
 
# 1b. Layer-2 re-check immediately before the live write (mint) — lock still held?
python scripts/rotation-log-entry.py check $env:LOCK_ID
if ($LASTEXITCODE -ne 0) { throw "lock lost — abort the mint and reconcile (do NOT overwrite blind)" }
 
# 2. Create new key for p24-infra IAM user
python -c "
import boto3, json, os
iam = boto3.client('iam',
    endpoint_url='https://iam.wasabisys.com',
    aws_access_key_id=os.environ['WAKI'],
    aws_secret_access_key=os.environ['WAKS'],
    region_name='us-east-1')
key = iam.create_access_key(UserName='p24-infra')['AccessKey']
# Write to .env.local entries for SOPS update (never print raw)
with open('_new_wasabi_key.txt','w') as f:
    f.write(key['AccessKeyId'] + '\n' + key['SecretAccessKey'])
print('New key created for p24-infra')
"
 
# 3. Read new key and update SOPS (see sops-edit-operations.md for full pattern)
$lines = Get-Content "_new_wasabi_key.txt"
$env:NEW_ACCESS = $lines[0]; $env:NEW_SECRET = $lines[1]
Remove-Item "_new_wasabi_key.txt"
 
# 4. Decrypt → update → re-encrypt monitoring.env.sops
$plain = sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops
$updated = $plain | ForEach-Object {
    if ($_ -match "^P24_INFRA_WASABI_ACCESS_KEY=") { "P24_INFRA_WASABI_ACCESS_KEY=$env:NEW_ACCESS" }
    elseif ($_ -match "^P24_INFRA_WASABI_SECRET_KEY=") { "P24_INFRA_WASABI_SECRET_KEY=$env:NEW_SECRET" }
    elseif ($_ -match "^WASABI_ACCESS_KEY=") { "WASABI_ACCESS_KEY=$env:NEW_ACCESS" }
    elseif ($_ -match "^WASABI_SECRET_KEY=") { "WASABI_SECRET_KEY=$env:NEW_SECRET" }
    else { $_ }
}
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring-edit.env.sops", ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\monitoring-edit.env.sops
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS canary FAILED" }
[System.IO.File]::Delete("$PWD\secrets\monitoring-edit.env.sops")
$env:NEW_ACCESS = ""; $env:NEW_SECRET = ""; $env:WAKI = ""; $env:WAKS = ""
Write-Host "SOPS updated and canary OK"
 
# 5. Delete old key (optional — do after confirming new key works on server)
# python -c "iam.delete_access_key(UserName='p24-infra', AccessKeyId='OLD_KEY_ID')"
 
# 6. Release the ADR 004 lock once the new key is verified + distributed (never leave it pending —
#    it blocks the next rotation and trips the 2h stalled-rotation alert):
python scripts/rotation-log-entry.py close $env:LOCK_ID   # or: fail $env:LOCK_ID --error "..."
$env:LOCK_ID = ""

Human-action rotation (admin key also broken)

If both WASABI_ADMIN_ACCESS_KEY and P24_INFRA_WASABI_ACCESS_KEY are invalid:

  1. Log in to Wasabi Console (browser) as root account
  2. Root account keys: top-right corner → account name → “Access Keys” → create new key → update secrets/administration.env.sops as WASABI_ADMIN_ACCESS_KEY + WASABI_ADMIN_SECRET_KEY and .env.local
  3. IAM → Users → p24-infra → Access Keys → create new key
  4. Add to .env.local temporarily: 
    P24_INFRA_WASABI_ACCESS_KEY=<new-id>
P24_INFRA_WASABI_SECRET_KEY=<new-secret>
  5. Tell Claude → Claude reads from .env.local and updates SOPS autonomously

After SOPS update

# 1. Commit and PR → main
git add secrets/monitoring.env.sops
git commit -m "fix: rotate P24_INFRA_WASABI_ACCESS_KEY in monitoring.env.sops"
gh pr create --base main ...
 
# 2. After merge — secrets-sync.yml auto-deploys to vps-i1
# OR manual apply on vps-i1:
ssh root@217.154.82.162 "cd /opt/p24-infra && git pull && cd monitoring && docker compose up -d --no-deps backup-exporter"
 
# 3. Verify
curl http://217.154.82.162:9220/metrics | grep wasabi

After rotation — catch-up tasks

If rotation was triggered by missed reports:

# Re-run przeglady-hu-sp-uvv (runs at 02:00 daily)
ssh root@217.154.82.162 "cd /opt/p24-infra && python scripts/report_scheduler.py --report przeglady-hu-sp-uvv"
 
# Re-run missed weekly przeglady-tacho-agregat (runs Sundays 05:00)
ssh root@217.154.82.162 "cd /opt/p24-infra && python scripts/report_scheduler.py --report przeglady-tacho-agregat --date 2026-06-22"

Append to rotation log

docs/secrets-rotation-log.md:

| 2026-06-27 HH:MM UTC | #ISSUE | P24_INFRA_WASABI_ACCESS_KEY | Key deleted in Wasabi console, backup-exporter down | Claude/Radek | yes |

Prevention

The 2026-06-27 and 2026-06-12 incidents share the same root cause: key deleted in Wasabi console without updating SOPS. Monitoring alert BackupExporterDown fires within 5 min of key invalidity — that is the earliest detection signal. No further prevention mechanism is needed beyond acting on the alert promptly.

If WASABI_ADMIN_ACCESS_KEY is valid in .env.local: Claude can rotate autonomously without human console access. Claude should attempt autonomous rotation first before creating a human-action issue.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="WASABI_ACCESS_KEY",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — Wasabi S3 access key rotated and deployed to monitoring stack",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'WASABI_ACCESS_KEY', 'success', 'Scheduled rotation — Wasabi S3 access key rotated and deployed to monitoring stack', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks