p24-infra Docs

credential-rotation-180d

20 min read

Playbook: 180-Day Credential Rotation (Manual)

Trigger: credential-rotation.yml (Monday 06:00 UTC) opens a GitHub issue with label human-action when next_due < today for any credential with auto_rotate=false. This playbook covers the manual rotation procedures for those credentials.

Automation coverage summary

Before doing anything manually, check whether credential-rotation.yml already handles the credential:

CredentialAuto-rotate?Script action
GRAFANA_ADMIN_PASSWORDYESbcrypt hash updated in Caddyfile; Grafana + Caddy restarted — but see note below
SUPABASE_GRAFANA_PASSWORDYESALTER ROLE grafana_readonly; Grafana restarted
EMAIL_SENDER_API_KEYYESCF worker key rotated via Cloudflare API
CLOUDFLARE_TOKEN_ZINTEGROWANAYESToken rolled via PUT /v1/user/tokens/{id}/value using global key
WASABI_KEYS_P24_INFRAYESNew IAM key created; old deleted; all consumers updated + restarted
BMS4_N8N_API_KEYYESNew key via n8n REST API; GH Secret + SOPS updated
SSH_KEY_CLAUDE_ADMINYESNew ed25519 pair; authorized_keys on all VPSes updated; GH Secret updated
MYSQL_PASSWORDYESALTER USER in Traccar MySQL; .env updated; container restarted
PDF_SERVICE_API_KEYYES (#1503)Random hex; vps-i1 .env + SOPS updated; pdf-service recreated
GF_RENDERING_RENDERER_TOKENYES (#1503)Random hex; vps-i1 .env + SOPS; grafana + grafana-image-renderer recreated
REDIS_PASSWORDYES (#1503)bms-4 .env + SOPS; redis + all n8n services recreated (BullMQ jobs re-queue on reconnect)
TRACCAR_PASSWORDYES (#1503)Traccar REST API (PUT /api/users/{id}) + SOPS
TRACCAR_FORWARD_TOKENYES (#1503)CF Worker traccar-gw FORWARD_TOKEN + Traccar /root/traccar/.env + SOPS
TRACCAR_GW_ADMIN_KEYYES (#1503)CF Worker traccar-gw ADMIN_KEY + SOPS
TRACCAR_ADMIN_KEYYES (#1503)SOPS only — no active runtime consumer
N8N_HU_SP_REPORT_SECRETYES (#1503)bms-4 .env + SOPS (n8n-bms4 + vps-h1); n8n services recreated
GITHUB_TRIGGER_SECRETYES (#1503)bms-4 + vps-h1 .env + SOPS; n8n recreated. No GH Secret (GitHub rejects GITHUB_ prefix)
CRON_SECRETYES (#1503)Vercel env (et-operational-platform) + SOPS
CLAUDE_SESSION_MANAGER_KEYYES (#1503)bms-4 .env + SOPS; claude-session-manager restarted
N8N_GPS_SYNC_SECRETNO (#1503)n8n validator is a credential-vault entry (no API) — rotate n8n UI + Vercel + SOPS together

#1503 handlers are latent. Adding a handler does not start rotating a secret. Rotation only fires when the credential’s dev_r_services row has auto_rotate=true and next_due is reached (or FORCE_ALL=true / ONLY_SERVICES=<name> is set). Before flipping auto_rotate=true for any of the above, validate with a dry run (DRY_RUN=true ONLY_SERVICES=<name> python scripts/rotate-credentials.py) then a single staged live rotation.

⚠️ GRAFANA_ADMIN_PASSWORD — mandatory manual step after every rotation

Grafana stores its admin password in its own SQLite database (/var/lib/grafana/grafana.db). Updating GF_SECURITY_ADMIN_PASSWORD in .env and restarting the container does not update the stored password — it is only read on first boot when the DB is empty. Failing to run the CLI reset step means the dashboard login breaks after every rotation.

After credential-rotation.yml completes (or after any manual GRAFANA_ADMIN_PASSWORD change), always run:

ssh root@217.154.82.162 'PASS=$(grep "^GRAFANA_ADMIN_PASSWORD=" /opt/p24-infra/monitoring/.env | cut -d= -f2-) && docker exec monitoring-grafana-1 grafana cli admin reset-admin-password "$PASS" 2>&1 | tail -3; unset PASS'
# Expected last line: Admin password changed successfully ✔

Then update .env.local on the dev workstation to match (read from SOPS — never from the terminal):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$dec = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops 2>$null
$v = ($dec | Select-String "^GRAFANA_ADMIN_PASSWORD=").ToString().Split("=",2)[1]
$content = Get-Content "C:\code_2026\p24-infra\.env.local" -Raw
$content = $content -replace "(?m)^GRAFANA_ADMIN_PASSWORD=.*$", "GRAFANA_ADMIN_PASSWORD=$v"
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\.env.local", $content, [System.Text.UTF8Encoding]::new($false))
$dec = $null; $v = $null

Verify login works: open https://grafana.vps-i1.infra.zintegrowana.online → username admin → password from .env.local.

The credentials below are not auto-rotated — manual action required.

1. CF_API_TOKEN (Cloudflare broad-scope token)

What it is: Cloudflare API token with broader-than-DNS scope — used for Workers, R2, email routing, KV namespace management.
Location: GH Secret CF_API_TOKEN · .env.local
Frequency: 180 days
Rotation method: Dashboard only — this token has User:API Tokens:Edit permission (required to roll CLOUDFLARE_TOKEN_ZINTEGROWANA), so it cannot roll itself.

# Step 1 — open Cloudflare dashboard
# https://dash.cloudflare.com → Profile → API Tokens
 
# Step 2 — create a new token with identical permissions to the existing one
# Scopes needed:
#   Zone:DNS:Edit   (for dns-manager.py)
#   User:API Tokens:Edit  (required for credential-rotation.yml to roll CLOUDFLARE_TOKEN_ZINTEGROWANA)
#   Zone:Cache Purge, Workers Scripts:Edit, R2:Edit  (if used)
 
# Step 3 — update GH Secret and .env.local
gh secret set CF_API_TOKEN --body "<new_token>" --repo radieu/p24-infra
(gc d:\code_2026\p24-infra\.env.local) -replace "^CF_API_TOKEN=.*", "CF_API_TOKEN=<new_token>" |
  [System.IO.File]::WriteAllText("d:\code_2026\p24-infra\.env.local", ($_ -join "`n") + "`n",
    [System.Text.UTF8Encoding]::new($false))
 
# Step 4 — verify credential-rotation.yml can roll CLOUDFLARE_TOKEN_ZINTEGROWANA
# Trigger workflow_dispatch with dry_run=true:
gh workflow run credential-rotation.yml --repo radieu/p24-infra -f dry_run=true -f force_all=false
 
# Step 5 — delete old token from Cloudflare dashboard
 
# Step 6 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'CF_API_TOKEN';

Log entry: | <date> HH:MM UTC | — | CF_API_TOKEN | scheduled | radieu | yes |

2. HSTGR_N8N_MCP_TOKEN

What it is: MCP server authentication token for the n8n MCP server integration (Claude ↔ n8n). Community edition has no REST API for token management — must be rotated through the n8n UI.
Location: GH Secret HSTGR_N8N_MCP_TOKEN · .env.local · n8n credential manager
Frequency: 180 days

Step 1 — open n8n instance:
  https://n8n.vps-h1.infra.zintegrowana.online  (Hostinger)
  or https://n8n.bms-4.infra.zintegrowana.online  (bms-4)

Step 2 — navigate to:
  Settings → API → Regenerate API key  (for HSTGR_N8N_API_KEY)
  
  For MCP token specifically:
  Settings → Community Nodes → MCP Server → Regenerate token
  (path varies by n8n version — may be under Settings → n8n API)

Step 3 — copy new token

Step 4 — update GH Secret:
  gh secret set HSTGR_N8N_MCP_TOKEN --body "<new>" --repo radieu/p24-infra

Step 5 — update .env.local on dev workstation:
  HSTGR_N8N_MCP_TOKEN=<new>

Step 6 — restart any Claude MCP session that uses n8n integration (session re-reads token)

Step 7 — verify: trigger a Claude ↔ n8n tool call and confirm it succeeds

Step 8 — update dev_r_services:
  UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
  WHERE service_name = 'HSTGR_N8N_MCP_TOKEN';

Note: If the n8n UI does not expose an MCP token, the token is configured in the community node settings. Check n8n/mcp-server community node settings in the n8n interface.

Log entry: | <date> HH:MM UTC | — | HSTGR_N8N_MCP_TOKEN | scheduled | radieu | yes |

4. OVH API Keys (OVH_APP_KEY / OVH_APP_SECRET / OVH_CONSUMER_KEY)

What they are: OVH API credentials used by cost-exporter to query OVH Kimsufi billing data and by Ansible/scripts to manage OVH servers.
Location: .env.local only (not in GH Secrets — not used by CI)
Frequency: 365 days (or on suspected exposure)

Step 1 — log into OVH API portal:
  https://eu.api.ovh.com/createApp/

Step 2 — create new application:
  Application name: p24-infra-monitoring-<year>
  Description: Billing + server monitoring for p24-infra
  → copies OVH_APP_KEY + OVH_APP_SECRET

Step 3 — generate consumer key (auth token):
  https://eu.api.ovh.com/createToken/
  Rights needed:
    GET /order/upgrade/*
    GET /dedicated/server/*
    GET /me/bill/*
  Validity: unlimited
  → copies OVH_CONSUMER_KEY

Step 4 — update .env.local on dev workstation:
  OVH_APP_KEY=<new>
  OVH_APP_SECRET=<new>
  OVH_CONSUMER_KEY=<new>

Step 5 — restart cost-exporter to pick up new credentials:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart cost-exporter"
  # Check logs:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose logs --tail=20 cost-exporter"

Step 6 — revoke old application from OVH console if desired (not required — token is independent of app key)

Step 7 — update dev_r_services:
  UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
  WHERE service_name = 'OVH_APP_KEY';

Log entry: | <date> HH:MM UTC | — | OVH_APP_KEY OVH_APP_SECRET OVH_CONSUMER_KEY | scheduled | radieu | yes |

5. SMTP_PASSWORD (Mailgun EU)

What it is: SMTP password for Alertmanager email delivery. Mailgun does not expire SMTP passwords, but rotation keeps the exposure window bounded.
Location: GH Secret SMTP_PASSWORD · secrets/monitoring.env.sops · /opt/p24-infra/monitoring/.env on vps-i1
Frequency: 365 days

Step 1 — log into Mailgun EU:
  https://app.eu.mailgun.com → Sending → Domains → <domain> → SMTP credentials

Step 2 — locate the SMTP user (check SMTP_USER in .env.local for the username)
  Click "Reset password" → copy new password

Step 3 — update SOPS file:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops
# Replace SMTP_PASSWORD line:
$updated = $plain | ForEach-Object {
    if ($_ -match '^SMTP_PASSWORD=') { "SMTP_PASSWORD=<new>" } else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\mon-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\mon-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\monitoring.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\mon-edit.env.sops -Force
Step 4 — update GH Secret:
  gh secret set SMTP_PASSWORD --body "<new>" --repo radieu/p24-infra

Step 5 — deploy to vps-i1 via secrets-sync.yml (merge to dev/main triggers auto-deploy)
  Or manually: ssh root@217.154.82.162 "sed -i 's/SMTP_PASSWORD=.*/SMTP_PASSWORD=<new>/' /opt/p24-infra/monitoring/.env"

Step 6 — restart Alertmanager:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart alertmanager"

Step 7 — send a test alert to verify delivery:
  ssh root@217.154.82.162 "curl -s -X POST http://localhost:9093/api/v2/alerts \
    -H 'Content-Type: application/json' \
    -d '[{\"labels\":{\"alertname\":\"TestAlert\",\"severity\":\"info\"},\"annotations\":{\"summary\":\"SMTP rotation test\"}}]'"
  # Check radieu@gmail.com inbox — alert email should arrive within 5 minutes

Step 8 — update .env.local on dev workstation

Step 9 — update dev_r_services:
  UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
  WHERE service_name = 'SMTP_PASSWORD';

Log entry: | <date> HH:MM UTC | — | SMTP_PASSWORD | scheduled | radieu | yes |

6. VPS_ROOT_SSH_KEY (root SSH key)

What it is: The ed25519 key pair used for root access to vps-i1 (217.154.82.162), vps-h1 (72.60.32.61), and all BMS bare-metal servers. Stored locally at C:\Users\konar\.ssh\id_ed25519.
Location: C:\Users\konar\.ssh\id_ed25519 (private) · /root/.ssh/authorized_keys on all servers · GH Secret VPS_ROOT_SSH_KEY (base64)
Frequency: 365 days

# Step 1 — generate new key pair
ssh-keygen -t ed25519 -f C:\Users\konar\.ssh\id_ed25519_new -C "radieu-root-$(Get-Date -Format 'yyyy-MM-dd')" -N ""
 
# Step 2 — add new public key to all servers while old key still works
$newPub = Get-Content "C:\Users\konar\.ssh\id_ed25519_new.pub"
$servers = @(
    @{host="217.154.82.162"; user="root"},   # vps-i1
    @{host="72.60.32.61";    user="root"},   # vps-h1
    @{host="94.23.26.113";   user="root"},   # bms-1
    @{host="145.239.133.104"; user="ubuntu"}, # bms-2
    @{host="51.68.155.224";  user="ubuntu"}, # bms-3
    @{host="54.36.123.110";  user="root"}    # bms-4
)
foreach ($s in $servers) {
    Write-Host "Adding to $($s.user)@$($s.host)"
    ssh -i C:\Users\konar\.ssh\id_ed25519 "$($s.user)@$($s.host)" `
        "echo '$newPub' >> ~/.ssh/authorized_keys && sort -u ~/.ssh/authorized_keys -o ~/.ssh/authorized_keys"
}
 
# Step 3 — verify new key works on each server
foreach ($s in $servers) {
    $result = ssh -i C:\Users\konar\.ssh\id_ed25519_new -o BatchMode=yes -o ConnectTimeout=5 `
        "$($s.user)@$($s.host)" "echo ok" 2>&1
    Write-Host "$($s.host): $result"
}
 
# Step 4 — remove old public key from all servers
$oldPub = Get-Content "C:\Users\konar\.ssh\id_ed25519.pub"
foreach ($s in $servers) {
    ssh -i C:\Users\konar\.ssh\id_ed25519_new "$($s.user)@$($s.host)" `
        "grep -v '$oldPub' ~/.ssh/authorized_keys | sponge ~/.ssh/authorized_keys 2>/dev/null || \
         grep -v '$oldPub' ~/.ssh/authorized_keys > /tmp/ak_tmp && mv /tmp/ak_tmp ~/.ssh/authorized_keys"
}
 
# Step 5 — replace local key files
Move-Item C:\Users\konar\.ssh\id_ed25519_new     C:\Users\konar\.ssh\id_ed25519     -Force
Move-Item C:\Users\konar\.ssh\id_ed25519_new.pub C:\Users\konar\.ssh\id_ed25519.pub -Force
 
# Step 6 — update GH Secret VPS_ROOT_SSH_KEY (base64-encoded private key)
$keyB64 = [Convert]::ToBase64String([IO.File]::ReadAllBytes("C:\Users\konar\.ssh\id_ed25519"))
gh secret set VPS_ROOT_SSH_KEY --body $keyB64 --repo radieu/p24-infra
 
# Step 7 — verify credential-rotation.yml SSH steps work (dry run)
gh workflow run credential-rotation.yml --repo radieu/p24-infra -f dry_run=true
 
# Step 8 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
# WHERE service_name = 'SSH_KEY_ROOT';

Escalation: If you get locked out of a server during rotation (step 4 removes old key before new key is confirmed), use the Hostinger/OVH/IONOS console (web-based terminal) to restore access. For vps-h1, log into Hostinger control panel → VNC console.

Log entry: | <date> HH:MM UTC | — | SSH_KEY_ROOT (all servers) | scheduled | radieu | yes |

7. CLOUDFLARE_TOKEN_ZINTEGROWANA (DNS-scoped token — auto-rotated, manual fallback)

What it is: Zone-scoped Cloudflare API token with Zone:DNS:Edit permission for the zintegrowana.online zone. Used by dns-manager.py, provision-new-vps.yml, and the waha-router deployment workflow. Auto-rotation: credential-rotation.yml rolls this token automatically via the CF API (using CF_API_TOKEN which holds User:API Tokens:Edit scope). Use this procedure only if automation fails. Location: GH Secret CLOUDFLARE_TOKEN_ZINTEGROWANA · secrets/monitoring.env.sops · .env.local Frequency: 180 days (next_due 2026-12-12)

# Step 1 — open Cloudflare dashboard
# https://dash.cloudflare.com → Profile → API Tokens
# Find token named "CLOUDFLARE_TOKEN_ZINTEGROWANA" (or similar DNS-edit token)
# Click "Roll" — CF generates a new value while the token ID stays the same
 
# Step 2 — update SOPS file (monitoring scope)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops
$newToken = Read-Host "Paste new CLOUDFLARE_TOKEN_ZINTEGROWANA value"
$updated = $plain | ForEach-Object {
    if ($_ -match '^CLOUDFLARE_TOKEN_ZINTEGROWANA=') { "CLOUDFLARE_TOKEN_ZINTEGROWANA=$newToken" } else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\mon-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\mon-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\monitoring.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\mon-edit.env.sops -Force
 
# Step 3 — update GH Secret
gh secret set CLOUDFLARE_TOKEN_ZINTEGROWANA --body "$newToken" --repo radieu/p24-infra
 
# Step 4 — update .env.local on dev workstation
# Edit d:\code_2026\p24-infra\.env.local — replace CLOUDFLARE_TOKEN_ZINTEGROWANA=<old> line
 
# Step 5 — verify dns-manager.py works with new token
# On vps-i1 (picks up new .env after secrets-sync.yml deploys):
ssh root@217.154.82.162 "cd /opt/p24-infra && python3 scripts/dns-manager.py list 2>&1 | tail -5"
 
# Step 6 — commit + PR → merge to main → secrets-sync.yml auto-deploys to vps-i1
 
# Step 7 — verify monitoring/.env on vps-i1 has the new value (after merge):
ssh root@217.154.82.162 "grep CLOUDFLARE_TOKEN_ZINTEGROWANA /opt/p24-infra/monitoring/.env | cut -c1-40"
# Should show the start of the new token (never the full value)
 
# Step 8 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'CLOUDFLARE_TOKEN_ZINTEGROWANA';

Escalation: If dns-manager.py returns error 9109 (Invalid access token), the monitoring/.env still has the old value. Check that secrets-sync.yml ran after the PR merged. As emergency workaround, SSH to vps-i1 and manually edit /opt/p24-infra/monitoring/.env.

Log entry: | <date> HH:MM UTC | — | CLOUDFLARE_TOKEN_ZINTEGROWANA | scheduled | radieu | yes |

8. BMS4_N8N_API_KEY (bms-4 n8n API key — auto-rotated, manual fallback)

What it is: REST API key for the self-hosted n8n instance on bms-4. Used by monitoring workflows, cost-exporter, and the credential-rotation.yml workflow itself to create new API keys. Auto-rotation: credential-rotation.yml rotates this via the n8n REST API (POST /api/v1/credentials). Use this procedure only if automation fails. Location: GH Secret BMS4_N8N_API_KEY · secrets/n8n-bms4.env.sops · .env.local Frequency: 180 days

# Step 1 — log into n8n on bms-4
# https://n8n.bms-4.infra.zintegrowana.online → Settings → n8n API
 
# Step 2 — create a new API key
# Click "Create API Key" → copy the new key
# Leave both the old and new key active until all consumers are updated
 
# Step 3 — update SOPS file (n8n-bms4 scope)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\n8n-bms4.env.sops
$newKey = Read-Host "Paste new BMS4_N8N_API_KEY value"
$updated = $plain | ForEach-Object {
    if ($_ -match '^BMS4_N8N_API_KEY=') { "BMS4_N8N_API_KEY=$newKey" } else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\n8n-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\n8n-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\n8n-bms4.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\n8n-edit.env.sops -Force
 
# Step 4 — update GH Secret
gh secret set BMS4_N8N_API_KEY --body "$newKey" --repo radieu/p24-infra
 
# Step 5 — update .env.local on dev workstation
 
# Step 6 — commit + PR → merge to main → secrets-sync.yml deploys to bms-4
 
# Step 7 — verify new key works
$apiKey = $newKey  # already in memory
Invoke-RestMethod -Uri "https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows" `
    -Headers @{ "X-N8N-API-KEY" = $apiKey } -Method Get | Select-Object -ExpandProperty count
# Should return the workflow count (>0)
 
# Step 8 — delete old API key from n8n UI
# Settings → n8n API → find old key → Delete
 
# Step 9 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'BMS4_N8N_API_KEY';

Escalation: If n8n API returns 401, the bms-4 .env has not yet received the new key. Check if secrets-sync.yml ran after the PR merged, or manually update: ssh root@54.36.123.110 "sed -i 's/BMS4_N8N_API_KEY=.*/BMS4_N8N_API_KEY=<new>/' /opt/p24-infra/bms-4/.env" then restart n8n workers.

Log entry: | <date> HH:MM UTC | — | BMS4_N8N_API_KEY | scheduled | radieu | yes |

9. WASABI_KEYS_P24_INFRA (Wasabi IAM key rotation — auto-rotated, manual fallback)

What it is: IAM access key pair for the p24-infra Wasabi user. Used by Thanos sidecar (long-term metrics storage), backup-exporter, cost-exporter, and pdf-service — all writing to the p24-infra bucket (eu-central-2). Auto-rotation: credential-rotation.yml rotates via boto3 iam.wasabisys.com (create_access_key → update all locations → delete_access_key). Use this procedure only if automation fails. Location: GH Secrets P24_INFRA_WASABI_ACCESS_KEY + P24_INFRA_WASABI_SECRET_KEY (and aliases WASABI_ACCESS_KEY / WASABI_SECRET_KEY) · secrets/monitoring.env.sops · /opt/p24-infra/monitoring/.env on vps-i1 Frequency: 180 days (next_due 2026-12-10)

# Step 1 — create new IAM key via boto3 (run from dev workstation with admin Wasabi key)
# Requires: WASABI_ADMIN_ACCESS_KEY + WASABI_ADMIN_SECRET_KEY in .env.local
 
import boto3, os
from dotenv import load_dotenv
load_dotenv("d:/code_2026/p24-infra/.env.local")
 
iam = boto3.client(
    "iam",
    endpoint_url="https://iam.wasabisys.com",
    aws_access_key_id=os.environ["WASABI_ADMIN_ACCESS_KEY"],
    aws_secret_access_key=os.environ["WASABI_ADMIN_SECRET_KEY"],
    region_name="us-east-1",
)
 
# List existing keys for p24-infra user
existing = iam.list_access_keys(UserName="p24-infra")["AccessKeyMetadata"]
print("Existing keys:", [(k["AccessKeyId"], k["Status"]) for k in existing])
 
# Create new key
new_key = iam.create_access_key(UserName="p24-infra")["AccessKey"]
print("New AccessKeyId:", new_key["AccessKeyId"])
# new_key["SecretAccessKey"] — store this IMMEDIATELY in SOPS before proceeding
# Step 2 — update SOPS (monitoring scope — stores both P24_INFRA and alias WASABI_ keys)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops
$newAccessKey = Read-Host "New P24_INFRA_WASABI_ACCESS_KEY"
$newSecretKey = Read-Host "New P24_INFRA_WASABI_SECRET_KEY"
$updated = $plain | ForEach-Object {
    if ($_ -match '^P24_INFRA_WASABI_ACCESS_KEY=') { "P24_INFRA_WASABI_ACCESS_KEY=$newAccessKey" }
    elseif ($_ -match '^P24_INFRA_WASABI_SECRET_KEY=') { "P24_INFRA_WASABI_SECRET_KEY=$newSecretKey" }
    elseif ($_ -match '^WASABI_ACCESS_KEY=') { "WASABI_ACCESS_KEY=$newAccessKey" }
    elseif ($_ -match '^WASABI_SECRET_KEY=') { "WASABI_SECRET_KEY=$newSecretKey" }
    else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\mon-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\mon-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\monitoring.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\mon-edit.env.sops -Force
 
# Step 3 — update GH Secrets (both canonical and alias names)
gh secret set P24_INFRA_WASABI_ACCESS_KEY --body "$newAccessKey" --repo radieu/p24-infra
gh secret set P24_INFRA_WASABI_SECRET_KEY --body "$newSecretKey" --repo radieu/p24-infra
gh secret set WASABI_ACCESS_KEY --body "$newAccessKey" --repo radieu/p24-infra
gh secret set WASABI_SECRET_KEY --body "$newSecretKey" --repo radieu/p24-infra
 
# Step 4 — push new keys to vps-i1 immediately (Thanos sidecar must not lose access)
ssh root@217.154.82.162 @"
sed -i 's/P24_INFRA_WASABI_ACCESS_KEY=.*/P24_INFRA_WASABI_ACCESS_KEY=$newAccessKey/' /opt/p24-infra/monitoring/.env
sed -i 's/P24_INFRA_WASABI_SECRET_KEY=.*/P24_INFRA_WASABI_SECRET_KEY=$newSecretKey/' /opt/p24-infra/monitoring/.env
sed -i 's/WASABI_ACCESS_KEY=.*/WASABI_ACCESS_KEY=$newAccessKey/' /opt/p24-infra/monitoring/.env
sed -i 's/WASABI_SECRET_KEY=.*/WASABI_SECRET_KEY=$newSecretKey/' /opt/p24-infra/monitoring/.env
"@
 
# Step 5 — restart all Wasabi-dependent containers
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart thanos-sidecar backup-exporter cost-exporter pdf-service"
 
# Step 6 — verify Thanos sidecar is uploading blocks
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose logs --tail=20 thanos-sidecar 2>&1 | grep -E '(upload|error|block)'"
# Should show "upload" lines, no auth errors
 
# Step 7 — verify backup-exporter connects to Wasabi
ssh root@217.154.82.162 "curl -s http://localhost:9220/metrics | grep wasabi_backup_last_success"
 
# Step 8 — delete old Wasabi IAM key (after verifying all containers work)
# In Python: iam.delete_access_key(UserName="p24-infra", AccessKeyId="<old_key_id>")
 
# Step 9 — commit SOPS + PR → merge to main
 
# Step 10 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'WASABI_KEYS_P24_INFRA';

Escalation: If Thanos sidecar logs show AccessDenied after container restart, the new key has not reached /opt/p24-infra/monitoring/.env. SSH to vps-i1 and grep the file directly. Do NOT delete the old key until Thanos is confirmed working — Wasabi IAM allows up to 2 active keys per user.

Log entry: | <date> HH:MM UTC | — | P24_INFRA_WASABI_ACCESS_KEY P24_INFRA_WASABI_SECRET_KEY | scheduled | radieu | yes |

11. VPS_SSH_PRIVATE_KEY (claude-admin SSH key — auto-rotated, manual fallback)

What it is: ed25519 key pair for the claude-admin user on vps-i1 and vps-h1. Used by CI/CD workflows (health-check.yml, secrets-sync.yml, credential-rotation.yml). Different from the root key (VPS_ROOT_SSH_KEY) — this key has passwordless sudo for: docker, systemctl, mkdir, chown, cp, tee. Auto-rotation: credential-rotation.yml generates a new ed25519 pair, pushes to authorized_keys on both VPSes, updates GH Secret. Use this procedure only if automation fails. Location: SOPS secrets/monitoring.env.sops key VPS_SSH_PRIVATE_KEY (base64-encoded, single line, no PEM headers) is the source of truth. Two mechanisms keep the GH Secret in sync with it: the auto-rotator (scripts/rotate-credentials.py’s rotate_ssh_key_claude_admin) writes both SOPS and the GH Secret in the same run, and secrets-sync.yml’s sync-gh-secrets job separately propagates SOPS → GH Secret (VPS_SSH_PRIVATE_KEY was added to its allowlist in 4916). For a manual rotation, write SOPS via scripts/sops-set.ps1 — never gh secret set directly — for the same reason: a manual gh secret set with no matching SOPS write gets silently reverted by the next scheduled sync (see #4902, #2725). Frequency: 365 days

⚠️ authorized_keys immutability (discovered #2725, 2026-08-01): /home/claude-admin/.ssh/authorized_keys on vps-i1 has the Linux immutable attribute set (chattr +i). Any edit — append or removal — fails silently/permission-denied unless you chattr -i first, and the file must be relocked with chattr +i in the same operation window (never left unlocked). Confirm/set: lsattr /home/claude-admin/.ssh/authorized_keys should show ----i---------. This is deliberate hardening against unauthorized key injection — it is not documented elsewhere, so do not assume a plain >> append will work on this file.

# `;` between every step (not `&&`) so the relock ALWAYS runs, even if an earlier
# step fails — an unlocked authorized_keys must never survive this command.
# `printf '\n%s\n'` (not `echo ... >>`) guards against a missing trailing newline
# on the existing file concatenating onto the new key and corrupting both entries.
ssh root@217.154.82.162 "chattr -i /home/claude-admin/.ssh/authorized_keys; \
  printf '\n%s\n' '<new_pub_key_line>' >> /home/claude-admin/.ssh/authorized_keys; \
  sort -u -o /home/claude-admin/.ssh/authorized_keys /home/claude-admin/.ssh/authorized_keys; \
  chattr +i /home/claude-admin/.ssh/authorized_keys; \
  lsattr /home/claude-admin/.ssh/authorized_keys"

This same immutability should be assumed for authorized_keys on other hardened hosts unless verified otherwise with lsattr first.

# Step 1 — generate new ed25519 key pair
$keyPath = "C:\Users\konar\.ssh\claude_admin_new"
ssh-keygen -t ed25519 -f $keyPath -C "claude-admin-$(Get-Date -Format 'yyyy-MM-dd')" -N ""
$newPub = Get-Content "$keyPath.pub"
 
# Step 2 — add new public key to claude-admin authorized_keys on both VPSes
# (vps-i1 and vps-h1 only — claude-admin does not exist on BMS servers)
foreach ($host in @("217.154.82.162", "72.60.32.61")) {
    # Use root key to add to claude-admin's authorized_keys
    ssh -i C:\Users\konar\.ssh\id_ed25519 root@$host `
        "echo '$newPub' >> /home/claude-admin/.ssh/authorized_keys && sort -u /home/claude-admin/.ssh/authorized_keys -o /home/claude-admin/.ssh/authorized_keys"
    Write-Host "$host: new key added"
}
 
# Step 3 — verify new key works on both VPSes
foreach ($host in @("217.154.82.162", "72.60.32.61")) {
    $result = ssh -i $keyPath -o BatchMode=yes -o ConnectTimeout=5 claude-admin@$host "echo ok" 2>&1
    Write-Host "$host: $result"
}
 
# Step 4 — update GH Secret VPS_SSH_PRIVATE_KEY
$privateKeyContent = Get-Content $keyPath -Raw
gh secret set VPS_SSH_PRIVATE_KEY --body $privateKeyContent --repo radieu/p24-infra
 
# Step 5 — verify CI workflows can use the new key (trigger health-check)
gh workflow run health-check.yml --repo radieu/p24-infra
Start-Sleep -Seconds 30
gh run list --repo radieu/p24-infra --workflow health-check.yml --limit 1
 
# Step 6 — get old public key fingerprint, then remove old key from authorized_keys
# (Use the previous GH Secret value's public key to identify and remove it)
# Get old pub key from old private key if still on disk, otherwise check authorized_keys
foreach ($host in @("217.154.82.162", "72.60.32.61")) {
    # Leaves only keys added AFTER the old claude-admin key — human review recommended
    Write-Host "Review and manually remove old key from $host:/home/claude-admin/.ssh/authorized_keys"
    ssh -i $keyPath claude-admin@$host "cat ~/.ssh/authorized_keys"
}
 
# Step 7 — clean up local temp key files
Remove-Item $keyPath, "$keyPath.pub" -Force
 
# Step 8 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
# WHERE service_name = 'SSH_KEY_CLAUDE_ADMIN';

Note on BMS servers: The claude-admin user does not exist on bms-1, bms-2, bms-3, or bms-4. Those servers use root (bms-1, bms-4) or ubuntu (bms-2, bms-3) with the root SSH key (VPS_ROOT_SSH_KEY). See §6 for root key rotation.

Escalation: If CI workflows fail with Permission denied (publickey) after updating the GH Secret, check that the authorized_keys file still has the new public key (not accidentally deleted). Use the root key to SSH in and verify: ssh root@217.154.82.162 "cat /home/claude-admin/.ssh/authorized_keys".

Log entry: | <date> HH:MM UTC | — | VPS_SSH_PRIVATE_KEY (claude-admin) | scheduled | radieu | yes |

12. SENTRY_AUTH_TOKEN (Sentry source-maps token — et-oper-plat)

What it is: Sentry User Auth Token for source map uploads during Vercel builds of et-operational-platform.
Location: secrets/monitoring.env.sops · Vercel env vars for et-operational-platform
Frequency: 90 days · Next due: 2026-08-01 · Tracked: #1530
Auto-rotate: NO — Sentry sntryu_ tokens cannot be created via API; UI-only.
Detailed playbook: docs/playbooks/sentry-token-rotation.md

Verifying automated rotations after credential-rotation.yml runs

After the Monday 06:00 UTC run, verify these services are healthy:

# 1. Grafana still accessible
curl -s -o /dev/null -w "%{http_code}" https://grafana.vps-i1.infra.zintegrowana.online/login
# Expected: 200
 
# 2. Prometheus still scraping all targets
ssh root@217.154.82.162 "curl -s http://localhost:9090/api/v1/targets | python3 -c 'import json,sys; t=json.load(sys.stdin)[\"data\"][\"activeTargets\"]; print(\"up:\", sum(1 for x in t if x[\"health\"]==\"up\"), \"/\", len(t))'"
 
# 3. Check credential-rotation.yml last run
gh run list --repo radieu/p24-infra --workflow credential-rotation.yml --limit 1

After any rotation — standard checklist

[ ] New value in SOPS (secrets/*.env.sops) — committed and pushed
[ ] New value in GH Secret (gh secret set NAME -R radieu/p24-infra)
[ ] New value in .env.local on dev workstation
[ ] New value deployed to relevant VPS .env (via secrets-sync.yml or SSH)
[ ] Affected service restarted and verified working
[ ] last_rotated + next_due updated in dev_r_services
[ ] Row appended to docs/secrets-rotation-log.md
[ ] Old credential revoked at the issuing service

Graph View

Backlinks