p24-infra Docs

credential-rotation-180d

21 min read

Playbook: 180-Day Credential Rotation (Manual)

Trigger: credential-rotation.yml (Monday 06:00 UTC) opens a GitHub issue with label human-action when next_due < today for any credential with auto_rotate=false. This playbook covers the manual rotation procedures for those credentials.

Automation coverage summary

Before doing anything manually, check whether credential-rotation.yml already handles the credential:

CredentialAuto-rotate?Script action
GRAFANA_ADMIN_PASSWORDYESbcrypt hash updated in Caddyfile; Grafana + Caddy restarted — but see note below
SUPABASE_GRAFANA_PASSWORDYESALTER ROLE grafana_readonly; Grafana restarted
EMAIL_SENDER_API_KEYYESCF worker key rotated via Cloudflare API
CLOUDFLARE_TOKEN_ZINTEGROWANAYESToken rolled via PUT /v1/user/tokens/{id}/value using global key
WASABI_KEYS_P24_INFRAYESNew IAM key created; old deleted; all consumers updated + restarted
BMS4_N8N_API_KEYYESNew key via n8n REST API; GH Secret + SOPS updated
SSH_KEY_CLAUDE_ADMINYESNew ed25519 pair; authorized_keys on all VPSes updated; GH Secret updated
MYSQL_PASSWORDYESALTER USER in Traccar MySQL; .env updated; container restarted
PDF_SERVICE_API_KEYYES (#1503)Random hex; vps-i1 .env + SOPS updated; pdf-service recreated
GF_RENDERING_RENDERER_TOKENYES (#1503)Random hex; vps-i1 .env + SOPS; grafana + grafana-image-renderer recreated
REDIS_PASSWORDYES (#1503)bms-4 .env + SOPS; redis + all n8n services recreated (BullMQ jobs re-queue on reconnect). ALSO update the n8n stored Redis credential xjD48jTskJlCyRY0 (“bms-4 queue Redis (w4 singleton lock)”) — the BullMQ env var is not the only consumer; the W4 Direct API Ingestion workflow’s singleton-lock Redis nodes authenticate with this separately-stored credential, which secrets-sync.yml does NOT touch. Missing it leaves the workflow WRONGPASS on the lock node while the queue itself is healthy (#6281). See the mandatory REDIS_PASSWORD note below (#6283) — full 5-consumer rotation + #4135 single-doc verification in n8n-bms4-redis-password-rotation.md; credential-only detail in n8n/n8n-bms4-redis-credential-rotation.md.
TRACCAR_PASSWORDYES (#1503)Traccar REST API (PUT /api/users/{id}) + SOPS
TRACCAR_FORWARD_TOKENYES (#1503)CF Worker traccar-gw FORWARD_TOKEN + Traccar /root/traccar/.env + SOPS
TRACCAR_GW_ADMIN_KEYYES (#1503)CF Worker traccar-gw ADMIN_KEY + SOPS
TRACCAR_ADMIN_KEYYES (#1503)SOPS only — no active runtime consumer
N8N_HU_SP_REPORT_SECRETYES (#1503)bms-4 .env + SOPS (n8n-bms4 + vps-h1); n8n services recreated
GITHUB_TRIGGER_SECRETYES (#1503)bms-4 + vps-h1 .env + SOPS; n8n recreated. No GH Secret (GitHub rejects GITHUB_ prefix)
CRON_SECRETYES (#1503)Vercel env (et-operational-platform) + SOPS
CLAUDE_SESSION_MANAGER_KEYYES (#1503)bms-4 .env + SOPS; claude-session-manager restarted
N8N_GPS_SYNC_SECRETNO (#1503)n8n validator is a credential-vault entry (no API) — rotate n8n UI + Vercel + SOPS together

#1503 handlers are latent. Adding a handler does not start rotating a secret. Rotation only fires when the credential’s dev_r_services row has auto_rotate=true and next_due is reached (or FORCE_ALL=true / ONLY_SERVICES=<name> is set). Before flipping auto_rotate=true for any of the above, validate with a dry run (DRY_RUN=true ONLY_SERVICES=<name> python scripts/rotate-credentials.py) then a single staged live rotation.

⚠️ GRAFANA_ADMIN_PASSWORD — mandatory manual step after every rotation

Grafana stores its admin password in its own SQLite database (/var/lib/grafana/grafana.db). Updating GF_SECURITY_ADMIN_PASSWORD in .env and restarting the container does not update the stored password — it is only read on first boot when the DB is empty. Failing to run the CLI reset step means the dashboard login breaks after every rotation.

After credential-rotation.yml completes (or after any manual GRAFANA_ADMIN_PASSWORD change), always run:

ssh root@217.154.82.162 'PASS=$(grep "^GRAFANA_ADMIN_PASSWORD=" /opt/p24-infra/monitoring/.env | cut -d= -f2-) && docker exec monitoring-grafana-1 grafana cli admin reset-admin-password "$PASS" 2>&1 | tail -3; unset PASS'
# Expected last line: Admin password changed successfully ✔

Then update .env.local on the dev workstation to match (read from SOPS — never from the terminal):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$dec = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops 2>$null
$v = ($dec | Select-String "^GRAFANA_ADMIN_PASSWORD=").ToString().Split("=",2)[1]
$content = Get-Content "C:\code_2026\p24-infra\.env.local" -Raw
$content = $content -replace "(?m)^GRAFANA_ADMIN_PASSWORD=.*$", "GRAFANA_ADMIN_PASSWORD=$v"
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\.env.local", $content, [System.Text.UTF8Encoding]::new($false))
$dec = $null; $v = $null

Verify login works: open https://grafana.vps-i1.infra.zintegrowana.online → username admin → password from .env.local.

⚠️ REDIS_PASSWORD (bms-4) — mandatory n8n stored-credential step after every rotation

Same class of gap as GRAFANA above: rotating REDIS_PASSWORD updates SOPS → .env → docker-compose env → redis requirepass → n8n QUEUE_BULL_REDIS_PASSWORD, but it does not update the n8n stored credential xjD48jTskJlCyRY0 (“bms-4 queue Redis”, type n8n-nodes-base.redis). That credential lives in n8n’s encrypted PostgreSQL vault, not .env. If it is not pushed, the W4 Direct API Ingestion workflow gets WRONGPASS on every run and silently stops ingesting (the 2026-08-13 outage, #6283).

The stored-credential push is now a phase-2 entry in bms-4/sync-n8n-credentials.py, so secrets-sync.yml’s post-deploy credential sync updates it automatically. After any REDIS_PASSWORD rotation, verify (or push out-of-band) with:

ssh root@54.36.123.110 'cd /opt/p24-infra && python3 bms-4/sync-n8n-credentials.py --cred REDIS_PASSWORD'
# expect: Summary: {'created': 0, 'updated': 1, ...}

Then run the #4135 single-document ingest smoke test. Full procedure (all 5 consumers + verification): n8n-bms4-redis-password-rotation.md.

The credentials below are not auto-rotated — manual action required.

1. CF_API_TOKEN (Cloudflare broad-scope token)

What it is: Cloudflare API token with broader-than-DNS scope — used for Workers, R2, email routing, KV namespace management.
Location: GH Secret CF_API_TOKEN · .env.local
Frequency: 180 days
Rotation method: Dashboard only — this token has User:API Tokens:Edit permission (required to roll CLOUDFLARE_TOKEN_ZINTEGROWANA), so it cannot roll itself.

# Step 1 — open Cloudflare dashboard
# https://dash.cloudflare.com → Profile → API Tokens
 
# Step 2 — create a new token with identical permissions to the existing one
# Scopes needed:
#   Zone:DNS:Edit   (for dns-manager.py)
#   User:API Tokens:Edit  (required for credential-rotation.yml to roll CLOUDFLARE_TOKEN_ZINTEGROWANA)
#   Zone:Cache Purge, Workers Scripts:Edit, R2:Edit  (if used)
 
# Step 3 — update GH Secret and .env.local
gh secret set CF_API_TOKEN --body "<new_token>" --repo radieu/p24-infra
(gc d:\code_2026\p24-infra\.env.local) -replace "^CF_API_TOKEN=.*", "CF_API_TOKEN=<new_token>" |
  [System.IO.File]::WriteAllText("d:\code_2026\p24-infra\.env.local", ($_ -join "`n") + "`n",
    [System.Text.UTF8Encoding]::new($false))
 
# Step 4 — verify credential-rotation.yml can roll CLOUDFLARE_TOKEN_ZINTEGROWANA
# Trigger workflow_dispatch with dry_run=true:
gh workflow run credential-rotation.yml --repo radieu/p24-infra -f dry_run=true -f force_all=false
 
# Step 5 — delete old token from Cloudflare dashboard
 
# Step 6 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'CF_API_TOKEN';

Log entry: | <date> HH:MM UTC | — | CF_API_TOKEN | scheduled | radieu | yes |

2. HSTGR_N8N_MCP_TOKEN

What it is: MCP server authentication token for the n8n MCP server integration (Claude ↔ n8n). Community edition has no REST API for token management — must be rotated through the n8n UI.
Location: GH Secret HSTGR_N8N_MCP_TOKEN · .env.local · n8n credential manager
Frequency: 180 days

Step 1 — open n8n instance:
  https://n8n.vps-h1.infra.zintegrowana.online  (Hostinger)
  or https://n8n.bms-4.infra.zintegrowana.online  (bms-4)

Step 2 — navigate to:
  Settings → API → Regenerate API key  (for HSTGR_N8N_API_KEY)
  
  For MCP token specifically:
  Settings → Community Nodes → MCP Server → Regenerate token
  (path varies by n8n version — may be under Settings → n8n API)

Step 3 — copy new token

Step 4 — update GH Secret:
  gh secret set HSTGR_N8N_MCP_TOKEN --body "<new>" --repo radieu/p24-infra

Step 5 — update .env.local on dev workstation:
  HSTGR_N8N_MCP_TOKEN=<new>

Step 6 — restart any Claude MCP session that uses n8n integration (session re-reads token)

Step 7 — verify: trigger a Claude ↔ n8n tool call and confirm it succeeds

Step 8 — update dev_r_services:
  UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
  WHERE service_name = 'HSTGR_N8N_MCP_TOKEN';

Note: If the n8n UI does not expose an MCP token, the token is configured in the community node settings. Check n8n/mcp-server community node settings in the n8n interface.

Log entry: | <date> HH:MM UTC | — | HSTGR_N8N_MCP_TOKEN | scheduled | radieu | yes |

4. OVH API Keys (OVH_APP_KEY / OVH_APP_SECRET / OVH_CONSUMER_KEY)

What they are: OVH API credentials used by cost-exporter to query OVH Kimsufi billing data and by Ansible/scripts to manage OVH servers.
Location: .env.local only (not in GH Secrets — not used by CI)
Frequency: 365 days (or on suspected exposure)

Step 1 — log into OVH API portal:
  https://eu.api.ovh.com/createApp/

Step 2 — create new application:
  Application name: p24-infra-monitoring-<year>
  Description: Billing + server monitoring for p24-infra
  → copies OVH_APP_KEY + OVH_APP_SECRET

Step 3 — generate consumer key (auth token):
  https://eu.api.ovh.com/createToken/
  Rights needed:
    GET /order/upgrade/*
    GET /dedicated/server/*
    GET /me/bill/*
  Validity: unlimited
  → copies OVH_CONSUMER_KEY

Step 4 — update .env.local on dev workstation:
  OVH_APP_KEY=<new>
  OVH_APP_SECRET=<new>
  OVH_CONSUMER_KEY=<new>

Step 5 — restart cost-exporter to pick up new credentials:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart cost-exporter"
  # Check logs:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose logs --tail=20 cost-exporter"

Step 6 — revoke old application from OVH console if desired (not required — token is independent of app key)

Step 7 — update dev_r_services:
  UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
  WHERE service_name = 'OVH_APP_KEY';

Log entry: | <date> HH:MM UTC | — | OVH_APP_KEY OVH_APP_SECRET OVH_CONSUMER_KEY | scheduled | radieu | yes |

5. SMTP_PASSWORD (Mailgun EU)

What it is: SMTP password for Alertmanager email delivery. Mailgun does not expire SMTP passwords, but rotation keeps the exposure window bounded.
Location: GH Secret SMTP_PASSWORD · secrets/monitoring.env.sops · /opt/p24-infra/monitoring/.env on vps-i1
Frequency: 365 days

Step 1 — log into Mailgun EU:
  https://app.eu.mailgun.com → Sending → Domains → <domain> → SMTP credentials

Step 2 — locate the SMTP user (check SMTP_USER in .env.local for the username)
  Click "Reset password" → copy new password

Step 3 — update SOPS file:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops
# Replace SMTP_PASSWORD line:
$updated = $plain | ForEach-Object {
    if ($_ -match '^SMTP_PASSWORD=') { "SMTP_PASSWORD=<new>" } else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\mon-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\mon-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\monitoring.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\mon-edit.env.sops -Force
Step 4 — update GH Secret:
  gh secret set SMTP_PASSWORD --body "<new>" --repo radieu/p24-infra

Step 5 — deploy to vps-i1 via secrets-sync.yml (merge to dev/main triggers auto-deploy)
  Or manually: ssh root@217.154.82.162 "sed -i 's/SMTP_PASSWORD=.*/SMTP_PASSWORD=<new>/' /opt/p24-infra/monitoring/.env"

Step 6 — restart Alertmanager:
  ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart alertmanager"

Step 7 — send a test alert to verify delivery:
  ssh root@217.154.82.162 "curl -s -X POST http://localhost:9093/api/v2/alerts \
    -H 'Content-Type: application/json' \
    -d '[{\"labels\":{\"alertname\":\"TestAlert\",\"severity\":\"info\"},\"annotations\":{\"summary\":\"SMTP rotation test\"}}]'"
  # Check radieu@gmail.com inbox — alert email should arrive within 5 minutes

Step 8 — update .env.local on dev workstation

Step 9 — update dev_r_services:
  UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
  WHERE service_name = 'SMTP_PASSWORD';

Log entry: | <date> HH:MM UTC | — | SMTP_PASSWORD | scheduled | radieu | yes |

6. VPS_ROOT_SSH_KEY (root SSH key)

What it is: The ed25519 key pair used for root access to vps-i1 (217.154.82.162), vps-h1 (72.60.32.61), and all BMS bare-metal servers. Stored locally at C:\Users\konar\.ssh\id_ed25519.
Location: C:\Users\konar\.ssh\id_ed25519 (private) · /root/.ssh/authorized_keys on all servers · GH Secret VPS_ROOT_SSH_KEY (base64)
Frequency: 365 days

# Step 1 — generate new key pair
ssh-keygen -t ed25519 -f C:\Users\konar\.ssh\id_ed25519_new -C "radieu-root-$(Get-Date -Format 'yyyy-MM-dd')" -N ""
 
# Step 2 — add new public key to all servers while old key still works
$newPub = Get-Content "C:\Users\konar\.ssh\id_ed25519_new.pub"
$servers = @(
    @{host="217.154.82.162"; user="root"},   # vps-i1
    @{host="72.60.32.61";    user="root"},   # vps-h1
    @{host="94.23.26.113";   user="root"},   # bms-1
    @{host="145.239.133.104"; user="ubuntu"}, # bms-2
    @{host="51.68.155.224";  user="ubuntu"}, # bms-3
    @{host="54.36.123.110";  user="root"}    # bms-4
)
foreach ($s in $servers) {
    Write-Host "Adding to $($s.user)@$($s.host)"
    ssh -i C:\Users\konar\.ssh\id_ed25519 "$($s.user)@$($s.host)" `
        "echo '$newPub' >> ~/.ssh/authorized_keys && sort -u ~/.ssh/authorized_keys -o ~/.ssh/authorized_keys"
}
 
# Step 3 — verify new key works on each server
foreach ($s in $servers) {
    $result = ssh -i C:\Users\konar\.ssh\id_ed25519_new -o BatchMode=yes -o ConnectTimeout=5 `
        "$($s.user)@$($s.host)" "echo ok" 2>&1
    Write-Host "$($s.host): $result"
}
 
# Step 4 — remove old public key from all servers
$oldPub = Get-Content "C:\Users\konar\.ssh\id_ed25519.pub"
foreach ($s in $servers) {
    ssh -i C:\Users\konar\.ssh\id_ed25519_new "$($s.user)@$($s.host)" `
        "grep -v '$oldPub' ~/.ssh/authorized_keys | sponge ~/.ssh/authorized_keys 2>/dev/null || \
         grep -v '$oldPub' ~/.ssh/authorized_keys > /tmp/ak_tmp && mv /tmp/ak_tmp ~/.ssh/authorized_keys"
}
 
# Step 5 — replace local key files
Move-Item C:\Users\konar\.ssh\id_ed25519_new     C:\Users\konar\.ssh\id_ed25519     -Force
Move-Item C:\Users\konar\.ssh\id_ed25519_new.pub C:\Users\konar\.ssh\id_ed25519.pub -Force
 
# Step 6 — update GH Secret VPS_ROOT_SSH_KEY (base64-encoded private key)
$keyB64 = [Convert]::ToBase64String([IO.File]::ReadAllBytes("C:\Users\konar\.ssh\id_ed25519"))
gh secret set VPS_ROOT_SSH_KEY --body $keyB64 --repo radieu/p24-infra
 
# Step 7 — verify credential-rotation.yml SSH steps work (dry run)
gh workflow run credential-rotation.yml --repo radieu/p24-infra -f dry_run=true
 
# Step 8 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
# WHERE service_name = 'SSH_KEY_ROOT';

Escalation: If you get locked out of a server during rotation (step 4 removes old key before new key is confirmed), use the Hostinger/OVH/IONOS console (web-based terminal) to restore access. For vps-h1, log into Hostinger control panel → VNC console.

Log entry: | <date> HH:MM UTC | — | SSH_KEY_ROOT (all servers) | scheduled | radieu | yes |

7. CLOUDFLARE_TOKEN_ZINTEGROWANA (DNS-scoped token — auto-rotated, manual fallback)

What it is: Zone-scoped Cloudflare API token with Zone:DNS:Edit permission for the zintegrowana.online zone. Used by dns-manager.py, provision-new-vps.yml, and the waha-router deployment workflow. Auto-rotation: credential-rotation.yml rolls this token automatically via the CF API (using CF_API_TOKEN which holds User:API Tokens:Edit scope). Use this procedure only if automation fails. Location: GH Secret CLOUDFLARE_TOKEN_ZINTEGROWANA · secrets/monitoring.env.sops · .env.local Frequency: 180 days (next_due 2026-12-12)

# Step 1 — open Cloudflare dashboard
# https://dash.cloudflare.com → Profile → API Tokens
# Find token named "CLOUDFLARE_TOKEN_ZINTEGROWANA" (or similar DNS-edit token)
# Click "Roll" — CF generates a new value while the token ID stays the same
 
# Step 2 — update SOPS file (monitoring scope)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops
$newToken = Read-Host "Paste new CLOUDFLARE_TOKEN_ZINTEGROWANA value"
$updated = $plain | ForEach-Object {
    if ($_ -match '^CLOUDFLARE_TOKEN_ZINTEGROWANA=') { "CLOUDFLARE_TOKEN_ZINTEGROWANA=$newToken" } else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\mon-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\mon-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\monitoring.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\mon-edit.env.sops -Force
 
# Step 3 — update GH Secret
gh secret set CLOUDFLARE_TOKEN_ZINTEGROWANA --body "$newToken" --repo radieu/p24-infra
 
# Step 4 — update .env.local on dev workstation
# Edit d:\code_2026\p24-infra\.env.local — replace CLOUDFLARE_TOKEN_ZINTEGROWANA=<old> line
 
# Step 5 — verify dns-manager.py works with new token
# On vps-i1 (picks up new .env after secrets-sync.yml deploys):
ssh root@217.154.82.162 "cd /opt/p24-infra && python3 scripts/dns-manager.py list 2>&1 | tail -5"
 
# Step 6 — commit + PR → merge to main → secrets-sync.yml auto-deploys to vps-i1
 
# Step 7 — verify monitoring/.env on vps-i1 has the new value (after merge):
ssh root@217.154.82.162 "grep CLOUDFLARE_TOKEN_ZINTEGROWANA /opt/p24-infra/monitoring/.env | cut -c1-40"
# Should show the start of the new token (never the full value)
 
# Step 8 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'CLOUDFLARE_TOKEN_ZINTEGROWANA';

Escalation: If dns-manager.py returns error 9109 (Invalid access token), the monitoring/.env still has the old value. Check that secrets-sync.yml ran after the PR merged. As emergency workaround, SSH to vps-i1 and manually edit /opt/p24-infra/monitoring/.env.

Log entry: | <date> HH:MM UTC | — | CLOUDFLARE_TOKEN_ZINTEGROWANA | scheduled | radieu | yes |

8. BMS4_N8N_API_KEY (bms-4 n8n API key — auto-rotated, manual fallback)

What it is: REST API key for the self-hosted n8n instance on bms-4. Used by monitoring workflows, cost-exporter, and the credential-rotation.yml workflow itself to create new API keys. Auto-rotation: credential-rotation.yml rotates this via the n8n REST API (POST /api/v1/credentials). Use this procedure only if automation fails. Location: GH Secret BMS4_N8N_API_KEY · secrets/n8n-bms4.env.sops · .env.local Frequency: 180 days

# Step 1 — log into n8n on bms-4
# https://n8n.bms-4.infra.zintegrowana.online → Settings → n8n API
 
# Step 2 — create a new API key
# Click "Create API Key" → copy the new key
# Leave both the old and new key active until all consumers are updated
 
# Step 3 — update SOPS file (n8n-bms4 scope)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\n8n-bms4.env.sops
$newKey = Read-Host "Paste new BMS4_N8N_API_KEY value"
$updated = $plain | ForEach-Object {
    if ($_ -match '^BMS4_N8N_API_KEY=') { "BMS4_N8N_API_KEY=$newKey" } else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\n8n-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\n8n-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\n8n-bms4.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\n8n-edit.env.sops -Force
 
# Step 4 — update GH Secret
gh secret set BMS4_N8N_API_KEY --body "$newKey" --repo radieu/p24-infra
 
# Step 5 — update .env.local on dev workstation
 
# Step 6 — commit + PR → merge to main → secrets-sync.yml deploys to bms-4
 
# Step 7 — verify new key works
$apiKey = $newKey  # already in memory
Invoke-RestMethod -Uri "https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows" `
    -Headers @{ "X-N8N-API-KEY" = $apiKey } -Method Get | Select-Object -ExpandProperty count
# Should return the workflow count (>0)
 
# Step 8 — delete old API key from n8n UI
# Settings → n8n API → find old key → Delete
 
# Step 9 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'BMS4_N8N_API_KEY';

Escalation: If n8n API returns 401, the bms-4 .env has not yet received the new key. Check if secrets-sync.yml ran after the PR merged, or manually update: ssh root@54.36.123.110 "sed -i 's/BMS4_N8N_API_KEY=.*/BMS4_N8N_API_KEY=<new>/' /opt/p24-infra/bms-4/.env" then restart n8n workers.

Log entry: | <date> HH:MM UTC | — | BMS4_N8N_API_KEY | scheduled | radieu | yes |

9. WASABI_KEYS_P24_INFRA (Wasabi IAM key rotation — auto-rotated, manual fallback)

What it is: IAM access key pair for the p24-infra Wasabi user. Used by Thanos sidecar (long-term metrics storage), backup-exporter, cost-exporter, and pdf-service — all writing to the p24-infra bucket (eu-central-2). Auto-rotation: credential-rotation.yml rotates via boto3 iam.wasabisys.com (create_access_key → update all locations → delete_access_key). Use this procedure only if automation fails. Location: GH Secrets P24_INFRA_WASABI_ACCESS_KEY + P24_INFRA_WASABI_SECRET_KEY (and aliases WASABI_ACCESS_KEY / WASABI_SECRET_KEY) · secrets/monitoring.env.sops · /opt/p24-infra/monitoring/.env on vps-i1 Frequency: 180 days (next_due 2026-12-10)

# Step 1 — create new IAM key via boto3 (run from dev workstation with admin Wasabi key)
# Requires: WASABI_ADMIN_ACCESS_KEY + WASABI_ADMIN_SECRET_KEY in .env.local
 
import boto3, os
from dotenv import load_dotenv
load_dotenv("d:/code_2026/p24-infra/.env.local")
 
iam = boto3.client(
    "iam",
    endpoint_url="https://iam.wasabisys.com",
    aws_access_key_id=os.environ["WASABI_ADMIN_ACCESS_KEY"],
    aws_secret_access_key=os.environ["WASABI_ADMIN_SECRET_KEY"],
    region_name="us-east-1",
)
 
# List existing keys for p24-infra user
existing = iam.list_access_keys(UserName="p24-infra")["AccessKeyMetadata"]
print("Existing keys:", [(k["AccessKeyId"], k["Status"]) for k in existing])
 
# Create new key
new_key = iam.create_access_key(UserName="p24-infra")["AccessKey"]
print("New AccessKeyId:", new_key["AccessKeyId"])
# new_key["SecretAccessKey"] — store this IMMEDIATELY in SOPS before proceeding
# Step 2 — update SOPS (monitoring scope — stores both P24_INFRA and alias WASABI_ keys)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops
$newAccessKey = Read-Host "New P24_INFRA_WASABI_ACCESS_KEY"
$newSecretKey = Read-Host "New P24_INFRA_WASABI_SECRET_KEY"
$updated = $plain | ForEach-Object {
    if ($_ -match '^P24_INFRA_WASABI_ACCESS_KEY=') { "P24_INFRA_WASABI_ACCESS_KEY=$newAccessKey" }
    elseif ($_ -match '^P24_INFRA_WASABI_SECRET_KEY=') { "P24_INFRA_WASABI_SECRET_KEY=$newSecretKey" }
    elseif ($_ -match '^WASABI_ACCESS_KEY=') { "WASABI_ACCESS_KEY=$newAccessKey" }
    elseif ($_ -match '^WASABI_SECRET_KEY=') { "WASABI_SECRET_KEY=$newSecretKey" }
    else { $_ }
}
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\mon-edit.env.sops",
    ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\mon-edit.env.sops
[System.IO.File]::WriteAllText("C:\code_2026\p24-infra\secrets\monitoring.env.sops",
    ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item C:\code_2026\p24-infra\secrets\mon-edit.env.sops -Force
 
# Step 3 — update GH Secrets (both canonical and alias names)
gh secret set P24_INFRA_WASABI_ACCESS_KEY --body "$newAccessKey" --repo radieu/p24-infra
gh secret set P24_INFRA_WASABI_SECRET_KEY --body "$newSecretKey" --repo radieu/p24-infra
gh secret set WASABI_ACCESS_KEY --body "$newAccessKey" --repo radieu/p24-infra
gh secret set WASABI_SECRET_KEY --body "$newSecretKey" --repo radieu/p24-infra
 
# Step 4 — push new keys to vps-i1 immediately (Thanos sidecar must not lose access)
ssh root@217.154.82.162 @"
sed -i 's/P24_INFRA_WASABI_ACCESS_KEY=.*/P24_INFRA_WASABI_ACCESS_KEY=$newAccessKey/' /opt/p24-infra/monitoring/.env
sed -i 's/P24_INFRA_WASABI_SECRET_KEY=.*/P24_INFRA_WASABI_SECRET_KEY=$newSecretKey/' /opt/p24-infra/monitoring/.env
sed -i 's/WASABI_ACCESS_KEY=.*/WASABI_ACCESS_KEY=$newAccessKey/' /opt/p24-infra/monitoring/.env
sed -i 's/WASABI_SECRET_KEY=.*/WASABI_SECRET_KEY=$newSecretKey/' /opt/p24-infra/monitoring/.env
"@
 
# Step 5 — restart all Wasabi-dependent containers
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart thanos-sidecar backup-exporter cost-exporter pdf-service"
 
# Step 6 — verify Thanos sidecar is uploading blocks
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose logs --tail=20 thanos-sidecar 2>&1 | grep -E '(upload|error|block)'"
# Should show "upload" lines, no auth errors
 
# Step 7 — verify backup-exporter connects to Wasabi
ssh root@217.154.82.162 "curl -s http://localhost:9220/metrics | grep wasabi_backup_last_success"
 
# Step 8 — delete old Wasabi IAM key (after verifying all containers work)
# In Python: iam.delete_access_key(UserName="p24-infra", AccessKeyId="<old_key_id>")
 
# Step 9 — commit SOPS + PR → merge to main
 
# Step 10 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+180d>'
# WHERE service_name = 'WASABI_KEYS_P24_INFRA';

Escalation: If Thanos sidecar logs show AccessDenied after container restart, the new key has not reached /opt/p24-infra/monitoring/.env. SSH to vps-i1 and grep the file directly. Do NOT delete the old key until Thanos is confirmed working — Wasabi IAM allows up to 2 active keys per user.

Log entry: | <date> HH:MM UTC | — | P24_INFRA_WASABI_ACCESS_KEY P24_INFRA_WASABI_SECRET_KEY | scheduled | radieu | yes |

11. VPS_SSH_PRIVATE_KEY (claude-admin SSH key — auto-rotated, manual fallback)

What it is: ed25519 key pair for the claude-admin user on vps-i1 and vps-h1. Used by CI/CD workflows (health-check.yml, secrets-sync.yml, credential-rotation.yml). Different from the root key (VPS_ROOT_SSH_KEY) — this key has passwordless sudo for: docker, systemctl, mkdir, chown, cp, tee. Auto-rotation: credential-rotation.yml generates a new ed25519 pair, pushes to authorized_keys on both VPSes, updates GH Secret. Use this procedure only if automation fails. Location: SOPS secrets/monitoring.env.sops key VPS_SSH_PRIVATE_KEY (base64-encoded, single line, no PEM headers) is the source of truth. Two mechanisms keep the GH Secret in sync with it: the auto-rotator (scripts/rotate-credentials.py’s rotate_ssh_key_claude_admin) writes both SOPS and the GH Secret in the same run, and secrets-sync.yml’s sync-gh-secrets job separately propagates SOPS → GH Secret (VPS_SSH_PRIVATE_KEY was added to its allowlist in 4916). For a manual rotation, write SOPS via scripts/sops-set.ps1 — never gh secret set directly — for the same reason: a manual gh secret set with no matching SOPS write gets silently reverted by the next scheduled sync (see #4902, #2725). Frequency: 365 days

⚠️ authorized_keys immutability (discovered #2725, 2026-08-01): /home/claude-admin/.ssh/authorized_keys on vps-i1 has the Linux immutable attribute set (chattr +i). Any edit — append or removal — fails silently/permission-denied unless you chattr -i first, and the file must be relocked with chattr +i in the same operation window (never left unlocked). Confirm/set: lsattr /home/claude-admin/.ssh/authorized_keys should show ----i---------. This is deliberate hardening against unauthorized key injection — it is not documented elsewhere, so do not assume a plain >> append will work on this file.

# `;` between every step (not `&&`) so the relock ALWAYS runs, even if an earlier
# step fails — an unlocked authorized_keys must never survive this command.
# `printf '\n%s\n'` (not `echo ... >>`) guards against a missing trailing newline
# on the existing file concatenating onto the new key and corrupting both entries.
ssh root@217.154.82.162 "chattr -i /home/claude-admin/.ssh/authorized_keys; \
  printf '\n%s\n' '<new_pub_key_line>' >> /home/claude-admin/.ssh/authorized_keys; \
  sort -u -o /home/claude-admin/.ssh/authorized_keys /home/claude-admin/.ssh/authorized_keys; \
  chattr +i /home/claude-admin/.ssh/authorized_keys; \
  lsattr /home/claude-admin/.ssh/authorized_keys"

This same immutability should be assumed for authorized_keys on other hardened hosts unless verified otherwise with lsattr first.

# Step 1 — generate new ed25519 key pair
$keyPath = "C:\Users\konar\.ssh\claude_admin_new"
ssh-keygen -t ed25519 -f $keyPath -C "claude-admin-$(Get-Date -Format 'yyyy-MM-dd')" -N ""
$newPub = Get-Content "$keyPath.pub"
 
# Step 2 — add new public key to claude-admin authorized_keys on both VPSes
# (vps-i1 and vps-h1 only — claude-admin does not exist on BMS servers)
foreach ($host in @("217.154.82.162", "72.60.32.61")) {
    # Use root key to add to claude-admin's authorized_keys
    ssh -i C:\Users\konar\.ssh\id_ed25519 root@$host `
        "echo '$newPub' >> /home/claude-admin/.ssh/authorized_keys && sort -u /home/claude-admin/.ssh/authorized_keys -o /home/claude-admin/.ssh/authorized_keys"
    Write-Host "$host: new key added"
}
 
# Step 3 — verify new key works on both VPSes
foreach ($host in @("217.154.82.162", "72.60.32.61")) {
    $result = ssh -i $keyPath -o BatchMode=yes -o ConnectTimeout=5 claude-admin@$host "echo ok" 2>&1
    Write-Host "$host: $result"
}
 
# Step 4 — update GH Secret VPS_SSH_PRIVATE_KEY
$privateKeyContent = Get-Content $keyPath -Raw
gh secret set VPS_SSH_PRIVATE_KEY --body $privateKeyContent --repo radieu/p24-infra
 
# Step 5 — verify CI workflows can use the new key (trigger health-check)
gh workflow run health-check.yml --repo radieu/p24-infra
Start-Sleep -Seconds 30
gh run list --repo radieu/p24-infra --workflow health-check.yml --limit 1
 
# Step 6 — get old public key fingerprint, then remove old key from authorized_keys
# (Use the previous GH Secret value's public key to identify and remove it)
# Get old pub key from old private key if still on disk, otherwise check authorized_keys
foreach ($host in @("217.154.82.162", "72.60.32.61")) {
    # Leaves only keys added AFTER the old claude-admin key — human review recommended
    Write-Host "Review and manually remove old key from $host:/home/claude-admin/.ssh/authorized_keys"
    ssh -i $keyPath claude-admin@$host "cat ~/.ssh/authorized_keys"
}
 
# Step 7 — clean up local temp key files
Remove-Item $keyPath, "$keyPath.pub" -Force
 
# Step 8 — update dev_r_services
# UPDATE dev_r_services SET last_rotated='<date>', next_due='<date+365d>'
# WHERE service_name = 'SSH_KEY_CLAUDE_ADMIN';

Note on BMS servers: The claude-admin user does not exist on bms-1, bms-2, bms-3, or bms-4. Those servers use root (bms-1, bms-4) or ubuntu (bms-2, bms-3) with the root SSH key (VPS_ROOT_SSH_KEY). See §6 for root key rotation.

Escalation: If CI workflows fail with Permission denied (publickey) after updating the GH Secret, check that the authorized_keys file still has the new public key (not accidentally deleted). Use the root key to SSH in and verify: ssh root@217.154.82.162 "cat /home/claude-admin/.ssh/authorized_keys".

Log entry: | <date> HH:MM UTC | — | VPS_SSH_PRIVATE_KEY (claude-admin) | scheduled | radieu | yes |

12. SENTRY_AUTH_TOKEN (Sentry source-maps token — et-oper-plat)

What it is: Sentry User Auth Token for source map uploads during Vercel builds of et-operational-platform.
Location: secrets/monitoring.env.sops · Vercel env vars for et-operational-platform
Frequency: 90 days · Next due: 2026-08-01 · Tracked: #1530
Auto-rotate: NO — Sentry sntryu_ tokens cannot be created via API; UI-only.
Detailed playbook: docs/playbooks/sentry-token-rotation.md

Verifying automated rotations after credential-rotation.yml runs

After the Monday 06:00 UTC run, verify these services are healthy:

# 1. Grafana still accessible
curl -s -o /dev/null -w "%{http_code}" https://grafana.vps-i1.infra.zintegrowana.online/login
# Expected: 200
 
# 2. Prometheus still scraping all targets
ssh root@217.154.82.162 "curl -s http://localhost:9090/api/v1/targets | python3 -c 'import json,sys; t=json.load(sys.stdin)[\"data\"][\"activeTargets\"]; print(\"up:\", sum(1 for x in t if x[\"health\"]==\"up\"), \"/\", len(t))'"
 
# 3. Check credential-rotation.yml last run
gh run list --repo radieu/p24-infra --workflow credential-rotation.yml --limit 1

After any rotation — standard checklist

[ ] New value in SOPS (secrets/*.env.sops) — committed and pushed
[ ] New value in GH Secret (gh secret set NAME -R radieu/p24-infra)
[ ] New value in .env.local on dev workstation
[ ] New value deployed to relevant VPS .env (via secrets-sync.yml or SSH)
[ ] Affected service restarted and verified working
[ ] last_rotated + next_due updated in dev_r_services
[x] Rotation entry written to dev_r_rotation_log via rotation-log-entry.py (docs/secrets-rotation-log.md is retired #5556)
[ ] Old credential revoked at the issuing service

Graph View

Backlinks