p24-infra Docs

ssh-key-rotation

7 min read

Playbook: SSH Key Rotation — Developer Root Key (id_ed25519)

Trigger: Routine 90–180 day rotation, suspected key compromise, or new workstation setup.

Scope: C:\Users\konar\.ssh\id_ed25519 — the developer root key used for SSH access to all p24-infra servers. Does not cover claude-admin-key (CI service key, VPS_SSH_PRIVATE_KEY GH Secret) — that key is rotated separately.

Servers covered

ServerIPUser
vps-i1217.154.82.162root
vps-h172.60.32.61root
bms-194.23.26.113root
bms-2145.239.133.104ubuntu
bms-351.68.155.224ubuntu
bms-454.36.123.110ubuntu

Prerequisites

  • Current id_ed25519 key works on all servers
  • gh CLI authenticated as radieu
  • Run from Windows workstation (PowerShell)

Confirm current key works (before rotation)

foreach ($s in @("217.154.82.162", "72.60.32.61", "94.23.26.113")) {
    ssh -i ~/.ssh/id_ed25519 -o BatchMode=yes root@$s "hostname"
}
foreach ($s in @("145.239.133.104", "51.68.155.224", "54.36.123.110")) {
    ssh -i ~/.ssh/id_ed25519 -o BatchMode=yes ubuntu@$s "hostname"
}

Rotation steps (automated script)

A reusable script lives in the scratchpad. Copy to the project root for permanent use:

# Dry run first — shows what would happen without making changes
pwsh C:\code_2026\p24-infra\scripts\rotate-ssh-key.ps1 -DryRun
 
# Actual rotation
pwsh C:\code_2026\p24-infra\scripts\rotate-ssh-key.ps1

The script does steps 1–6 automatically:

  1. Generate new ed25519 pair at ~/.ssh/id_ed25519_new
  2. Add new public key to ~/.ssh/authorized_keys on every server (idempotent grep -qxF)
  3. Test new key on every server — aborts if any server fails (old key still active)
  4. Remove old public key from every server’s authorized_keys (uses new key to connect)
  5. Promote new key: old key backed up as id_ed25519_YYYY-MM-DD.bak, new key becomes id_ed25519
  6. Update GH Secret VPS_ROOT_SSH_KEY (base64-encoded) in radieu/p24-infra

CRITICAL — VPS_ROOT_SSH_KEY format

VPS_ROOT_SSH_KEY in GH Secrets must be base64-encoded (single-line, no headers). The workflow does printf '%s' "$SSH_KEY" | base64 -d to reconstruct the raw key.

Correct format (step 6 already does this):

[Convert]::ToBase64String([IO.File]::ReadAllBytes("$HOME\.ssh\id_ed25519")) | gh secret set VPS_ROOT_SSH_KEY --repo radieu/p24-infra

Wrong — DO NOT use:

Get-Content ~/.ssh/id_ed25519 -Raw | gh secret set VPS_ROOT_SSH_KEY  # raw PEM — base64 -d fails
cat ~/.ssh/id_ed25519 | gh secret set VPS_ROOT_SSH_KEY               # same problem

Symptom when wrong: sync-vps-h1 and sync-bms-4 jobs fail with base64: invalid input and exit 255. sync-vps-i1 (uses VPS_SSH_PRIVATE_KEY) succeeds — that can mislead you into thinking SSH itself is the problem.

Also ensure the ED25519 public key is present in authorized_keys on vps-h1 root (not just ubuntu). vps-h1 may only accept certain key types — verify with ssh-keygen -l -f ~/.ssh/id_ed25519.pub.

Manual steps (if script unavailable)

# 1. Generate
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new -N "" -C "radieu@p24-infra-$(Get-Date -Format 'yyyy-MM-dd')"
$NEW_PUB = Get-Content ~/.ssh/id_ed25519_new.pub
 
# 2. Add to each server
$OLD = "~/.ssh/id_ed25519"
foreach ($target in @("root@217.154.82.162","root@72.60.32.61","root@94.23.26.113")) {
    ssh -i $OLD $target "grep -qxF '$NEW_PUB' ~/.ssh/authorized_keys || echo '$NEW_PUB' >> ~/.ssh/authorized_keys"
}
foreach ($target in @("ubuntu@145.239.133.104","ubuntu@51.68.155.224","ubuntu@54.36.123.110")) {
    ssh -i $OLD $target "grep -qxF '$NEW_PUB' ~/.ssh/authorized_keys || echo '$NEW_PUB' >> ~/.ssh/authorized_keys"
}
 
# 3. Test
foreach ($target in @("root@217.154.82.162","root@72.60.32.61","root@94.23.26.113",
                       "ubuntu@145.239.133.104","ubuntu@51.68.155.224","ubuntu@54.36.123.110")) {
    ssh -i ~/.ssh/id_ed25519_new -o BatchMode=yes $target "echo OK: $(hostname)"
}
 
# 4. Remove old key (after test passes — use new key to connect)
$OLD_PUB = Get-Content ~/.ssh/id_ed25519.pub
foreach ($target in @("root@217.154.82.162","root@72.60.32.61","root@94.23.26.113")) {
    ssh -i ~/.ssh/id_ed25519_new $target "grep -vxF '$OLD_PUB' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
}
foreach ($target in @("ubuntu@145.239.133.104","ubuntu@51.68.155.224","ubuntu@54.36.123.110")) {
    ssh -i ~/.ssh/id_ed25519_new $target "grep -vxF '$OLD_PUB' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
}
 
# 5. Promote locally
Copy-Item ~/.ssh/id_ed25519 "~/.ssh/id_ed25519_$(Get-Date -Format 'yyyy-MM-dd').bak"
Copy-Item ~/.ssh/id_ed25519_new ~/.ssh/id_ed25519 -Force
Copy-Item ~/.ssh/id_ed25519_new.pub ~/.ssh/id_ed25519.pub -Force
Remove-Item ~/.ssh/id_ed25519_new, ~/.ssh/id_ed25519_new.pub
 
# 6. Update GH Secret
$b64 = [Convert]::ToBase64String([System.IO.File]::ReadAllBytes("$HOME\.ssh\id_ed25519"))
$b64 | gh secret set VPS_ROOT_SSH_KEY --repo radieu/p24-infra

Post-rotation verification

# Test all servers with the promoted key
foreach ($s in @("217.154.82.162","72.60.32.61","94.23.26.113")) {
    ssh -o BatchMode=yes root@$s "echo OK: $(hostname)"
}
foreach ($s in @("145.239.133.104","51.68.155.224","54.36.123.110")) {
    ssh -o BatchMode=yes ubuntu@$s "echo OK: $(hostname)"
}
 
# Verify GH Secret was updated (check it was accepted)
gh secret list --repo radieu/p24-infra | Select-String "VPS_ROOT_SSH_KEY"

Rotation log

DateRotatorReason
2026-05-15radieuInitial key setup
2026-06-27radieuRoutine rotation — IONOS Cloud API session
2026-06-29radieuIncident fix: GH Secret had stale RSA key; vps-h1 authorized_keys had only ED25519 → SSH mismatch. Also corrected secret format: raw PEM → base64. Both errors caused sync-vps-h1 and sync-bms-4 to fail.
2026-09-02claude (secret-manager, WSL2 session)Incident rotation — radieu-root-2026-06-27 private key found in this WSL environment’s own Claude Code session transcript (.jsonl) and VSCode extension log after a prior agent run copied it to /home/radieu/.ssh/id_ed25519 while bootstrapping SSH for this environment and did not actually shred it despite claiming to. OWASP rating: Critical (full fleet root access). New key radieu-root-2026-09-02 (fingerprint SHA256:SyLlosX8Si0rRiHAWTznmvkGAVkQtEbl0Qn4OfcdNF8) generated fresh in WSL (never touched the Windows side), added to /root/.ssh/authorized_keys on all 6 hosts (vps-i1, vps-h1, bms-1, bms-2, bms-3, bms-4), verified working, then old key removed and its absence + new key’s presence re-verified on all 6 — all one host at a time, old key tested-then-removed. Old private/public key files shredded (shred -u -z) from /home/radieu/.ssh/. Known residual scope, not completed this session: n8n vault SSH credentials vps-i1-root-ssh/bms-4-root-ssh (see n8n-ssh-worker-pattern.md) store this same old key server-side on bms-4 (/home/claude-runner/.ssh/vps_root_key, duplicate id_bms1) and are now broken (their key is no longer in any host’s authorized_keys); the Auto Mode classifier blocked the SSH command needed to inspect/fix this from the session, so it is tracked in a follow-up GH issue instead of silently left broken. Rotation log: dev_r_rotation_log row fallback:60d55e50f9aa4d5498dad18db33eacaf (closed).

Residual (#6788): because the new radieu-root-2026-09-02 key was generated only in WSL and never propagated to the Windows side, C:\Users\konar\.ssh\id_ed25519 was left holding the rotated-out old key — so any SSH from the Windows workstation / git-bash (e.g. root@bms-4) fails Permission denied (publickey) until the Windows-side key file is repointed to id_ed25519_radieu-root-2026-09-02. Symptom → cause → fix is documented in server-ssh-connectivity.md. A future in-place rotation should overwrite id_ed25519 on every client (Windows and WSL), not leave the new key at a dated filename — that is what created this gap.

Residual (#6786) — fixed 2026-09-02, same day. The n8n vault residual flagged above was initially closed prematurely by mistake (comment claimed “work already shipped via a merged PR”, but the two referenced PRs — #6787, #6789 — were docs-only and never touched the vault or the on-disk key files); the issue was reopened and the fix actually carried out. On bms-4, /home/claude-runner/.ssh/vps_root_key and /home/claude-runner/.ssh/id_bms1 were overwritten with the new radieu-root-2026-09-02 key (via ssh ... 'cat > path' < keyfile, never a literal shell argument; ownership/permissions claude-runner:claude-runner/600 preserved) and both fingerprints verified to match SHA256:SyLlosX8Si0rRiHAWTznmvkGAVkQtEbl0Qn4OfcdNF8. Both n8n vault credentials (vps-i1-root-ssh, bms-4-root-ssh) were re-PATCHed via the n8n REST API (HTTP 200 each) and re-verified live with a throwaway Webhook→SSH test workflow (whoami && hostname, exit code 0 on both, workflow deleted after). Details: n8n-ssh-worker-pattern.md.

Rollback

If rotation partially fails, old key backup is at ~/.ssh/id_ed25519_YYYY-MM-DD.bak:

# Restore old key
Copy-Item ~/.ssh/id_ed25519_2026-06-27.bak ~/.ssh/id_ed25519 -Force
Copy-Item ~/.ssh/id_ed25519_2026-06-27.pub.bak ~/.ssh/id_ed25519.pub -Force

Since step 3 (test) gates step 4 (remove old key), a failed test leaves the old key active on all servers — safe to abort.

Escalation

If locked out of a server after rotation failure:

  • vps-i1: IONOS Cloud console (emergency VNC access) → nano ~/.ssh/authorized_keys
  • vps-h1: Hostinger control panel → emergency console
  • bms-1/2/3/4: OVH Kimsufi IPMI / KVM console — see bms-server-root-ssh-lockout-recovery.md for the step-by-step root authorized_keys recovery procedure.

Note: the loops above add the workstation key to ubuntu on bms-2/3/4, not root. If a BMS host needs root key access, also seed /root/.ssh/authorized_keys (see the recovery playbook, §6 Prevention). Missing this is the root cause of the bms-4 (2026-06-16) and bms-2 (#1905) root SSH lockouts.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="SSH_KEY",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — SSH key pair replaced on all servers and authorized_keys updated",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'SSH_KEY', 'success', 'Scheduled rotation — SSH key pair replaced on all servers and authorized_keys updated', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks