p24-infra Docs

ssh-key-rotation

5 min read

Playbook: SSH Key Rotation — Developer Root Key (id_ed25519)

Trigger: Routine 90–180 day rotation, suspected key compromise, or new workstation setup.

Scope: C:\Users\konar\.ssh\id_ed25519 — the developer root key used for SSH access to all p24-infra servers. Does not cover claude-admin-key (CI service key, VPS_SSH_PRIVATE_KEY GH Secret) — that key is rotated separately.

Servers covered

ServerIPUser
vps-i1217.154.82.162root
vps-h172.60.32.61root
bms-194.23.26.113root
bms-2145.239.133.104ubuntu
bms-351.68.155.224ubuntu
bms-454.36.123.110ubuntu

Prerequisites

  • Current id_ed25519 key works on all servers
  • gh CLI authenticated as radieu
  • Run from Windows workstation (PowerShell)

Confirm current key works (before rotation)

foreach ($s in @("217.154.82.162", "72.60.32.61", "94.23.26.113")) {
    ssh -i ~/.ssh/id_ed25519 -o BatchMode=yes root@$s "hostname"
}
foreach ($s in @("145.239.133.104", "51.68.155.224", "54.36.123.110")) {
    ssh -i ~/.ssh/id_ed25519 -o BatchMode=yes ubuntu@$s "hostname"
}

Rotation steps (automated script)

A reusable script lives in the scratchpad. Copy to the project root for permanent use:

# Dry run first — shows what would happen without making changes
pwsh C:\code_2026\p24-infra\scripts\rotate-ssh-key.ps1 -DryRun
 
# Actual rotation
pwsh C:\code_2026\p24-infra\scripts\rotate-ssh-key.ps1

The script does steps 1–6 automatically:

  1. Generate new ed25519 pair at ~/.ssh/id_ed25519_new
  2. Add new public key to ~/.ssh/authorized_keys on every server (idempotent grep -qxF)
  3. Test new key on every server — aborts if any server fails (old key still active)
  4. Remove old public key from every server’s authorized_keys (uses new key to connect)
  5. Promote new key: old key backed up as id_ed25519_YYYY-MM-DD.bak, new key becomes id_ed25519
  6. Update GH Secret VPS_ROOT_SSH_KEY (base64-encoded) in radieu/p24-infra

CRITICAL — VPS_ROOT_SSH_KEY format

VPS_ROOT_SSH_KEY in GH Secrets must be base64-encoded (single-line, no headers). The workflow does printf '%s' "$SSH_KEY" | base64 -d to reconstruct the raw key.

Correct format (step 6 already does this):

[Convert]::ToBase64String([IO.File]::ReadAllBytes("$HOME\.ssh\id_ed25519")) | gh secret set VPS_ROOT_SSH_KEY --repo radieu/p24-infra

Wrong — DO NOT use:

Get-Content ~/.ssh/id_ed25519 -Raw | gh secret set VPS_ROOT_SSH_KEY  # raw PEM — base64 -d fails
cat ~/.ssh/id_ed25519 | gh secret set VPS_ROOT_SSH_KEY               # same problem

Symptom when wrong: sync-vps-h1 and sync-bms-4 jobs fail with base64: invalid input and exit 255. sync-vps-i1 (uses VPS_SSH_PRIVATE_KEY) succeeds — that can mislead you into thinking SSH itself is the problem.

Also ensure the ED25519 public key is present in authorized_keys on vps-h1 root (not just ubuntu). vps-h1 may only accept certain key types — verify with ssh-keygen -l -f ~/.ssh/id_ed25519.pub.

Manual steps (if script unavailable)

# 1. Generate
ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519_new -N "" -C "radieu@p24-infra-$(Get-Date -Format 'yyyy-MM-dd')"
$NEW_PUB = Get-Content ~/.ssh/id_ed25519_new.pub
 
# 2. Add to each server
$OLD = "~/.ssh/id_ed25519"
foreach ($target in @("root@217.154.82.162","root@72.60.32.61","root@94.23.26.113")) {
    ssh -i $OLD $target "grep -qxF '$NEW_PUB' ~/.ssh/authorized_keys || echo '$NEW_PUB' >> ~/.ssh/authorized_keys"
}
foreach ($target in @("ubuntu@145.239.133.104","ubuntu@51.68.155.224","ubuntu@54.36.123.110")) {
    ssh -i $OLD $target "grep -qxF '$NEW_PUB' ~/.ssh/authorized_keys || echo '$NEW_PUB' >> ~/.ssh/authorized_keys"
}
 
# 3. Test
foreach ($target in @("root@217.154.82.162","root@72.60.32.61","root@94.23.26.113",
                       "ubuntu@145.239.133.104","ubuntu@51.68.155.224","ubuntu@54.36.123.110")) {
    ssh -i ~/.ssh/id_ed25519_new -o BatchMode=yes $target "echo OK: $(hostname)"
}
 
# 4. Remove old key (after test passes — use new key to connect)
$OLD_PUB = Get-Content ~/.ssh/id_ed25519.pub
foreach ($target in @("root@217.154.82.162","root@72.60.32.61","root@94.23.26.113")) {
    ssh -i ~/.ssh/id_ed25519_new $target "grep -vxF '$OLD_PUB' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
}
foreach ($target in @("ubuntu@145.239.133.104","ubuntu@51.68.155.224","ubuntu@54.36.123.110")) {
    ssh -i ~/.ssh/id_ed25519_new $target "grep -vxF '$OLD_PUB' ~/.ssh/authorized_keys > /tmp/ak && mv /tmp/ak ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys"
}
 
# 5. Promote locally
Copy-Item ~/.ssh/id_ed25519 "~/.ssh/id_ed25519_$(Get-Date -Format 'yyyy-MM-dd').bak"
Copy-Item ~/.ssh/id_ed25519_new ~/.ssh/id_ed25519 -Force
Copy-Item ~/.ssh/id_ed25519_new.pub ~/.ssh/id_ed25519.pub -Force
Remove-Item ~/.ssh/id_ed25519_new, ~/.ssh/id_ed25519_new.pub
 
# 6. Update GH Secret
$b64 = [Convert]::ToBase64String([System.IO.File]::ReadAllBytes("$HOME\.ssh\id_ed25519"))
$b64 | gh secret set VPS_ROOT_SSH_KEY --repo radieu/p24-infra

Post-rotation verification

# Test all servers with the promoted key
foreach ($s in @("217.154.82.162","72.60.32.61","94.23.26.113")) {
    ssh -o BatchMode=yes root@$s "echo OK: $(hostname)"
}
foreach ($s in @("145.239.133.104","51.68.155.224","54.36.123.110")) {
    ssh -o BatchMode=yes ubuntu@$s "echo OK: $(hostname)"
}
 
# Verify GH Secret was updated (check it was accepted)
gh secret list --repo radieu/p24-infra | Select-String "VPS_ROOT_SSH_KEY"

Rotation log

DateRotatorReason
2026-05-15radieuInitial key setup
2026-06-27radieuRoutine rotation — IONOS Cloud API session
2026-06-29radieuIncident fix: GH Secret had stale RSA key; vps-h1 authorized_keys had only ED25519 → SSH mismatch. Also corrected secret format: raw PEM → base64. Both errors caused sync-vps-h1 and sync-bms-4 to fail.

Rollback

If rotation partially fails, old key backup is at ~/.ssh/id_ed25519_YYYY-MM-DD.bak:

# Restore old key
Copy-Item ~/.ssh/id_ed25519_2026-06-27.bak ~/.ssh/id_ed25519 -Force
Copy-Item ~/.ssh/id_ed25519_2026-06-27.pub.bak ~/.ssh/id_ed25519.pub -Force

Since step 3 (test) gates step 4 (remove old key), a failed test leaves the old key active on all servers — safe to abort.

Escalation

If locked out of a server after rotation failure:

  • vps-i1: IONOS Cloud console (emergency VNC access) → nano ~/.ssh/authorized_keys
  • vps-h1: Hostinger control panel → emergency console
  • bms-1/2/3/4: OVH Kimsufi IPMI / KVM console — see bms-server-root-ssh-lockout-recovery.md for the step-by-step root authorized_keys recovery procedure.

Note: the loops above add the workstation key to ubuntu on bms-2/3/4, not root. If a BMS host needs root key access, also seed /root/.ssh/authorized_keys (see the recovery playbook, §6 Prevention). Missing this is the root cause of the bms-4 (2026-06-16) and bms-2 (#1905) root SSH lockouts.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="SSH_KEY",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — SSH key pair replaced on all servers and authorized_keys updated",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'SSH_KEY', 'success', 'Scheduled rotation — SSH key pair replaced on all servers and authorized_keys updated', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks