p24-infra Docs

bms-server-root-ssh-lockout-recovery

5 min read

Playbook: BMS Server Root SSH Lockout Recovery (OVH Kimsufi console)

Trigger: ssh root@<bms-ip> fails with Permission denied (publickey,password) and a jump host (e.g. via bms-1) also fails — i.e. the developer workstation key (C:\Users\konar\.ssh\id_ed25519) is not present in /root/.ssh/authorized_keys on the target BMS host.

Why this recurs: the SSH key rotation flow (ssh-key-rotation.md) and per-host provisioning add the workstation key to the ubuntu account on bms-2/bms-3/bms-4 (see that playbook’s server table and step 2), but not to root. When a host is later switched to root-key access without first seeding /root/.ssh/authorized_keys, root login fails. This already happened on bms-4 (fixed 2026-06-16) and recurred on bms-2 (#1905).

Check the documented access path FIRST — root may be the wrong account. Per CLAUDE.md, the documented human SSH path for bms-2 is ubuntu@145.239.133.104 (with claude-admin@… for Claude), not root. bms-4 is the host documented for root login. If you only need to run a command, try ubuntu@<host> + sudo before assuming a root lockout — the workstation key is normally present on ubuntu. Only proceed with console recovery below if root access is genuinely required and ubuntu/claude-admin access is also confirmed broken.

Affected hosts

ServerIPProvider console
bms-194.23.26.113OVH Kimsufi IPMI / KVM
bms-2145.239.133.104OVH Kimsufi IPMI / KVM
bms-351.68.155.224OVH Kimsufi IPMI / KVM
bms-454.36.123.110OVH Kimsufi IPMI / KVM

1. Confirm the lockout (do NOT print key values)

# Exit code only — never use -v (leaks key negotiation), never cat authorized_keys
ssh -o BatchMode=yes -o ConnectTimeout=8 root@145.239.133.104 "echo ok" ; echo "exit=$?"
# Try the documented ubuntu / claude-admin path too — if either works, no console needed:
ssh -o BatchMode=yes -o ConnectTimeout=8 ubuntu@145.239.133.104 "echo ok" ; echo "exit=$?"
ssh -o BatchMode=yes -o ConnectTimeout=8 claude-admin@145.239.133.104 "echo ok" ; echo "exit=$?"

A non-zero exit on all accounts confirms a full key lockout → use the console (step 3). If ubuntu/claude-admin succeed, skip the console — use that account with sudo instead (step 4).

2. Get the public key to add (this is NOT a secret — public keys are safe to read)

On the developer workstation (PowerShell):

Get-Content C:\Users\konar\.ssh\id_ed25519.pub

Copy the single ssh-ed25519 AAAA… radieu@… line. (The matching private key never leaves the workstation.)

3. Add the key via OVH Kimsufi console (human action — requires OVH account)

  1. Log in to the OVH Manager → Bare Metal Cloud → select the server (bms-2 = ns3087638) → KVM / IPMI → open the console.

  2. Log in at the console as root (or ubuntu then sudo -i). Console uses the local password, not the SSH key, so it works while SSH is locked out.

  3. Seed root’s authorized_keys idempotently (paste the public key from step 2 in place of KEY):

    install -d -m 700 -o root -g root /root/.ssh
grep -qxF 'KEY' /root/.ssh/authorized_keys 2>/dev/null \
  || echo 'KEY' >> /root/.ssh/authorized_keys
chmod 600 /root/.ssh/authorized_keys
chown root:root /root/.ssh/authorized_keys

  4. Confirm sshd permits root key login (default for these hosts is prohibit-password, which allows keys):

    sshd -T 2>/dev/null | grep -i permitrootlogin   # expect: prohibit-password (or yes)

    If it shows no, root key login is intentionally disabled on this host — use the ubuntu account path (step 4) instead of re-enabling root.

4. Alternative — use the ubuntu account + sudo (no console needed)

If ubuntu@<host> still accepts the workstation key (it usually does — rotation seeds it there):

ssh ubuntu@145.239.133.104 "sudo grep -c '' /root/.ssh/authorized_keys"   # check, no values printed
# Add the workstation key to root from inside the ubuntu session:
ssh ubuntu@145.239.133.104 \
  "sudo bash -c \"install -d -m700 /root/.ssh; grep -qxF 'KEY' /root/.ssh/authorized_keys || echo 'KEY' >> /root/.ssh/authorized_keys; chmod 600 /root/.ssh/authorized_keys\""

5. Verify recovery

ssh -o BatchMode=yes root@145.239.133.104 "hostname"   # expect: ns3087638

6. Prevention

  • When rotating the workstation key, add it to both ubuntu and root on every BMS host — update ssh-key-rotation.md loops if a host needs root access.
  • When provisioning or onboarding a new BMS host, seed /root/.ssh/authorized_keys with the workstation key during the initial console session, before SSH-only access is assumed.
  • The Ansible common role only pre-checks that /root/.ssh/authorized_keys is non-empty (a lockout guard) — it does not push keys. Seeding root keys remains a manual/console step.

Escalation

If the OVH console itself is unavailable or root has no working password:

  • Open an OVH support ticket for IPMI/KVM access to the affected ns… server.
  • Send a Discord alert via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (red) and keep the tracking GitHub issue open with label human-action until root SSH is restored.

Recovery log

DateHostActorNotes
2026-06-16bms-4radieuroot authorized_keys seeded (first occurrence)
2026-06-28bms-2pending#1905 — OVH console action required

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="radieu",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="bms-root-ssh",
    result="success",  # "success" | "failed" | "skipped"
    detail="BMS server root SSH lockout recovery — authorized_keys restored via IPMI/KVM",
    env="bms-1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('radieu', 'config_change', 'bms-root-ssh', 'success', 'BMS server root SSH lockout recovery — authorized_keys restored via IPMI/KVM', 'bms-1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks