p24-infra Docs

bms3-ssh-key-auth-only

4 min read

Playbook: bms-3 root SSH — key-only auth (password path deprecated)

Closes: #2369 · Related: #2367, #2377 / cross-server-ssh-key-standard.md

What triggers this

Any session that needs root SSH to bms-3 (OVH Kimsufi ns3129867, 51.68.155.224 — MongoDB rs0 member + Pinbox24 staging) and instinctively reaches for the SOPS bare_metal_server_3_root_password in secrets/bms-servers.env.sops first.

Stop — that password is no longer a valid auth path for root on this host. Root password login is disabled at the sshd level (see below), independent of whether the SOPS value itself is current. Use the developer workstation SSH key instead.

Root cause of the original #2369 confusion

sshd -T on bms-3 reports:

permitrootlogin without-password
pubkeyauthentication yes
passwordauthentication yes        <- global default, but irrelevant for root (see next line)

PermitRootLogin without-password (a.k.a. prohibit-password) means: root may authenticate via public key (or other non-password method), but password authentication is rejected for the root account specifically — even though PasswordAuthentication yes is the sshd-wide default for other accounts (ubuntu, gitlab-runner, claude-runner). This is why every attempt to use bare_metal_server_3_root_password failed with Permission denied (publickey,password): it wasn’t (only) that the SOPS value had drifted — root password auth is structurally disabled on this host. #2369 initially framed this as “stale password, needs rotation”; the actual fix is stop depending on the password for root entirely.

Current state (verified 2026-08-01)

  • Root key-based SSH already works: the developer workstation key (C:\Users\konar\.ssh\id_ed25519) is present in /root/.ssh/authorized_keys on bms-3 (5 keys total — dev key + bms-2/bms-4 mutual root keys per cross-server-ssh-key-standard.md).
  • sshd -T confirms permitrootlogin without-password + pubkeyauthentication yes — no sshd_config change was needed; the host was already converged onto key-only root auth by the time this was checked. (Likely landed via the ssh-authorized-keys Ansible role / #3171 IaC persistence work — see git log --oneline -i --grep bms-3, e.g. 43be4035, 71ed639a.)
  • The SOPS key bare_metal_server_3_root_password in secrets/bms-servers.env.sops is now deprecated / not the primary auth method for bms-3 root. Do not delete it without secret-manager sign-off — it may still be read by other tooling or used as an emergency console-recovery credential (KVM console login is separate from sshd and still uses the local password). Its staleness relative to the live host is no longer an operational blocker.

Going forward — access method for bms-3 root

# From the Windows dev workstation — exit code / stdout only, never -v
ssh -o BatchMode=yes -o ConnectTimeout=8 -i C:\Users\konar\.ssh\id_ed25519 root@51.68.155.224 "echo ok"
  • Do not attempt bare_metal_server_3_root_password for root SSH — it will not work by design (without-password), regardless of whether the SOPS value is rotated.
  • If key auth ever fails (full lockout), follow bms-server-root-ssh-lockout-recovery.md — OVH Kimsufi KVM/IPMI console, which authenticates with the local root password (a different trust boundary from sshd), independent of sshd’s PermitRootLogin policy.
  • ubuntu@51.68.155.224 + sudo remains available as a secondary path (same dev key).

Verify

ssh -o BatchMode=yes -i C:\Users\konar\.ssh\id_ed25519 root@51.68.155.224 hostname   # expect: ns3129867

Resolved item — the #2367 cron follow-up (both action items now landed)

#2367 (“add dev key to bms-3 + change mongodb-backup cron to daily”) was closed 2026-07-01 via an unrelated merged PR (#2371, a mongodb-restore-drill-cron.sh fix) — not because its own two action items were verified complete. The dev-key item was confirmed done (see above); the mongodb-backup cron item lagged behind and was still weekly (0 1 * * 0) as late as 2026-08-01.

Resolved 2026-08-01 (#4833): the bms-3 root crontab now runs the backup daily (30 1 * * * /root/mongodb-backup.sh full >> /var/log/mongodb-backup.log 2>&1) — script path and log redirection unchanged. The pre-change crontab was saved to /root/.crontab.bak.20260801 on bms-3 before the edit. This closes the RPO gap tracked in #2148 (backup RPO 7 days → 1 day). The edit was applied over key-auth root SSH from the bms-4 infra-task worker, per this playbook.

Prevention

  • Never assume a closed issue’s every action item shipped — closure via an unrelated PR’s auto-close keyword does not verify unrelated scope in the same issue body.
  • When checking bms-3 (or any BMS host) root SSH, run sshd -T | grep -i permitrootlogin before assuming a password rotation will fix access — without-password/prohibit-password makes password rotation a no-op for root.

Graph View