p24-infra Docs

cross-server-ssh-key-standard

3 min read

Standard: Cross-Server SSH Key Distribution

Issue: #2377 · Origin: 2026-07-01 w4.pinbox24.com incident (#2376) — bms-3 had no authorized dev key and a stale SOPS root password, leaving no way to SSH in to diagnose a MongoDB replication failure.

This standard makes the trusted SSH-key set declarative and repeatable so no server is ever left unreachable again.

The standard

  1. Dev key (id_ed25519.pub) → authorized on every managed host (all BMS + all VPSes).
  2. bms-2 root public key → authorized on bms-1, bms-3, bms-4 (mutual server access).
  3. bms-4 root public key → authorized on bms-1, bms-2, bms-3 (mutual server access).

Mutual server access lets an operator/agent on one BMS host reach the others directly (e.g. for MongoDB replica-set diagnostics) without depending on the dev workstation being online.

Access matrix

Key ↓ authorized on →vps-i1vps-h1bms-1bms-2bms-3bms-4
dev id_ed25519✅✅✅✅✅✅
bms-2 root pubkey——✅(self)✅✅
bms-4 root pubkey——✅✅✅(self)

Implemented as code

ComponentPath
Role (additive authorized_key management)ansible/roles/ssh-authorized-keys/
Fleet-wide key listansible/inventory/group_vars/all.yml → root_trusted_authorized_keys
Playbook (hosts: all)ansible/playbooks/ssh-authorized-keys.yml
Wired into master runansible/playbooks/site.yml

Lockout safety: the role adds keys with exclusive: false — it only ever adds a key, never removes one, and never edits sshd_config or restarts sshd. It cannot cause an SSH lockout. Public keys are not secrets; the private half never lives in this repo.

Apply

# ALWAYS dry-run first — extra caution on bms-1 (live Pinbox24 production)
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml --check --diff
 
# Apply fleet-wide
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml
 
# Or scope to one host
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml -l bms-3 --check --diff

Add a host’s root public key to the trusted set

The dev key is already committed. To complete the mutual-access matrix, capture each host’s own root public key (a public value — safe to commit) and add it to root_trusted_authorized_keys:

# On the source host (public key only — never the private id_ed25519):
cat /root/.ssh/id_ed25519.pub        # e.g. bms-2, bms-4
# If root has no key pair yet: ssh-keygen -t ed25519 -C "root@bms-2" -f /root/.ssh/id_ed25519 -N ''

Then uncomment/add the entry under root_trusted_authorized_keys in ansible/inventory/group_vars/all.yml:

  - name: bms-2-root
    key: "ssh-ed25519 AAAA... root@bms-2"
  - name: bms-4-root
    key: "ssh-ed25519 AAAA... root@bms-4"

Re-run the playbook. (Because it is additive, applying it fleet-wide simply authorizes these keys everywhere; that is a superset of the matrix above and is harmless — access is only ever granted between our own trusted hosts.)

⚠️ Immediate human action — bms-3 recovery (NOT done by this PR)

bms-3 (51.68.155.224) is currently unreachable via SSH — no worker/agent can fix it. A human must, via the OVH KVM console:

  1. Append the dev public key to /root/.ssh/authorized_keys: 
    ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH6tT80EqDe2B5RnzPSwFgMWOGNta1rOhIy+nMmxiC5b radieu-root-2026-06-27
  2. Update bare_metal_server_3_root_password in secrets/bms-servers.env.sops to match the actual server password (see docs/playbooks/bms-server-root-ssh-lockout-recovery.md).

Once bms-3 is reachable again, run the playbook with -l bms-3 --check --diff to converge it onto the standard.

  • docs/playbooks/ssh-key-rotation.md — developer root key rotation
  • docs/playbooks/bms-server-root-ssh-lockout-recovery.md — recover a locked-out BMS host
  • ansible/roles/ssh-authorized-keys/README.md — role reference

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="ssh-authorized-keys",
    result="success",  # "success" | "failed" | "skipped"
    detail="SSH authorized_keys updated across servers per cross-server key standard",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'ssh-authorized-keys', 'success', 'SSH authorized_keys updated across servers per cross-server key standard', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks