Standard: Cross-Server SSH Key Distribution

Issue: #2377 · Origin: 2026-07-01 w4.pinbox24.com incident (#2376) — bms-3 had no authorized dev key and a stale SOPS root password, leaving no way to SSH in to diagnose a MongoDB replication failure.

This standard makes the trusted SSH-key set declarative and repeatable so no server is ever left unreachable again.

The standard

Dev key (id_ed25519.pub) → authorized on every managed host (all BMS + all VPSes).
bms-2 root public key → authorized on bms-1, bms-3, bms-4 (mutual server access).
bms-4 root public key → authorized on bms-1, bms-2, bms-3 (mutual server access).

Mutual server access lets an operator/agent on one BMS host reach the others directly (e.g. for MongoDB replica-set diagnostics) without depending on the dev workstation being online.

Access matrix

Key ↓ authorized on →	vps-i1	vps-h1	bms-1	bms-2	bms-3	bms-4
dev `id_ed25519`	✅	✅	✅	✅	✅	✅
bms-2 root pubkey	—	—	✅	(self)	✅	✅
bms-4 root pubkey	—	—	✅	✅	✅	(self)

Implemented as code

Component	Path
Role (additive `authorized_key` management)	`ansible/roles/ssh-authorized-keys/`
Fleet-wide key list	`ansible/inventory/group_vars/all.yml` → `root_trusted_authorized_keys`
Playbook (`hosts: all`)	`ansible/playbooks/ssh-authorized-keys.yml`
Wired into master run	`ansible/playbooks/site.yml`

Lockout safety: the role adds keys with exclusive: false — it only ever adds a key, never removes one, and never edits sshd_config or restarts sshd. It cannot cause an SSH lockout. Public keys are not secrets; the private half never lives in this repo.

Apply

# ALWAYS dry-run first — extra caution on bms-1 (live Pinbox24 production)
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml --check --diff
 
# Apply fleet-wide
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml
 
# Or scope to one host
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml -l bms-3 --check --diff

Audit for drift (#6356)

The apply path is additive only, so it cannot see keys added out-of-band — the drift that accumulated to 7–16 live root keys per host against a 2-key declared set (#6356). Run the role in audit mode to get a read-only report of that drift (it adds/removes nothing):

# Read-only drift report fleet-wide — lists keys present on each host that are
# NOT in root_trusted_authorized_keys, and declared keys missing from the host.
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml -e ssh_authorized_keys_audit=true
 
# One host (extra caution on bms-1 — live Pinbox24 production)
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml -e ssh_authorized_keys_audit=true -l bms-1
 
# Strict — non-zero exit when any undeclared key is found (for CI / scheduled audits)
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml \
  -e ssh_authorized_keys_audit=true -e ssh_authorized_keys_audit_strict=true

Audit mode only surfaces drift. Deciding which undeclared keys are legitimate is a security-review call (sys-security); documenting the legitimate ones goes in root_trusted_authorized_keys; removing an illegitimate one from a live host is a sys-admin action — the additive role never removes a key. Reconcile in that order.

Add a host’s root public key to the trusted set

The dev key is already committed. To complete the mutual-access matrix, capture each host’s own root public key (a public value — safe to commit) and add it to root_trusted_authorized_keys:

# On the source host (public key only — never the private id_ed25519):
cat /root/.ssh/id_ed25519.pub        # e.g. bms-2, bms-4
# If root has no key pair yet: ssh-keygen -t ed25519 -C "root@bms-2" -f /root/.ssh/id_ed25519 -N ''

Then uncomment/add the entry under root_trusted_authorized_keys in ansible/inventory/group_vars/all.yml:

  - name: bms-2-root
    key: "ssh-ed25519 AAAA... root@bms-2"
  - name: bms-4-root
    key: "ssh-ed25519 AAAA... root@bms-4"

Re-run the playbook. (Because it is additive, applying it fleet-wide simply authorizes these keys everywhere; that is a superset of the matrix above and is harmless — access is only ever granted between our own trusted hosts.)

⚠️ Immediate human action — bms-3 recovery (NOT done by this PR)

bms-3 (51.68.155.224) is currently unreachable via SSH — no worker/agent can fix it. A human must, via the OVH KVM console:

Append the dev public key to /root/.ssh/authorized_keys:
```
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMEhAVkEBVGBbb2tzJmkcOb6ucfuno40GOQzEzuHoaec radieu-root-2026-09-02
```
(Superseded 2026-09-02 — radieu-root-2026-06-27 was rotated out fleet-wide after a private-key exposure incident; see the Rotation log in ssh-key-rotation.md.)
Update bare_metal_server_3_root_password in secrets/bms-servers.env.sops to match the actual server password (see docs/playbooks/bms-server-root-ssh-lockout-recovery.md).

Once bms-3 is reachable again, run the playbook with -l bms-3 --check --diff to converge it onto the standard.

Decommissioning a key (removal)

The ssh-authorized-keys role is deliberately additive-only (exclusive: false) — it cannot remove a key, by design, so a routine apply can never cause a lockout. Removing a key that no longer needs standing access (a retired workstation, a rotated-out key, a compromised key) is therefore a separate, explicit, human-run action — never automated through the fleet-wide playbook. This was a documented gap until #6860 (lap2 onboarding) — no equivalent to .sops.yaml’s age-key decommission procedure existed for authorized_keys entries. This section closes it.

When to decommission

A developer workstation (laptop, desktop) is retired, sold, or permanently out of service.
A private key is confirmed or suspected compromised — see docs/playbooks/static-api-key-incident-rotation.md for the exposure-response SLA; SSH keys follow the same 1-hour urgency, just via this procedure instead of SOPS rotation.
A key was granted for a time-boxed purpose (e.g. temporary diagnostic access) that has ended.

Procedure

Remove the declaration. Delete (don’t just comment out) the entry from root_trusted_authorized_keys in ansible/inventory/group_vars/all.yml, and open a PR. Name the key and the reason (retired / rotated / compromised) in the commit message.
Remove it live from every host it was ever pushed to. The additive role cannot do this — run an ad-hoc play from a host with fleet SSH reach:
```
ansible all -m ansible.posix.authorized_key \
  -a "user=root key='<the full public key line>' state=absent" \
  -b -i ansible/inventory/hosts.yml
```
Manual fallback per host (all hosts are managed as user: root by the role regardless of the login account — become: true handles the bms-2/bms-3 ubuntu-login escalation):
```
ssh <login-user>@<host> "grep -v '<unique fragment of the key, e.g. its comment>' \
  /root/.ssh/authorized_keys > /tmp/ak.new && \
  sudo install -m 600 -o root -g root /tmp/ak.new /root/.ssh/authorized_keys && rm /tmp/ak.new"
```
Never sed -i or truncate-and-rewrite authorized_keys directly on a connection you are depending on to not get locked out — write to a temp file and swap it in, so a mistake in the filter leaves the original untouched until the swap succeeds.
Verify. ssh -o BatchMode=yes <login-user>@<host> true using the retired private key (if still available) should now fail with Permission denied; every still-trusted key should still succeed.
Run audit mode fleet-wide afterward, to confirm no host still carries the removed key out-of-band (a stale second copy under a different comment, added manually at some point):
```
ansible-playbook ansible/playbooks/ssh-authorized-keys.yml -e ssh_authorized_keys_audit=true
```
Log it — same log_op pattern as §Audit Log below, op_type="config_change", result="success", naming the key and the reason in detail.
Update the owning environment doc, if any (docs/environments/<host>.md §Keys/§Access), to reflect the removal.

docs/playbooks/ssh-key-rotation.md — developer root key rotation
docs/playbooks/bms-server-root-ssh-lockout-recovery.md — recover a locked-out BMS host
ansible/roles/ssh-authorized-keys/README.md — role reference

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="ssh-authorized-keys",
    result="success",  # "success" | "failed" | "skipped"
    detail="SSH authorized_keys updated across servers per cross-server key standard",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'ssh-authorized-keys', 'success', 'SSH authorized_keys updated across servers per cross-server key standard', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

cross-server-ssh-key-standard

Standard: Cross-Server SSH Key Distribution

The standard

Access matrix

Implemented as code

Apply

Audit for drift (#6356)

Add a host’s root public key to the trusted set

⚠️ Immediate human action — bms-3 recovery (NOT done by this PR)

Decommissioning a key (removal)

When to decommission

Procedure

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

cross-server-ssh-key-standard

Standard: Cross-Server SSH Key Distribution

The standard

Access matrix

Implemented as code

Apply

Audit for drift (#6356)

Add a host’s root public key to the trusted set

⚠️ Immediate human action — bms-3 recovery (NOT done by this PR)

Decommissioning a key (removal)

When to decommission

Procedure

Related

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks