Playbook: Server SSH Connectivity
Status (2026-07-08): Headscale VPN fully decommissioned — container, config/data, and Caddy vhost removed from vps-i1; Tailscale disconnected on vps-i1 and windows-dev. Port 22 is open from internet on all servers. Use public IPs directly.
Note: the 2026-07-06 entry below claiming decommission was inaccurate — the Headscale container was still running (
restart: unless-stoppedkept reviving it) until this date, and bms-4’s~/.ssh/confighost aliases (vps-i1, bms-1/2/3, vps-h1, dev-laptop) still pointed at dead VPN IPs the whole time. Both are now fixed. dev-laptop has no public IP — it’s only reachable via the autossh reverse tunnel through vps-i1 (port 2222), which requires dev-laptop itself to be powered on and connected; this could not be verified end-to-end today.
Server address book
| Server | Public IP | SSH key |
|---|---|---|
| vps-i1 (IONOS) | 217.154.82.162 | developer id_ed25519 |
| vps-h1 (Hostinger) | 72.60.32.61 | developer id_ed25519 |
| bms-1 (OVH ns367522) | 94.23.26.113 | developer id_ed25519 |
| bms-2 (OVH ns3087638) | 145.239.133.104 | developer id_ed25519 |
| bms-3 (OVH ns3129867) | 51.68.155.224 | developer id_ed25519 |
| bms-4 (OVH ns3101999) | 54.36.123.110 | developer id_ed25519 |
| dev-laptop | LAN: 192.168.0.161 | — |
| windows-dev (WSL2 Docker worker) | CF Tunnel: windows-dev-ssh.tunnel.zintegrowana.online (no public IP) | CF Tunnel — no Tailscale |
SSH key: C:\Users\konar\.ssh\id_ed25519 (local) / ~/.ssh/id_ed25519 (on servers)
windows-dev (#6110 / ADR-005 §3): reached over CF Tunnel inbound, the same non-mesh pattern as
lap1/wsl1— extended so the tunnel carries thedocker runinvocation ofscripts/run-windows-dev-worker.sh, not just a bare SSH shell. Explicitly no Tailscale (the mesh VPN removal, #3294, is not reversed). The node ships DISABLED until its tunnel + image + dispatcher docker-path exist — see provision-windows-dev-docker-worker.md.
How to connect
From Windows dev machine
ssh -i C:\Users\konar\.ssh\id_ed25519 root@217.154.82.162 # vps-i1
ssh -i C:\Users\konar\.ssh\id_ed25519 root@72.60.32.61 # vps-h1
ssh -i C:\Users\konar\.ssh\id_ed25519 root@94.23.26.113 # bms-1
ssh -i C:\Users\konar\.ssh\id_ed25519 root@145.239.133.104 # bms-2
ssh -i C:\Users\konar\.ssh\id_ed25519 root@51.68.155.224 # bms-3
ssh -i C:\Users\konar\.ssh\id_ed25519 root@54.36.123.110 # bms-4From bms-4 CI runner (GH Actions self-hosted)
ssh -i ~/.ssh/vps_root_key root@217.154.82.162 # vps-i1
ssh -i ~/.ssh/vps_root_key root@72.60.32.61 # vps-h1vps_root_key is decoded from the VPS_ROOT_SSH_KEY GH Secret (base64-encoded developer key).
From vps-i1 or any server
ssh root@54.36.123.110 # bms-4
ssh root@72.60.32.61 # vps-h1
ssh root@94.23.26.113 # bms-1Never enumerate SSH usernames — fail2ban ban hazard (#5603, #3591)
Each host accepts exactly one canonical login user. Never probe a host by trying several
candidate users in turn (root@, then claude-admin@, then ubuntu@) to see which one connects —
claude-admin does not exist on any BMS server (bms-1/2/3/4), it exists only on the two VPSes
(vps-i1, vps-h1). Every attempt against a non-existent user is an invalid-user auth failure in the
target’s sshd, and repeated failures across retries trip that host’s fail2ban sshd jail, which
then bans the source IP fleet-wide — silently stranding all future SSH (and infra-task dispatch)
to that host. This is exactly how bms-1’s fail2ban banned bms-4 (#5603, root-caused during #3591).
Canonical user per host: root on vps-i1 / vps-h1 / bms-1 / bms-4; ubuntu on bms-2 / bms-3. If the
canonical user fails, escalate via bms-server-root-ssh-lockout-recovery.md — do not retry with
a different username. Automated tooling reads ssh_user per host from dev_r_server_capacity (the
dispatcher) or a per-host map (scripts/p24-status.py); keep those the single source of truth.
Emergency access (server unreachable via SSH)
Try the CF Tunnel backup first — every server (bms-1/2/3/4, vps-i1, vps-h1) has a secondary
SSH path via Cloudflare Tunnel, independent of the public IP: ssh <label>-tunnel (e.g. ssh bms4-tunnel, ssh i1-tunnel) from the dev workstation. Covers IP changes, accidental firewall
lockouts, provider network issues — anything short of the server/tunnel-connector itself being
down. One-time Cloudflare Access browser login per client machine. Use one alias at a time —
concurrent aliases hit a shared token-lock bug, see docs/playbooks/fleet-backup-ssh-cf-tunnel.md.
If that also fails, fall through to the provider console:
- OVH BMS servers (bms-1/2/3/4): OVH IPMI console — credentials in
secrets/administration.env.sops→IONOS_API_KEY/OVH_*keys. Usesecrets/ovh-api.env.sopsfor API-based rescue mode. - vps-i1: IONOS Cloud Console
- vps-h1: Hostinger control panel
Firewall details (post-VPN decommission)
Port 22 is open from internet on all servers. Headscale VPN and the 100.64.0.0/10 rules have been removed.
| Server | Firewall | SSH from internet |
|---|---|---|
| vps-i1 | firewalld | ✅ open |
| vps-h1 | ufw | ✅ open (Tailscale removed 2026-07-06) |
| bms-1/2/3/4 | ufw | ✅ open (Tailscale removed 2026-07-06) |
History
- 2026-07-05 Phase 5: Port 22 blocked from internet, VPN-only — superseded 2026-07-06
- 2026-07-06: Headscale VPN decommissioned (server restart loop during recovery); all servers reverted to public-IP SSH
Delivering a secret to a server via SSH stdin — Windows PowerShell gotcha (#5225, 2026-08-03)
Never do this — $env:VAR | ssh user@host "remote-script-that-does-read -r" — for two stacked
reasons, confirmed during a GH_TOKEN redeploy that leaked the value into an agent’s own tool
output:
- PowerShell’s pipe-to-native-process prepends a UTF-8 BOM and uses CRLF line endings. A
remote
read -r NEWTOKreads the BOM-prefixed,\r-suffixed line as-is. If the remote script then tries to use that value as a shell argument/command (or the BOM breaks a guard condition), bash can echo the raw value back in an error message — a full secret exposure. - Passing the secret inline as an argument after the remote command
(
ssh host "bash -s" $secretScriptOrValue) is separately wrong:sshjoins all trailing arguments into a single space-separated remote command line, so anything passed this way is NOT delivered as intended — it becomes part of the command string (visible inps auxon the remote host at minimum, and can desync stdin handling entirely).
Safe pattern:
# 1. Write the remote script (NO secret inside it) to a local file, LF-only, no BOM:
$path = "$env:TEMP\deploy-secret.sh"
[System.IO.File]::WriteAllText($path, (Get-Content -Raw $path) -replace "`r`n","`n", [System.Text.UTF8Encoding]::new($false))
# 2. Ship the script itself over scp (safe — contains no secret):
scp -o BatchMode=yes $path root@<host>:/tmp/deploy-secret.sh
ssh -o BatchMode=yes root@<host> "chmod 700 /tmp/deploy-secret.sh"
# 3. The remote script MUST strip both the BOM and CRLF from stdin before using the value:
# NEWTOK=$(cat | LC_ALL=C sed 's/^\xef\xbb\xbf//' | tr -d '\r\n')
# (do NOT use `read -r` for this — it does not strip a leading BOM and only strips
# the trailing \n, not \r)
# 4. Verify the stripping logic with a harmless dummy value FIRST (e.g. "test123" → expect
# length 7 back), before ever piping the real secret through it.
# 5. Only then pipe the real secret as the sole content of a single remote command:
$env:THE_SECRET | ssh -o BatchMode=yes root@<host> "/tmp/deploy-secret.sh"
$env:THE_SECRET = ""
# 6. Clean up: rm the remote script, never leave secret-delivery scripts sitting in /tmp.
ssh -o BatchMode=yes root@<host> "rm -f /tmp/deploy-secret.sh"If a value is suspected to have leaked via this vector (garbled read output, a “command not
found” error containing what looks like a token, or a write that silently produced an
empty/wrong-length value), stop and follow docs/playbooks/static-api-key-incident-rotation.md
immediately — do not assume the write succeeded without an explicit length/fingerprint check
(Get-SopsKeyFingerprint-style, never a raw value comparison).
Related
docs/playbooks/bms-server-root-ssh-lockout-recovery.md— if locked out of a BMS serverdocs/playbooks/ssh-agent-pattern.md— silent-unless-error pattern for automated SSHsecrets/ovh-api.env.sops— OVH API credentials for rescue mode managementdocs/playbooks/static-api-key-incident-rotation.md— exposure incident response