p24-infra Docs

fleet-backup-ssh-cf-tunnel

4 min read

Playbook — Fleet Backup SSH via Cloudflare Tunnel

Status: ACTIVE Scope: Secondary SSH access path for every server in the fleet, independent of each server’s public IP.

What this is

Every server already has primary SSH via its public IP + key (docs/playbooks/server-ssh-connectivity.md). This playbook adds a secondary, outbound-only path via a per-server Cloudflare Tunnel — useful if a server’s public IP becomes unreachable (provider network issue, accidental firewall lockout, IP change) without requiring any inbound port to be open for it. This is the same mechanism already used for lap1 and wsl1 (hosts with no public IP at all), extended to the 6 servers that do have one, purely as a backup.

This does not enable, change, or affect worker dispatch on any host. It is SSH access only — no dev_r_server_capacity rows were touched for bms-1/2/3/4, vps-i1, or vps-h1. vps-i1 in particular runs the production monitoring stack; this addition was scoped deliberately narrow (install cloudflared, one systemd service, nothing else) to avoid any risk to it.

Hostname map

ServerPrimary (public IP)Backup (CF Tunnel)SSH user
bms-194.23.26.113bms1-ssh.zintegrowana.onlineroot
bms-2145.239.133.104bms2-ssh.zintegrowana.onlineroot
bms-351.68.155.224bms3-ssh.zintegrowana.onlineubuntu
bms-454.36.123.110bms4-ssh.zintegrowana.onlineubuntu
vps-i1217.154.82.162i1-ssh.zintegrowana.onlineroot
vps-h172.60.32.61h1-ssh.zintegrowana.onlineroot
lap1— (no public IP)lap1-ssh.zintegrowana.onlineroot
wsl1— (no public IP)wsl1-ssh.zintegrowana.onlineroot

All hostnames are single-level subdomains of zintegrowana.online — not X.tunnel.zintegrowana.online. This is load-bearing: the zone’s free Universal SSL wildcard (*.zintegrowana.online) covers exactly one subdomain level. A two-level hostname (X.tunnel.zintegrowana.online, the original naming used for wsl1/lap1 before 2026-08-11) is outside that coverage — Cloudflare’s edge resets the TLS handshake before completing it, and fixing that requires either a paid Advanced Certificate Pack or renaming to a single level. See docs/environments/wsl1.md §Resolved Issues for the full diagnosis. Never add a new CF Tunnel hostname under a grouping subdomain in this zone — always <name>.zintegrowana.online directly.

What was provisioned per server (2026-08-11)

  • cloudflared installed (.deb on Ubuntu hosts, .rpm on vps-i1/AlmaLinux — vps-i1 already had a stale 2026.3.0 binary from an earlier, unrelated setup; upgraded to 2026.7.3 to match)
  • A dedicated Cloudflare Tunnel per server (config_src: cloudflare, remotely-managed), ingress service: ssh://localhost:22 — confirmed status: healthy for all 8 (6 new + lap1 + wsl1)
  • DNS CNAME <hostname> → <tunnel-id>.cfargotunnel.com (proxied)
  • A Cloudflare Access application (type ssh) + policy <name>-owner-only (allow radieu@gmail.com) per hostname — first connection from any client machine needs a one-time browser login (email OTP), cached for 24h afterward. This is normal Access behavior.
  • cloudflared installed as a systemd service via connector token (cloudflared service install <token>)
  • Connector run-tokens stored in secrets/administration.env.sops as CF_TUNNEL_<LABEL>_TOKEN (CF_TUNNEL_BMS1_TOKEN … CF_TUNNEL_H1_TOKEN), same pattern as the existing CF_TUNNEL_LAP1_TOKEN / CF_TUNNEL_WSL1_TOKEN
  • SSH config aliases on the dev workstation (C:\Users\konar\.ssh\config): bms1-tunnel, bms2-tunnel, bms3-tunnel, bms4-tunnel, i1-tunnel, h1-tunnel (existing wsl1/lap1 aliases unchanged) — each ProxyCommand cloudflared access ssh --hostname %h

Using the backup channel

ssh bms4-tunnel     # instead of ssh bms-4 / ssh root@54.36.123.110
ssh i1-tunnel

First use per client machine triggers a Cloudflare Access browser login (email OTP to radieu@gmail.com) — complete it once, cached for 24h.

Verifying a tunnel is healthy

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# retrieve CF_GLOBAL_API_KEY via the safe extraction pattern, then:
Invoke-RestMethod -Uri "https://api.cloudflare.com/client/v4/accounts/acc5b2c956dd488265003364e2c7f84a/cfd_tunnel?is_deleted=false" `
  -Headers @{"X-Auth-Email"="radieu@gmail.com"; "X-Auth-Key"=$globalKey} |
  Select-Object -ExpandProperty result | Where-Object { $_.name -match "-ssh$" } |
  Select-Object name, status

Or directly on the server: systemctl is-active cloudflared (expect active).

Known gotcha — one alias at a time only (org-level token lock)

cloudflared.exe’s Access token cache (%USERPROFILE%\.cloudflared\) uses a single shared org-level lock file (p24-infra-team.cloudflareaccess.com-org-token.lock), not a per-hostname lock. Consequences, found 2026-08-11 while validating this rollout:

  • Running ssh <label>-tunnel for a second alias while a first alias’s session/login is still active hangs or fails with websocket: bad handshake — the second process blocks on (or loses a race for) the shared lock.
  • Force-killing cloudflared.exe (Stop-Process -Force) mid-operation can leave orphaned *-token.lock files behind, and in one case deleted the actual *-token file while leaving its .lock/.url sentinels — silently breaking a previously-working alias (wsl1 regressed this way after unrelated batch operations on the other 6 hosts).

Fix / working pattern: use one alias at a time. Close the SSH session (exit) before starting the next ssh <label>-tunnel. If something looks stuck, clear the whole cache and start fresh rather than half-fixing it:

Get-Process cloudflared -ErrorAction SilentlyContinue | Stop-Process -Force
Remove-Item "$env:USERPROFILE\.cloudflared\*-token*" -Force -ErrorAction SilentlyContinue
# then retry ssh <label>-tunnel — triggers a fresh browser login

Escalation

If a server’s primary public IP fails and the CF Tunnel backup is also unreachable, fall back to the provider console (see docs/playbooks/p24-network-admin.md §Emergency Procedures — IONOS / Hostinger / OVH KVM per server).

  • docs/environments/wsl1.md §Resolved Issues — the SSL-coverage bug this rollout avoided by design
  • docs/environments/lap1.md §Remote Access — the original single-host precedent
  • docs/playbooks/p24-network-admin.md — CF Tunnel operational patterns
  • docs/playbooks/server-ssh-connectivity.md — primary (public IP) SSH access

Graph View