p24-infra Docs

fleet-backup-ssh-cf-tunnel

9 min read

Playbook — Fleet Backup SSH via Cloudflare Tunnel

Status: ACTIVE Scope: Secondary SSH access path for every server in the fleet, independent of each server’s public IP.

What this is

Every server already has primary SSH via its public IP + key (docs/playbooks/server-ssh-connectivity.md). This playbook adds a secondary, outbound-only path via a per-server Cloudflare Tunnel — useful if a server’s public IP becomes unreachable (provider network issue, accidental firewall lockout, IP change) without requiring any inbound port to be open for it. This is the same mechanism already used for lap1 and wsl1 (hosts with no public IP at all), extended to the 6 servers that do have one, purely as a backup.

This does not enable, change, or affect worker dispatch on any host. It is SSH access only — no dev_r_server_capacity rows were touched for bms-1/2/3/4, vps-i1, or vps-h1. vps-i1 in particular runs the production monitoring stack; this addition was scoped deliberately narrow (install cloudflared, one systemd service, nothing else) to avoid any risk to it.

Hostname map

ServerPrimary (public IP)Backup (CF Tunnel)SSH user
bms-194.23.26.113bms1-ssh.zintegrowana.onlineroot
bms-2145.239.133.104bms2-ssh.zintegrowana.onlineroot
bms-351.68.155.224bms3-ssh.zintegrowana.onlineubuntu
bms-454.36.123.110bms4-ssh.zintegrowana.onlineubuntu
vps-i1217.154.82.162i1-ssh.zintegrowana.onlineroot
vps-h172.60.32.61h1-ssh.zintegrowana.onlineroot
lap1— (no public IP)lap1-ssh.zintegrowana.onlineroot
wsl1— (no public IP)wsl1-ssh.zintegrowana.onlineroot

All hostnames are single-level subdomains of zintegrowana.online — not X.tunnel.zintegrowana.online. This is load-bearing: the zone’s free Universal SSL wildcard (*.zintegrowana.online) covers exactly one subdomain level. A two-level hostname (X.tunnel.zintegrowana.online, the original naming used for wsl1/lap1 before 2026-08-11) is outside that coverage — Cloudflare’s edge resets the TLS handshake before completing it, and fixing that requires either a paid Advanced Certificate Pack or renaming to a single level. See docs/environments/wsl1.md §Resolved Issues for the full diagnosis. Never add a new CF Tunnel hostname under a grouping subdomain in this zone — always <name>.zintegrowana.online directly.

What was provisioned per server (2026-08-11)

  • cloudflared installed (.deb on Ubuntu hosts, .rpm on vps-i1/AlmaLinux — vps-i1 already had a stale 2026.3.0 binary from an earlier, unrelated setup; upgraded to 2026.7.3 to match)
  • A dedicated Cloudflare Tunnel per server (config_src: cloudflare, remotely-managed), ingress service: ssh://localhost:22 — confirmed status: healthy for all 8 (6 new + lap1 + wsl1)
  • DNS CNAME <hostname> → <tunnel-id>.cfargotunnel.com (proxied)
  • A Cloudflare Access application (type ssh) + policy <name>-owner-only (allow radieu@gmail.com) per hostname — first connection from any client machine needs a one-time browser login (email OTP), cached for 24h afterward. This is normal Access behavior.
  • cloudflared installed as a systemd service via connector token (cloudflared service install <token>)
  • Connector run-tokens stored in secrets/administration.env.sops as CF_TUNNEL_<LABEL>_TOKEN (CF_TUNNEL_BMS1_TOKEN … CF_TUNNEL_H1_TOKEN), same pattern as the existing CF_TUNNEL_LAP1_TOKEN / CF_TUNNEL_WSL1_TOKEN
  • SSH config aliases on the dev workstation (C:\Users\konar\.ssh\config): bms1-tunnel, bms2-tunnel, bms3-tunnel, bms4-tunnel, i1-tunnel, h1-tunnel (existing wsl1/lap1 aliases unchanged) — each ProxyCommand cloudflared access ssh --hostname %h

Using the backup channel

ssh bms4-tunnel     # instead of ssh bms-4 / ssh root@54.36.123.110
ssh i1-tunnel

First use per client machine triggers a Cloudflare Access browser login (email OTP to radieu@gmail.com) — complete it once, cached for 24h.

Verifying a tunnel is healthy

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# retrieve CF_GLOBAL_API_KEY via the safe extraction pattern, then:
Invoke-RestMethod -Uri "https://api.cloudflare.com/client/v4/accounts/acc5b2c956dd488265003364e2c7f84a/cfd_tunnel?is_deleted=false" `
  -Headers @{"X-Auth-Email"="radieu@gmail.com"; "X-Auth-Key"=$globalKey} |
  Select-Object -ExpandProperty result | Where-Object { $_.name -match "-ssh$" } |
  Select-Object name, status

Or directly on the server: systemctl is-active cloudflared (expect active).

Known gotcha — one alias at a time only (org-level token lock)

cloudflared.exe’s Access token cache (%USERPROFILE%\.cloudflared\) uses a single shared org-level lock file (p24-infra-team.cloudflareaccess.com-org-token.lock), not a per-hostname lock. Consequences, found 2026-08-11 while validating this rollout:

  • Running ssh <label>-tunnel for a second alias while a first alias’s session/login is still active hangs or fails with websocket: bad handshake — the second process blocks on (or loses a race for) the shared lock.
  • Force-killing cloudflared.exe (Stop-Process -Force) mid-operation can leave orphaned *-token.lock files behind, and in one case deleted the actual *-token file while leaving its .lock/.url sentinels — silently breaking a previously-working alias (wsl1 regressed this way after unrelated batch operations on the other 6 hosts).

Fix / working pattern: use one alias at a time. Close the SSH session (exit) before starting the next ssh <label>-tunnel. If something looks stuck, clear the whole cache and start fresh rather than half-fixing it:

Get-Process cloudflared -ErrorAction SilentlyContinue | Stop-Process -Force
Remove-Item "$env:USERPROFILE\.cloudflared\*-token*" -Force -ErrorAction SilentlyContinue
# then retry ssh <label>-tunnel — triggers a fresh browser login

Known gotcha — mid-session CF token 401/403 → check for a concurrent rotation FIRST (#7005)

Provisioning steps here decrypt CF_API_TOKEN (secrets/monitoring.env.sops) via the sops-invoke.ps1 broker and make a burst of CF API calls (verify → list tunnels → list Access apps/service-tokens → create). If the first few calls succeed and then every call — including /user/tokens/verify — flips to 403/401, the most likely cause is not a WAF/bot heuristic or a per-token IP allowlist. It is a concurrent incident-rotation deleting the token out from under your session:

CF_API_TOKEN is a Tier-1 auto-rotating credential (rotate_cf_api_token in scripts/rotate-credentials.py, run by .github/workflows/credential-rotation.yml). A rotation mints the new token and DELETES the old one at Cloudflare via API immediately, then opens a separate PR to commit the new value to secrets/monitoring.env.sops. So there is a window where:

  • the old token is already dead at Cloudflare (every call 401/403), but
  • secrets/monitoring.env.sops on main still holds the old value — until the auto-rotate PR merges and secrets-sync.yml redistributes.

Any consumer that decrypted the token before the rotation keeps using the now-deleted value. This is exactly what happened to lap2 on 2026-09-14 (#7005): the #7002 exposure triggered a rotation at 14:18:13Z, the GH Secret was rewritten at 14:18:46Z, and the lap2 failure was filed at 14:23:49Z — ~5 min after the old token was killed. Checking git log for a SOPS file commit in the session window does not rule this out, because the token dies at rotation time while the SOPS commit lands later (auto-rotate PR #7004 was still open at that point).

Triage order when a CF token 401/403s mid-session:

# 1. Was there a rotation run around your failure window?
gh run list --repo radieu/p24-infra --workflow credential-rotation.yml --limit 5 \
  --json createdAt,conclusion --jq '.[] | "\(.createdAt) \(.conclusion)"'
# 2. Is an auto-rotate SOPS PR still open (new value not yet on main)?
gh pr list --repo radieu/p24-infra --search 'auto-rotate credentials in:title' --state open \
  --json number,title,createdAt
# 3. GH Secret timestamp (metadata only) — did the value change under you?
gh secret list --repo radieu/p24-infra | grep '^CF_API_TOKEN'

Fix: do NOT open a CF-dashboard investigation or add an IP allowlist. Wait for the auto-rotate PR to merge and secrets-sync.yml to redistribute, then re-decrypt CF_API_TOKEN fresh (via the sops-invoke.ps1 broker) and retry. The token is healthy; you were holding a stale value. Only if steps 1–3 show no concurrent rotation should you suspect a genuine WAF/IP-filter block and escalate to the CF dashboard.

Dispatcher-side pinned cloudflared (bms-4) — cloudflared#1673 regression workaround (#7072)

The queue dispatcher (scripts/queue-dispatcher-loop.py) reaches CF-Tunnel-backed hosts (windows-dev, wsl1, lap1, lap2) by wrapping SSH in a cloudflared access ssh --hostname %h ProxyCommand and authenticating non-interactively with a CF Access Service Token (the tunnel’s non_identity policy). cloudflared 2026.6.0+ silently ignores the Service Token for access ssh (upstream cloudflare/cloudflared#1673) — the connection fails at the Access layer with websocket: bad handshake before SSH auth even starts. bms-4’s system cloudflared (/usr/bin/cloudflared, 2026.7.x) is on the broken side of that regression.

The dispatcher therefore pins a known-good older binary via DISPATCHER_CF_TUNNEL_PROXYCOMMAND (in secrets/n8n-bms4.env.sops, deployed to /opt/p24-infra/bms-4/.env), which points at:

/opt/p24-infra/bin/cloudflared-2026.5.1      # sha256 3c6a5ba995a258dbe90f98e5fdb2c2620b7be72c3ca761614f6eb52aee252cea

If that binary is missing the ProxyCommand fails with No such file or directory → kex_exchange_identification: Connection closed by remote host, and every automated SSH-through-tunnel dispatch breaks (root cause of #7072 — the pin was referenced by the env var but never actually placed on bms-4). This is not a per-host lap2 problem: the env var is shared config for all CF-Tunnel dispatch targets.

Restore / verify the pin (idempotent, safe to re-run):

# no-op when already correct; downloads + checksum-verifies + installs the official release if not
/opt/p24-infra/scripts/ensure-dispatcher-cloudflared-pin.sh
/opt/p24-infra/scripts/ensure-dispatcher-cloudflared-pin.sh --check   # verify only, exit 1 if bad

/opt/p24-infra/bin/ is group-writable by claude-runner (setgid), so the dispatcher’s worker accounts can restore it without root.

Self-healing mechanisms (#7898) — the pin now survives across all deploy and restart paths:

TriggerMechanismBehaviour on failure
Every p24-queue-dispatcher.service startExecStartPre=-/opt/p24-infra/scripts/ensure-dispatcher-cloudflared-pin.sh (the - prefix makes it non-blocking)Dispatcher still starts; missing-binary Discord alert fires from queue-dispatcher.sh
Every scripts/** deploy to bms-4deploy-vps-scripts.yml runs ensure-dispatcher-cloudflared-pin.sh after git reset --hard origin/mainCI step fails if download fails (non-best-effort — pin must survive scripts deploys)
Every Ansible bms4-workers role applyansible/roles/bms4-workers/tasks/main.yml runs the script with failed_when: rc != 0Ansible apply fails if binary cannot be restored (provisioning gate)
Pin missing after ExecStartPrequeue-dispatcher.sh section 1a checks DISPATCHER_CF_TUNNEL_PROXYCOMMAND’s first wordDiscord alert via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL, rate-limited to once per hour (sentinel /tmp/.dispatcher-cf-pin-alert)

Quick confidence check that the binary gap is closed (does not need the Service Token):

timeout 25 /opt/p24-infra/bin/cloudflared-2026.5.1 access ssh \
  --hostname lap2-ssh.zintegrowana.online < /dev/null 2>&1 | tail -1
# GOOD: reaches the Access layer (e.g. `websocket: bad handshake` when no token is present).
# BAD:  `No such file or directory` — the pin is missing again; run the restore script.

Full end-to-end SSH-through-tunnel to a given host ALSO requires that host’s CF Access Service Token to be deployed into the dispatcher env. For lap2 the keys are CF_ACCESS_LAP2_CLIENT_ID / CF_ACCESS_LAP2_CLIENT_SECRET (secrets/n8n-bms4.env.sops). Until those land on main and secrets-sync.yml redistributes them to /opt/p24-infra/bms-4/.env, the probe above stays at websocket: bad handshake even with the pin present — that is the token-not-deployed state, not the binary state. (lap2’s token was provisioned on the still-open PR #7071 at the time of #7072; see that PR + docs/adr/006-wsl-worker-host-class.md.) Note also that _cf_tunnel_service_token_env() currently resolves a single global token-var name (CF_ACCESS_SERVICE_TOKEN_ID_VAR, default CF_ACCESS_WINDOWS_DEV_CLIENT_ID) — per-host token selection for lap2 is tracked with the lap2 onboarding work, not here.

Escalation

If a server’s primary public IP fails and the CF Tunnel backup is also unreachable, fall back to the provider console (see docs/playbooks/p24-network-admin.md §Emergency Procedures — IONOS / Hostinger / OVH KVM per server).

  • docs/environments/wsl1.md §Resolved Issues — the SSL-coverage bug this rollout avoided by design
  • docs/environments/lap1.md §Remote Access — the original single-host precedent
  • docs/playbooks/p24-network-admin.md — CF Tunnel operational patterns
  • docs/playbooks/server-ssh-connectivity.md — primary (public IP) SSH access

Graph View

Backlinks