SSH brute-force mitigation applied to the two VPS hosts (vps-i1 IONOS / AlmaLinux 9.7 and vps-h1 Hostinger / Ubuntu 24.04) and the bms-1 bare-metal host (Pinbox24 production). Covers the hardened sshd policy and the fail2ban jail that bans abusive source IPs.
The bms-1 posture (fleet-IP ufw SSH allowlist + strengthened fail2ban, applied 2026-08-05 under #5604) has its own section below: bms-* bare-metal fleet.
Why
Both hosts were receiving ongoing internet SSH brute-force traffic — tens of thousands of failed login attempts per day from bot IP ranges. No password login ever succeeded (all real access is publickey only), but the noise inflated auth.log/secure and the SSHAuthFailures Prometheus rule, and represented standing risk. The hardening below removes password/keyboard-interactive auth as an attack surface entirely and auto-bans repeat offenders.
What was hardened
Setting
Value
Effect
PasswordAuthentication
no
Password logins refused — publickey only
PermitRootLogin
prohibit-password
root may log in by key, never by password
KbdInteractiveAuthentication
no
Disables PAM keyboard-interactive fallback (closes the password backdoor)
fail2bansshd jail
enabled
Bans source IPs after repeated auth failures
All real access remains publickey — see docs/vps-i1-operations.md and docs/hostinger-runbook.md for the per-host SSH key matrix. No legitimate workflow used password auth, so this change is non-breaking.
sshd settings + file locations per host
vps-i1 (IONOS / AlmaLinux 9.7)
The hardened settings live in a drop-in (so they survive package upgrades to the base /etc/ssh/sshd_config):
/etc/ssh/sshd_config.d/00-hardening.conf
PasswordAuthentication noPermitRootLogin prohibit-passwordKbdInteractiveAuthentication no
Apply / reload after edits:
sshd -t # validate config — MUST pass before reloadsystemctl reload sshd
vps-h1 (Hostinger / Ubuntu 24.04)
The PasswordAuthentication no / PermitRootLogin prohibit-password / KbdInteractiveAuthentication no settings were already applied by Hostinger cloud-init (typically in /etc/ssh/sshd_config.d/50-cloud-init.conf and the base config). No drop-in was added for the auth policy — only fail2ban was installed. Verify the effective policy with:
passwordauthentication no
permitrootlogin prohibit-password
kbdinteractiveauthentication no
If any value drifts, add /etc/ssh/sshd_config.d/00-hardening.conf with the three lines above (drop-ins are read in lexical order; a 00- prefix wins over cloud-init’s 50-).
fail2ban jail config
fail2ban is a systemd-managed daemon installed on both hosts. The sshd jail config:
findtime = 10m — …counted over a 10-minute sliding window…
bantime = 1h — …triggers a 1-hour IP ban (via the firewall / nftables ban action).
backend = systemd — fail2ban reads auth events from the systemd journal rather than tailing a log file. This matters for Ubuntu 24.04 (see caveat below).
Jail config file location is the standard fail2ban layout:
# Is the daemon up?systemctl status fail2ban# Jail summary — currently banned IPs, total failures, total bansfail2ban-client status sshd
Example fail2ban-client status sshd output:
Status for the jail: sshd
|- Filter
| |- Currently failed: 3
| |- Total failed: 48211
| `- File list: <journal>
`- Actions
|- Currently banned: 7
|- Total banned: 1902
`- Banned IP list: 45.95.147.x 193.32.162.x ...
How to unban an IP
# Remove a single IP from the sshd jail ban listfail2ban-client set sshd unbanip <IP># Unban everything in the sshd jail (use sparingly)fail2ban-client unban --all
Whitelist trusted IPs/CIDRs permanently with ignoreip in the jail config (e.g. the developer workstation or CI egress) so they are never banned:
[sshd]ignoreip = 127.0.0.1/8 ::1 <trusted-cidr>
Fleet-wide ignoreip allowlist — all p24 infra IPs (#5603, #3591)
Why: the bms-4 infra-task worker SSHes out to every fleet host to dispatch/execute work.
A single stray invalid-user auth failure across dispatch retries can trip a peer’s sshd jail and
ban the source fleet IP, silently stranding all future SSH (and infra-task dispatch) to that host.
This is exactly how bms-1’s fail2ban banned bms-4, root-caused during #3591. Fix #1 removed the
invalid-user probe at the source (PR #5610); this allowlist is the defense-in-depth half — no probe
pattern from any p24 infra IP can ever trip a peer’s jail again.
Rollout method (additive, easily reversible): a dedicated drop-in
/etc/fail2ban/jail.d/zz-p24-fleet-ignoreip.local carrying just the [sshd] ignoreip line above.
The zz- prefix makes it the last jail.d file read, so its ignoreip wins over any pre-existing
value; rollback is rm of the single file + fail2ban-client reload sshd. Apply with:
# root hosts: vps-i1, vps-h1, bms-1, bms-4tee /etc/fail2ban/jail.d/zz-p24-fleet-ignoreip.local >/dev/null <<'EOF'[sshd]ignoreip = 127.0.0.1/8 ::1 54.36.123.110 217.154.82.162 145.239.133.104 51.68.155.224 94.23.26.113 72.60.32.61EOFfail2ban-client reload sshdfail2ban-client get sshd ignoreip # verify all 6 fleet IPs are listed# ubuntu hosts (bms-2, bms-3): prefix the tee/reload/get with `sudo -n`
On a host that already has a non-fleetignoreip entry (e.g. bms-3 carried a developer IP
95.91.246.219), append that IP to the drop-in’s line so it is preserved — the drop-in replaces,
not merges, the effective list for the sshd jail.
Rollout status (2026-08-05, #5603 fix #2):
Host
fail2ban
Action taken
bms-1
active
Already carried the full fleet allowlist (immediate fix from #3591) — verified, no change
Drop-in written for future-proofing; service left inactive (not started — its state is unchanged, no reload)
bms-2
not installed
Out of scope — no fail2ban package present
No fleet IP was banned on any host at rollout time, so no unbanip was required.
bms-* bare-metal fleet (bms-1, #5604)
The bms-* bare-metal boxes were historically not covered by this workbook — it documented only
the two VPS hosts. bms-1 (OVH ns367522, 94.23.26.113, Ubuntu, Pinbox24 P0 EOL production)
was the first bms-* host to be brought under it, on 2026-08-05 under #5604, after a
sys-security access review and architect human sign-off (staged Option 3).
Context
During #3591 remediation, bms-1’s sshdfail2ban jail showed ~847 total bans / ~31.7k failed
attempts — port 22 was ALLOW IN Anywhere (v4 and v6) and under constant internet brute-force.
bms-1 is already publickey-only (PasswordAuthentication no, PermitRootLogin without-password
— the legacy alias for prohibit-password, KbdInteractiveAuthentication no), so none of that
traffic could ever succeed; the exposure was noise / log-volume / CPU plus residual risk, not a
password-guessing risk. This change is a fail2ban-noise reduction + defense-in-depth, not a
credential-theft mitigation.
What was applied (staged Option 3 — additive / allowlist-only)
Port 22 was deliberately left open to Anywhere (not closed to the allowlist) because two
legitimate source IPs are dynamic and not yet resolved: (a) the GitHub-hosted runner in
audit-bms1-git-deploy-v42-prod.yml (runs-on: ubuntu-latest, rotating egress IP — exists
precisely because bms-1 is unreachable from the self-hosted AI-runner pool), and (b) the developer
workstation (dynamic ISP IP, undocumented by design). Because port 22 stays open, adding ALLOW
rules and tightening fail2ban is purely additive and cannot lock anyone out. Closing port 22
to the allowlist (full Option 1) is tracked as a separate follow-up once (a) and (b) are resolved.
1. ufw — allowlist the 5 known-static fleet IPs for SSH (these are informational/priority
ALLOW rules today since Anywhere is still open; they become the enforcing allowlist when full
Option 1 lands):
ufw allow from 54.36.123.110 to any port 22 proto tcp comment "bms-4 fleet-ssh (#5604)"ufw allow from 217.154.82.162 to any port 22 proto tcp comment "vps-i1 fleet-ssh (#5604)"ufw allow from 145.239.133.104 to any port 22 proto tcp comment "bms-2 fleet-ssh (#5604)"ufw allow from 51.68.155.224 to any port 22 proto tcp comment "bms-3 fleet-ssh (#5604)"ufw allow from 72.60.32.61 to any port 22 proto tcp comment "vps-h1 fleet-ssh (#5604)"
2. ufw — remove the stale Headscale VPN rule. Headscale was removed fleet-wide 2026-07-08
(PR #3294) but the 22 ALLOW IN 100.64.0.0/10 # p24-vpn-ssh rule survived on bms-1 (inert — that
CGNAT range no longer routes to bms-1). Removed as IaC-drift cleanup:
ufw --force delete allow from 100.64.0.0/10 to any port 22
3. fail2ban — strengthen the sshd jail (Option 2 parameters). bms-1 keeps its jail config in
/etc/fail2ban/jail.local (not a jail.d/*.local drop-in). maxretry5 → 3, findtime600 → 300, and bantime.increment = true added so repeat offenders age out progressively slower.
The #3591 fleet ignoreip allowlist is preserved verbatim (never ban our own infra hosts):
[sshd]enabled = trueport = sshmaxretry = 3bantime = 3600findtime = 300bantime.increment = true# p24-infra host allowlist (#3591) — never ban our own infra hosts.# loopback, bms-4, vps-i1, bms-2, bms-3, vps-h1ignoreip = 127.0.0.1/8 ::1 54.36.123.110 217.154.82.162 145.239.133.104 51.68.155.224 72.60.32.61
fail2ban-client reload sshd
Note: bms-1’s fail2ban DB has dbpurgeage = 1d (/etc/fail2ban/fail2ban.conf), so
bantime.increment history is retained for 1 day — long enough to escalate persistent same-day
offenders, short enough that a reformed IP resets. Not changed here (out of the approved scope).
Verification (2026-08-05, post-change)
ufw: 100.64.0.0/10 rule gone; 5 fleet /32 ALLOW rules present; 22/tcp + 22/tcp(v6) still ALLOW Anywhere
fail2ban: maxretry=3 findtime=300 bantime=3600 bantime.increment=True; ignoreip = all 5 fleet IPs + loopback
fail2ban-client status sshd → active, still banning external attackers
bms-4 → bms-1 SSH confirmed working under the new rules (the change was applied over that path)
Rollback
# ── fail2ban ──cp -a /etc/fail2ban/jail.local.bak-5604 /etc/fail2ban/jail.local # backup taken pre-changefail2ban-client reload sshd# ── ufw ── (remove the 5 fleet rules; restore the stale VPN rule only if truly needed)ufw delete allow from 54.36.123.110 to any port 22 proto tcpufw delete allow from 217.154.82.162 to any port 22 proto tcpufw delete allow from 145.239.133.104 to any port 22 proto tcpufw delete allow from 51.68.155.224 to any port 22 proto tcpufw delete allow from 72.60.32.61 to any port 22 proto tcp# ufw allow from 100.64.0.0/10 to any port 22 # only if Headscale is ever reinstated (it is not)
The pre-change ufw status numbered snapshot is saved on bms-1 at /root/ufw-status-before-5604.txt.
SSH access note for this fleet (#5604)
bms-1 accepts root by key only — the canonical infra-task worker path from bms-4 uses the
dedicated ~/.ssh/id_bms1 key (the generic ~/.ssh/id_ed25519 is not authorized on bms-1).
Never enumerate SSH usernames against any bms-* host — claude-admin does not exist there and
invalid-user probes trip the peer’s fail2ban jail (#5603/#3591). Use root@94.23.26.113 with
-i ~/.ssh/id_bms1 only; if that fails, escalate via
docs/playbooks/bms-server-root-ssh-lockout-recovery.md rather than trying another user.
Full Option 1 — close port 22 to allowlist-only (#5642): status & remaining gates
Status: NOT YET APPLIED. Port 22 on bms-1 is still ALLOW Anywhere (v4 + v6). #5642 set out to
close it to the allowlist. Investigation from the bms-4 infra-task worker found the real blocker set
is larger than #5642 assumed (it named one dynamic SSH source; there are three) and two of the
three plus the developer-workstation source still need decisions. The ufw close must not run
until every item below is cleared and it is scheduled into a W3/W4 prod window with architect
sign-off — same caution as #5604. Closing early would lock out the sources still on a dynamic IP and
strand a weekly cron on P0 EOL production.
Complete inventory of automated SSH access to bms-1 (verified 2026-08-06)
Every workflow/path that opens an SSH session to root@94.23.26.113, and whether it survives the
allowlist-only close:
The 5 static fleet /32ALLOW rules from #5604 (bms-4, vps-i1, bms-2, bms-3, vps-h1) become the
enforcing allowlist on close. bms-4 + vps-i1 cover the two self-hosted paths and the worker path.
Gate 1 — three GitHub-hosted (ubuntu-latest) SSH sources, not one
#5642 named only audit-bms1-git-deploy-v42-prod.yml. There are three:
audit-bms1-git-deploy-v42-prod.yml — workflow_dispatch, read-only, light.
RESOLVED (this PR): re-homed to [self-hosted, secrets-deploy]. That label’s hosts
(bms-4 54.36.123.110, vps-i1 217.154.82.162) are already on the allowlist, and the identical
self-hosted→bms-1 SSH pattern is proven by deploy-bms1-configs.yml + secrets-sync.yml. The
runner reconstructs id_bms1 from VPS_ROOT_SSH_KEY exactly as those do; the bms-4→bms-1 path
was verified live during this work. No behaviour change beyond egress IP.
ansible-drift.yml — weekly cron (Mondays 06:00 UTC) + workflow_dispatch; runs
ansible-playbook --check --diff against vps-i1,vps-h1,bms-4,bms-1. OPEN. Re-homing to a
self-hosted runner is non-trivial: the runner would need the full Ansible control-node env
(ansible, collections, all four hosts’ known_hosts) and inherits the existing bms-1 focal /
Python-3.9 fact-gathering caveat already documented in that file. Because it is a weekly cron,
closing port 22 before this is re-homed produces a recurring Monday failure on P0 production.
Recommended: re-home to [self-hosted, secrets-deploy] (light enough — --check only) after
confirming the ansible env is present on both label hosts, in a dedicated follow-up with a
manual workflow_dispatch test run before merge.
pinbox24-image-backup.yml — workflow_dispatch; streams docker save '<img>' | gzip over
SSH to the runner’s local disk, then uploads to Wasabi. OPEN, and it CANNOT use secrets-deploy:
docs/playbooks/gh-runner-assignment-policy.md explicitly forbids heavy-compute jobs on that
shared label. Options: (a) [self-hosted, bms4, heavy] (bms-4 egress is allowlisted; but a
multi-GB docker save adds disk/CPU pressure to the arbiter+n8n+AI host — verify free disk
first); or (b) keep it GitHub-hosted and give it a bms-1 access path that does not depend on a
static IP (e.g. a short-lived, workflow-scoped ufw allow opened via an already-allowlisted
self-hosted pre-step and torn down in a post/always step). Needs a design decision.
Gate 2 — developer-workstation IP (source #2, from #5604)
The architect’s workstation SSHes to bms-1 directly on a dynamic/undocumented ISP IP. This needs a
decision before the close, since a strict allowlist would otherwise remove the architect’s own
access. Options, roughly in order of preference:
Confirm-and-document: many residential/office ISP IPs are effectively static for long periods.
If the current IP has been stable, add it as a /32ufw allow ... comment "dev-workstation (#5642)"
and record it (privately — never commit the value). Lowest complexity; revisit if the ISP rotates.
Jump host via bms-4: drop direct workstation→bms-1 SSH; reach bms-1 as
ssh -J root@54.36.123.110 root@94.23.26.113 (bms-4 is allowlisted). No firewall change tracks the
workstation IP at all. Preferred if the IP is genuinely volatile.
Dynamic-DNS ufw rule: a ddclient-style hostname + a periodic job that rewrites the workstation
ufw rule when the IP changes. More moving parts; only if direct (non-jump) access is required.
VPN/bastion: heaviest; not justified for a single P0 EOL host on its way out.
This gate requires architect input — a queue worker cannot determine or choose the workstation IP.
Ready-to-execute procedure (DO NOT RUN until Gates 1 + 2 cleared and a prod window is scheduled)
Run from bms-4 over the verified id_bms1 path. Take a fresh numbered snapshot first, keep a second
SSH session open (per the checklist at the end of this doc), and confirm each allowlisted path still
connects before deleting the open-Anywhere rules.
# 0. Snapshot current state (compare against /root/ufw-status-before-5604.txt).ssh -i ~/.ssh/id_bms1 root@94.23.26.113 'ufw status numbered' | tee /root/ufw-status-before-5642.txt# 1. Confirm the 5 fleet /32 ALLOW rules from #5604 are present (they become the enforcing allowlist).# Add the dev-workstation /32 here IF Gate 2 resolved to "confirm-and-document":# ssh -i ~/.ssh/id_bms1 root@94.23.26.113 'ufw allow from <DEV_WS_IP> to any port 22 proto tcp comment "dev-workstation (#5642)"'# 2. Close the open-Anywhere SSH rules (v4 + v6). Delete by rule; re-check numbering after each delete.# ufw delete allow 22/tcp # removes the v4 + v6 Anywhere ALLOW pair# (verify with `ufw status numbered` that ONLY the /32 allowlist rules remain for port 22)# 3. Verify from EACH allowlisted path that SSH still connects (bms-4, vps-i1, + the re-homed workflows# via a manual workflow_dispatch smoke run) BEFORE closing the applying session.
Rollback (re-open immediately if any allowlisted path is locked out):
ssh -i ~/.ssh/id_bms1 root@94.23.26.113 'ufw allow 22/tcp' # restores open-Anywhere v4+v6# then re-investigate which allowlist entry was missing.
Summary of what #5642 delivered vs. deferred
Delivered (this PR): source #1 (audit workflow) re-homed to an allowlisted self-hosted runner;
full access inventory + ready procedure + rollback documented above.
Deferred (issue stays open, human-action): re-home ansible-drift.yml (Gate 1.2) and decide
pinbox24-image-backup.yml’s path (Gate 1.3); architect decision on the workstation IP (Gate 2);
then run the ufw close in a scheduled W3/W4 prod window with sign-off. No firewall change was made
by this worker.
Unit-name / journalmatch note (Ubuntu vs AlmaLinux)
The systemd unit differs by distro: on Ubuntu the SSH daemon runs as ssh.service, on AlmaLinux as sshd.service. The default fail2ban sshd filter uses journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd. In fail2ban the + separator is a logical OR, so the _COMM=sshd clause matches auth failures regardless of which unit name the daemon runs under — the Ubuntu ssh.service vs sshd.service difference is therefore harmless with the stock filter.
This was verified empirically on vps-h1 (OpenSSH_9.6p1):
The same 253 lines match whether the unit clause says sshd.service or ssh.service — confirming the filter catches real failures on this host. On Ubuntu 24.04’s OpenSSH 9.6p1, auth failures are still emitted with _COMM=sshd (there were nosshd-session-tagged lines in the journal), so no journalmatch override is needed.
Caveat for the future: newer OpenSSH releases split per-connection handling into an sshd-session process. If a future upgrade starts emitting auth failures under _COMM=sshd-sessionand you see fail2ban-client status sshd stuck at Total failed: 0 while journalctl -u ssh shows Failed password lines, broaden the jail’s journalmatch:
Then fail2ban-client reload and confirm Total failed climbs.
Why Total failed: 0 right after install is normal
fail2ban’s systemd backend only counts failures inside the live findtime window after the daemon starts; it does not retroactively ban from old journal history. So a freshly-started jail on a host that simply isn’t being hit at that moment correctly shows Total failed: 0 / Total banned: 0. (vps-i1 showed bans immediately only because it was under active attack during install.) Confirm wiring with fail2ban-regex as above rather than relying on the live counter right after install.