SSH Hardening — Operations Workbook

SSH brute-force mitigation applied to both VPS hosts (vps-i1 IONOS / AlmaLinux 9.7 and vps-h1 Hostinger / Ubuntu 24.04). Covers the hardened sshd policy and the fail2ban jail that bans abusive source IPs.


Why

Both hosts were receiving ongoing internet SSH brute-force traffic — tens of thousands of failed login attempts per day from bot IP ranges. No password login ever succeeded (all real access is publickey only), but the noise inflated auth.log/secure and the SSHAuthFailures Prometheus rule, and represented standing risk. The hardening below removes password/keyboard-interactive auth as an attack surface entirely and auto-bans repeat offenders.


What was hardened

SettingValueEffect
PasswordAuthenticationnoPassword logins refused — publickey only
PermitRootLoginprohibit-passwordroot may log in by key, never by password
KbdInteractiveAuthenticationnoDisables PAM keyboard-interactive fallback (closes the password backdoor)
fail2ban sshd jailenabledBans source IPs after repeated auth failures

All real access remains publickey — see docs/vps-i1-operations.md and docs/hostinger-runbook.md for the per-host SSH key matrix. No legitimate workflow used password auth, so this change is non-breaking.


sshd settings + file locations per host

vps-i1 (IONOS / AlmaLinux 9.7)

The hardened settings live in a drop-in (so they survive package upgrades to the base /etc/ssh/sshd_config):

/etc/ssh/sshd_config.d/00-hardening.conf
PasswordAuthentication no
PermitRootLogin prohibit-password
KbdInteractiveAuthentication no

Apply / reload after edits:

sshd -t                 # validate config — MUST pass before reload
systemctl reload sshd

vps-h1 (Hostinger / Ubuntu 24.04)

The PasswordAuthentication no / PermitRootLogin prohibit-password / KbdInteractiveAuthentication no settings were already applied by Hostinger cloud-init (typically in /etc/ssh/sshd_config.d/50-cloud-init.conf and the base config). No drop-in was added for the auth policy — only fail2ban was installed. Verify the effective policy with:

sshd -T | grep -Ei 'passwordauthentication|permitrootlogin|kbdinteractive'

Expected output:

passwordauthentication no
permitrootlogin prohibit-password
kbdinteractiveauthentication no

If any value drifts, add /etc/ssh/sshd_config.d/00-hardening.conf with the three lines above (drop-ins are read in lexical order; a 00- prefix wins over cloud-init’s 50-).


fail2ban jail config

fail2ban is a systemd-managed daemon installed on both hosts. The sshd jail config:

[sshd]
enabled  = true
backend  = systemd
bantime  = 1h
findtime = 10m
maxretry = 5
  • maxretry = 5 — 5 failures within findtime
  • findtime = 10m — …counted over a 10-minute sliding window…
  • bantime = 1h — …triggers a 1-hour IP ban (via the firewall / nftables ban action).
  • backend = systemd — fail2ban reads auth events from the systemd journal rather than tailing a log file. This matters for Ubuntu 24.04 (see caveat below).

Jail config file location is the standard fail2ban layout:

  • AlmaLinux: /etc/fail2ban/jail.d/sshd.local (or jail.local)
  • Ubuntu: /etc/fail2ban/jail.d/sshd.local (or jail.local)

Enable + start the service (idempotent):

systemctl enable --now fail2ban

How to check status

# Is the daemon up?
systemctl status fail2ban
 
# Jail summary — currently banned IPs, total failures, total bans
fail2ban-client status sshd

Example fail2ban-client status sshd output:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  |- Total failed:     48211
|  `- File list:        <journal>
`- Actions
   |- Currently banned: 7
   |- Total banned:     1902
   `- Banned IP list:   45.95.147.x 193.32.162.x ...

How to unban an IP

# Remove a single IP from the sshd jail ban list
fail2ban-client set sshd unbanip <IP>
 
# Unban everything in the sshd jail (use sparingly)
fail2ban-client unban --all

Whitelist trusted IPs/CIDRs permanently with ignoreip in the jail config (e.g. the developer workstation or CI egress) so they are never banned:

[sshd]
ignoreip = 127.0.0.1/8 ::1 <trusted-cidr>

Unit-name / journalmatch note (Ubuntu vs AlmaLinux)

The systemd unit differs by distro: on Ubuntu the SSH daemon runs as ssh.service, on AlmaLinux as sshd.service. The default fail2ban sshd filter uses journalmatch = _SYSTEMD_UNIT=sshd.service + _COMM=sshd. In fail2ban the + separator is a logical OR, so the _COMM=sshd clause matches auth failures regardless of which unit name the daemon runs under — the Ubuntu ssh.service vs sshd.service difference is therefore harmless with the stock filter.

This was verified empirically on vps-h1 (OpenSSH_9.6p1):

$ fail2ban-regex --journalmatch "_SYSTEMD_UNIT=sshd.service + _COMM=sshd" systemd-journal sshd
Lines: 1161 lines, 722 ignored, 253 matched, 186 missed

The same 253 lines match whether the unit clause says sshd.service or ssh.service — confirming the filter catches real failures on this host. On Ubuntu 24.04’s OpenSSH 9.6p1, auth failures are still emitted with _COMM=sshd (there were no sshd-session-tagged lines in the journal), so no journalmatch override is needed.

Caveat for the future: newer OpenSSH releases split per-connection handling into an sshd-session process. If a future upgrade starts emitting auth failures under _COMM=sshd-session and you see fail2ban-client status sshd stuck at Total failed: 0 while journalctl -u ssh shows Failed password lines, broaden the jail’s journalmatch:

[sshd]
backend      = systemd
journalmatch = _SYSTEMD_UNIT=ssh.service + _COMM=sshd + _COMM=sshd-session

Then fail2ban-client reload and confirm Total failed climbs.

Why Total failed: 0 right after install is normal

fail2ban’s systemd backend only counts failures inside the live findtime window after the daemon starts; it does not retroactively ban from old journal history. So a freshly-started jail on a host that simply isn’t being hit at that moment correctly shows Total failed: 0 / Total banned: 0. (vps-i1 showed bans immediately only because it was under active attack during install.) Confirm wiring with fail2ban-regex as above rather than relying on the live counter right after install.


Verification checklist (post-change / post-reboot)

# 1. sshd refuses passwords (should print "permission denied (publickey)")
ssh -o PreferredAuthentications=password -o PubkeyAuthentication=no root@<host> true
 
# 2. effective sshd policy
sshd -T | grep -Ei 'passwordauthentication|permitrootlogin|kbdinteractive'
 
# 3. fail2ban running and counting
systemctl is-active fail2ban
fail2ban-client status sshd

Confirm your publickey login still works in a separate session before closing the one you used to apply the change.