p24-infra Docs

docker-port-firewall

7 min read

Playbook — DOCKER-USER firewall for container-published ports

Status: MECHANISM SHIPPED, NOT YET APPLIED on bms-4 (observe-first, #7660; partially implements #7609) Applies to: any Docker host that publishes ports on a public interface (bms-4 first; bms-1/2/3 and vps-h1 are follow-ups).

⚠️ Owner boundary (2026-10-09). The first cut of this mechanism (#7661, merged) used a single global FLEET_IPS=217.154.82.162 and a trailing blanket DROP. Applied as written it would have cut production Pinbox24 off RabbitMQ: bms-1 (94.23.26.113) holds live AMQP connections to bms-4 :5672, and docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md confirms W4 request handlers fail without that broker. The mechanism was corrected to observe-first and nothing has been installed on bms-4. Do not install enforcement blindly — follow the staged procedure below.

Why

Docker publishes container ports through its own nftables/iptables DNAT rules (the DOCKER chain), evaluated on the FORWARD path before ufw’s docker handling. A container port published on 0.0.0.0 is therefore reachable from the whole internet even while ufw is active with no rule allowing it — ufw only guards host-process (INPUT) listeners, not container-published (FORWARD/DNAT) ones. This is exactly how the MongoDB restore-drill port and, per the #7660 audit, the bms-4 exporters/RabbitMQ/n8n/pushgateway ports were all internet-reachable despite active ufw.

The only supported layer that can filter container-published ports is the DOCKER-USER chain, which Docker guarantees to traverse before its own rules and never flushes except on a full daemon restart.

bms-4 exposure + consumer map (audit 2026-10-09, corrected)

Every port below was published on 0.0.0.0/:: and WAN-reachable. The original consumer map was derived from the Prometheus scrape config only and therefore missed application clients — the cause of the RabbitMQ near-miss. The corrected map adds them:

PortContainerConsumer(s)Policy in this mechanism
80, 443traefikinternet (reverse proxy)public (always allowed)
8080cadvisorvps-i1 Prometheusfleet (managed)
9091pushgatewayvps-i1 Prometheus (local pushers use the docker net)fleet (managed)
9121redis-exportervps-i1 Prometheusfleet (managed)
9216 / 9217mongodb-exporter-bms2 / -bms3vps-i1 Prometheusfleet (managed)
9477wa-web-collectorvps-i1 Prometheusfleet (managed)
5679 / 5680 / 5681n8n worker 1/2/3vps-i1 Prometheus (metrics)fleet (managed)
5672rabbitmqvps-i1 blackbox (tcp_connect) + bms-1 94.23.26.113 production AMQP (evidence: rabbitmqctl list_connections peer_host showed 3 live connections from bms-1)NOT restricted by this mechanism
5678n8n mainvps-i1 Prometheus metrics; UI/webhooks via Traefik 443NOT restricted by this mechanism (pending counter evidence)
15672rabbitmq mgmtlocal onlyalready bound to 127.0.0.1 — correct ✓

5672 and 5678 are deliberately excluded from MANAGED_PORTS in bms-4/docker-user-firewall.conf, so P24-FW emits no rule for them and they pass through the chain’s final RETURN untouched — even in enforce mode. They keep their current WAN binding until counters (below) prove the full set of real sources and the owner approves restricting them.

No bms-4 port can be rebound to 127.0.0.1: every WAN-bound port is a legitimate cross-host consumer target, so the fix is the DOCKER-USER mechanism, not a rebind.

RabbitMQ default-credential posture (read-only check, 2026-10-09)

rabbitmqctl list_users on bms-4-rabbitmq-1 returns exactly one user: pinbox24 (administrator). The default guest account does not exist (deleted), so the default-credential remote-login vector is closed even though loopback_users is empty (empty is expected once guest is gone). The one remaining account must stay remotely reachable because bms-1 connects to it — which is precisely why 5672 is on the unrestricted list above. No change made; re-check with list_users after any RabbitMQ rebuild.

Mechanism (in repo)

  • scripts/docker-user-firewall.sh — builds an idempotent private P24-FW chain that DOCKER-USER jumps to first. Subcommands:
    • install — count-only (DEFAULT): per-port counting RETURN rules, never a DROP. Use this to observe who actually connects before restricting anything.
    • enforce — explicit, separate: on managed ports, RETURN the allow-listed source(s) and DROP everything else. Unmanaged ports (5672/5678) still pass the final RETURN.
    • confirm — cancel the dead-man auto-rollback so the current rules persist.
    • remove — rollback (also cancels the dead-man).
    • status / counters — inspect the chain and per-rule packet counters.
    • render count|enforce — print the exact rules without root/without applying (dry run; used by scripts/tests/test_docker_user_firewall.py and for review).
  • bms-4/docker-user-firewall.conf — PUB_IF=eno1, PUBLIC_PORTS=80,443, MANAGED_PORTS (5672/5678 excluded), per-port PORT_ALLOW_<port> source lists, DEADMAN_MINUTES=15.
  • bms-4/docker-user-firewall.service + .timer — persistence layer (reassert after a docker daemon restart via PartOf=docker.service, and every 15 min). Runs count-only install with P24_FW_NO_DEADMAN=1. Enable only AFTER a confirmed manual apply.
  • scripts/check-docker-port-exposure.sh + bms-4/docker-port-allowlist.conf — regression guard: fails if any container publishes a WAN port not in the allowlist (each entry records consumer + owner).

Every rule is scoped to -i eno1, so inter-container (br-*/docker0) and host-local (lo) traffic is never affected. The chain’s final rule is always RETURN, so a port that is not in MANAGED_PORTS can never receive a DROP.

Observe → enforce procedure (ROOT, maintenance window)

Hard pre-conditions (#7660): do nothing on bms-4 before the Q4 restore drill (Sat 2026-10-10 01:30 UTC) and the arbiter restart (Sun 2026-10-11 03:30 UTC) have both finished. Keep a second SSH session open (OVH KVM console as the recovery path) before any apply. Run the W3/W4 health check (docs/playbooks/pinbox24-w3-w4-health-verification.md) before and after every step; roll back immediately if it degrades.

Phase 1 — observe (count-only, no traffic dropped)

cd /opt/p24-infra && git pull
# Dry-run review first (no root, no apply):
bash scripts/docker-user-firewall.sh render count bms-4/docker-user-firewall.conf
 
# Apply count-only. Harmless (no DROP), but it arms a 15-min dead-man rollback:
sudo scripts/docker-user-firewall.sh install bms-4/docker-user-firewall.conf
sudo scripts/docker-user-firewall.sh status  bms-4/docker-user-firewall.conf
# Verify monitoring is unaffected (it will be — nothing is dropped), then make it persist:
sudo scripts/docker-user-firewall.sh confirm bms-4/docker-user-firewall.conf

Leave it for 24–48 h, then read who actually connected to each managed port:

sudo scripts/docker-user-firewall.sh counters bms-4/docker-user-firewall.conf

For each managed port compare the per-source RETURN counter (known source, e.g. vps-i1) against the per-port catch-all RETURN counter (all NEW connections). A non-trivial gap means an unlisted source is connecting — identify it before enforcing, and add it to PORT_ALLOW_<port> (or escalate). Do the same sanity read for 5672/5678 if you are ever asked to bring them in-scope.

Phase 2 — enforce (DROP), one port at a time, owner-approved only

Enforcement is not part of #7660 and must not be switched on without owner sign-off on the observed counters. When approved:

# Review the DROP rules first:
bash scripts/docker-user-firewall.sh render enforce bms-4/docker-user-firewall.conf
# Apply (arms the dead-man again):
sudo scripts/docker-user-firewall.sh enforce bms-4/docker-user-firewall.conf

Then run the verification below. If anything degrades, do nothing — the dead-man auto-removes the chain after DEADMAN_MINUTES. If healthy, confirm within that window.

Verify (every step, before confirm)

# A. From an UNLISTED host (NOT vps-i1) — in ENFORCE these must FAIL/time out (in count-only they stay OPEN):
for p in 8080 9091 9121 9216 9217 9477 5679 5680 5681; do
  timeout 5 bash -c "echo > /dev/tcp/54.36.123.110/$p" 2>/dev/null \
    && echo "$p OPEN" || echo "$p filtered"
done
# 443 must stay OPEN (Traefik). 5672 and 5678 must stay OPEN (unmanaged, by design).
 
# B. From vps-i1 (217.154.82.162) — scrapes must still connect in both modes:
#    ssh root@217.154.82.162 'for p in 8080 9121 9216 9217 9091 9477 5679 5680 5681; do timeout 5 bash -c "echo > /dev/tcp/54.36.123.110/$p" && echo "$p ok"; done'
 
# C. Prometheus targets for bms-4 stay UP (no new *Down alerts for 2 scrape intervals):
#    up{instance=~"54.36.123.110:.*"} == 1 in Grafana / Thanos.
 
# D. W3/W4 health: docs/playbooks/pinbox24-w3-w4-health-verification.md — must stay green.
#    RabbitMQ (5672) consumers from bms-1 unaffected: ssh root@54.36.123.110 \
#      'docker exec bms-4-rabbitmq-1 rabbitmqctl list_connections peer_host' still shows 94.23.26.113.

Dead-man’s-switch

Any live install/enforce schedules a transient systemd unit (p24-fw-deadman) that runs docker-user-firewall.sh remove after DEADMAN_MINUTES (default 15). If you apply a rule set that breaks monitoring or W3/W4 and then lose your session, it rolls itself back automatically. To keep a verified-healthy rule set, run confirm within the window (cancels the timer). remove also cancels it. The persistence .service/.timer set P24_FW_NO_DEADMAN=1 so re-asserts never self-remove.

Rollback (immediate, if any bms-4 target drops)

sudo scripts/docker-user-firewall.sh remove bms-4/docker-user-firewall.conf
# Ports are WAN-reachable again (pre-#7660 state). Re-diagnose PORT_ALLOW_* / PUB_IF before retrying.

After a confirmed-healthy apply, log it:

source /opt/p24-infra/scripts/lib/log_op.sh
log_op "human" "config_change" "docker-user-firewall" "success" \
  "DOCKER-USER <count-only|enforce> installed; external ports as expected, vps-i1 scrapes UP, W3/W4 green" "bms-4"

Follow-ups (other hosts, #7660 step 5)

Run the same read-only audit (docker ps + external TCP connect) on bms-1, bms-2, bms-3, vps-h1 and any Docker-running dev machine, and open a per-host hardening issue. Do not touch the rs0 members’ own firewall rules on bms-2/bms-3 without the db-admin role and a maintenance window.

Graph View