Playbook: MongoDB rs0 heartbeat blocked by stray runtime iptables DROP

Trigger

A MongoDB rs0 replica-set member cannot exchange heartbeats with the PRIMARY and/or ARBITER because a runtime-only iptables INPUT DROP rule silently blocks its inbound 27017 traffic. The rule is present in the live packet filter but absent from ufw status and /etc/iptables/rules.v4, so it is invisible to every normal config check and would vanish on reboot — making it extremely easy to overlook.

Symptoms:

Replication lag climbs without bound on one member (observed ~10h during the 2026-07-01 incident)
rs.status() shows the blocked member with health: 0, a stale optimeDate, and a lastHeartbeatMessage such as connection refused / Couldn't get a connection ... timed out
Applications using { w: "majority" } hang indefinitely when wtimeout is 0 (the majority can never be acknowledged) — e.g. Pinbox24 auth infinite hang (#2376)
ufw status and /etc/iptables/rules.v4 look clean — the DROP is only in runtime

Discovered: 2026-07-01. Stray INPUT DROP rules for 51.68.155.224 (bms-3) on port 27017 existed on bms-2 (PRIMARY) and bms-4 (ARBITER), inserted before the UFW rules and never persisted. bms-3 could not heartbeat either voting member → ~10h lag.

rs0 topology (reference)

Node	IP	rs0 role
bms-2	`145.239.133.104`	PRIMARY (voting)
bms-3	`51.68.155.224`	SECONDARY (voting)
bms-4	`54.36.123.110`	ARBITER

Every member must reach every other member on 27017. A DROP on any pair breaks heartbeats in that direction.

Confirm

The rule is runtime-only, so ufw status is not sufficient. You must inspect the live INPUT chain with packet counters:

# On EACH voting member (bms-2, bms-4 — and bms-3 for completeness):
ssh ubuntu@145.239.133.104   # bms-2 (repeat for bms-4 / bms-3)
 
# 1. Live INPUT chain — look for DROP on 27017 with a NON-ZERO packet count
sudo iptables -L INPUT -n -v --line-numbers | grep -E '27017|DROP'
 
# 2. Cross-check against the PERSISTED ruleset — a DROP present in (1) but ABSENT
#    here is the smoking gun (runtime-only, will disappear on reboot)
sudo iptables-save | grep 27017 || echo "no persisted 27017 rule"
 
# 3. Confirm the replication symptom from the PRIMARY
mongosh --quiet --eval '
  rs.status().members.forEach(m =>
    print(m.name, "state=" + m.stateStr, "health=" + m.health,
          "lag_s=" + ((rs.status().date - m.optimeDate)/1000),
          "hb=" + (m.lastHeartbeatMessage || "-")))'

Problem indicator: a line like DROP tcp src=51.68.155.224 dpt:27017 (66553 packets) in step 1 that has no counterpart in step 2, plus a member showing health=0 and large lag_s in step 3.

Step-by-step fix

# On the member that holds the stray DROP (bms-2 and bms-4 in the 2026-07-01 incident):
ssh ubuntu@145.239.133.104
 
# 1. Delete the stray runtime DROP (repeat per offending source IP)
sudo iptables -D INPUT -s 51.68.155.224 -p tcp --dport 27017 -j DROP
 
# 2. Verify the DROP is gone and traffic now ACCEPTs
sudo iptables -L INPUT -n -v --line-numbers | grep 27017
 
# 3. Confirm the blocked member rejoins and lag drains (from PRIMARY)
mongosh --quiet --eval 'rs.status().members.forEach(m =>
  print(m.name, m.stateStr, "health=" + m.health,
        "lag_s=" + ((rs.status().date - m.optimeDate)/1000)))'
# Expect: blocked member back to SECONDARY, health=1, lag_s → 0

Once the member is healthy again, restore the application write concern if it was relaxed during the incident (e.g. Pinbox24 back to { w: "majority", wtimeout: 5000 } from a degraded wtimeout: 0).

Root cause

The DROP rules were inserted directly into the runtime INPUT chain (likely during an earlier manual “security isolation” of bms-3) and were never persisted to /etc/iptables/rules.v4 nor expressed as UFW rules. Because they sat before the UFW jump, first-match-wins dropped the heartbeat packets. Being runtime-only, they:

did not appear in ufw status
did not appear in iptables-save / /etc/iptables/rules.v4
would have silently disappeared on the next reboot

so no config-diff or UFW audit could ever have caught them — only iptables -L INPUT -n -v did.

Prevention

Runtime-vs-persisted iptables drift monitor (nightly-infra-check Phase 2b): the nightly check now SSHes each rs0 node and raises a P1 when a peer rs0 IP is DROP’d on 27017 in the live INPUT chain but is not present in iptables-save. This is the only check that would have caught the 2026-07-01 rules.
rs0 heartbeat/lag health (nightly-infra-check Phase 2): the rs.status() parse now flags any member with health != 1, replication lag > 300s, or a non-empty lastHeartbeatMessage — so a blocked member surfaces within a day instead of after a multi-hour outage.
Never insert ad-hoc iptables DROP rules on production without persisting them. If a member must be isolated, express it as a UFW rule (ufw deny) so it is visible in ufw status and survives reboot — and remove it explicitly when the isolation ends.
After any manual firewall change on an rs0 node: run sudo iptables-save > /etc/iptables/rules.v4 (or netfilter-persistent save) so runtime and persisted state never diverge.

Escalation

If deleting the DROP does not restore heartbeats:

Check for DROP rules in both directions and on all three nodes (the blocked member may also be dropping the peers).
Confirm no fail2ban / nftables layer is re-inserting the rule (journalctl -u fail2ban --since -1h, nft list ruleset | grep 27017).
Verify the port is actually listening: ss -tlnp | grep 27017.
From the blocked member, test raw TCP to each peer: nc -vz -w5 145.239.133.104 27017 and nc -vz -w5 54.36.123.110 27017.
If lag persists after connectivity is restored, see mongodb-rs0-full-restore.md for resync options.

Refs

Incident: #2392 (this playbook’s origin), related to #2376 (w3 auth hang) + bms-3 recovery 2026-07-01
Related: mongotimeout-false-positive.md, mongodb-rs0-full-restore.md

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="mongodb-rs0",
    result="success",  # "success" | "failed" | "skipped"
    detail="MongoDB rs heartbeat block resolved — replica set member reconfigured",
    env="bms-2",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'mongodb-rs0', 'success', 'MongoDB rs heartbeat block resolved — replica set member reconfigured', 'bms-2')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

mongodb-rs-heartbeat-block