p24-infra Docs

mongodb-rs0-drop-detector-operations

2 min read

mongodb-rs0-drop-detector — Operations

Service: p24-rs0-drop-detector (systemd oneshot + timer) Hosts: bms-2 (145.239.133.104) and bms-4 (54.36.123.110) Script: scripts/mongodb-rs0-drop-detector.sh Units: deploy/p24-rs0-drop-detector.service, deploy/p24-rs0-drop-detector.timer Cadence: every minute (OnCalendar=*:0/1) Issue: #2471 Triage playbook: docs/playbooks/mongodb-rs0-heartbeat-drop-block.md

What it does

Detects raw iptables DROP/REJECT rules that block MongoDB rs0 heartbeats between replica-set members but stay invisible to ufw status (see the origin incident 2026-07-01). It scans iptables-save — the only view that reveals a UFW-hidden DROP — for any rule targeting DROP/REJECT and referencing a peer rs0 member IP, then alerts.

Read-only: the detector never modifies the firewall. Remediation is manual (playbook).

Alerting (edge-triggered)

State is tracked in /var/lib/p24-rs0-drop-detector/state:

TransitionAction
clean → dropDiscord alert (red) + GH bug issue (once)
drop → dropno-op (no per-minute spam)
drop → cleanDiscord “cleared” self-resolve (green)

Alerts use P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL and GH_TOKEN from the host env file.

Configuration

Environment overrides (optional):

VarDefaultPurpose
RS0_MEMBER_IPS145.239.133.104 51.68.155.224 54.36.123.110watch-set (space/comma separated)
STATE_DIR/var/lib/p24-rs0-drop-detectoredge-trigger state location

The script skips the host’s own IPs automatically (hostname -I).

Deploy (per host: bms-2 and bms-4)

Run as root on each host. Assumes the repo is checked out at /opt/p24-infra.

cd /opt/p24-infra && git pull
 
install -m 0755 scripts/mongodb-rs0-drop-detector.sh /opt/p24-infra/scripts/mongodb-rs0-drop-detector.sh
cp deploy/p24-rs0-drop-detector.service /etc/systemd/system/
cp deploy/p24-rs0-drop-detector.timer   /etc/systemd/system/
 
systemctl daemon-reload
systemctl enable --now p24-rs0-drop-detector.timer
 
# Fire once immediately to seed state + confirm it runs clean:
systemctl start p24-rs0-drop-detector.service
journalctl -u p24-rs0-drop-detector --no-pager -n 20

The unit ships three optional EnvironmentFile paths (bms-4/.env, bms-2/.env, monitoring/.env, all with a leading -), so the same unit works on either host — only the present one is loaded.

Verify

systemctl list-timers p24-rs0-drop-detector.timer --no-pager
cat /var/lib/p24-rs0-drop-detector/state        # 'clean' when healthy
journalctl -u p24-rs0-drop-detector --since "10 min ago" --no-pager

Test (safe, self-cleaning)

# Add a harmless DROP for a non-routed test IP, point the detector at it, run, then remove.
iptables -I INPUT -s 203.0.113.99 -p tcp --dport 27017 -j DROP
RS0_MEMBER_IPS=203.0.113.99 STATE_DIR=/tmp/rs0-test /opt/p24-infra/scripts/mongodb-rs0-drop-detector.sh
iptables -D INPUT -s 203.0.113.99 -p tcp --dport 27017 -j DROP
RS0_MEMBER_IPS=203.0.113.99 STATE_DIR=/tmp/rs0-test /opt/p24-infra/scripts/mongodb-rs0-drop-detector.sh  # resolve

Compliance

  • Registered in Supabase dev_r_services (compliance_workbook='yes').
  • Not AI-powered — no dev_r_ai_systems entry required.

Graph View

Backlinks