mongonoprimary-bms4-exporter-link-loss

Playbook: MongoNoPrimary firing while rs0 is actually healthy — bms-4 exporter link packet loss

Trigger

The MongoNoPrimary alert (absent(mongodb_mongod_replset_my_state == 1), rule monitoring/prometheus/rules/mongodb-replset.yml) is firing, but a direct check shows rs0 has a healthy writable PRIMARY. Origin incident: #7209 (2026-09-22).

This alert is driven entirely by the two mongodb-exporter containers that run on bms-4 (bms-4-mongodb-exporter-bms2-1 → bms-2, bms-4-mongodb-exporter-bms3-1 → bms-3). If those exporters cannot scrape, mongodb_mongod_replset_my_state==1 disappears from Prometheus and the absent(...) rule fires — even when the cluster is perfectly healthy. So a firing MongoNoPrimary is not proof of a real no-primary; it is proof the bms-4 exporters can’t reach the primary, which can be a pure monitoring-path problem.

rs0 topology (reference)

Node	IP	rs0 role	mongod host
bms-2	145.239.133.104	PRIMARY (voting)	own host
bms-3	51.68.155.224	SECONDARY (voting)	own host
bms-4	54.36.123.110	ARBITER + both exporters	own host

Step 0 — Is the cluster ACTUALLY down, or is it just the exporters? (do this FIRST)

Confirm the real state before touching anything. From bms-4, open a fresh authenticated connection directly to the reported primary (do not rely only on the local arbiter’s cached rs.status() — that can look healthy via stale heartbeats). Extract creds silently from SOPS:

export SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt   # or claude-runner-2
DEC=$(sops -d --input-type dotenv --output-type dotenv secrets/mongodb-bms.env.sops)
AU=$(echo "$DEC" | grep '^mongodb_rs0_admin_user=' | cut -d= -f2- | tr -d '"')
AP=$(echo "$DEC" | grep '^MONGODB_RS0_ADMIN_PASSWORD=' | cut -d= -f2- | tr -d '"'); unset DEC
mongosh --quiet "mongodb://145.239.133.104:27017/admin?directConnection=true&serverSelectionTimeoutMS=10000" \
  -u "$AU" -p "$AP" --eval 'print("primary="+db.hello().isWritablePrimary)'
unset AU AP

• primary=true → the cluster is fine; this is a monitoring/exporter problem → go to Step 1.
• Times out / not writable → a real replica-set problem → see mongodb-rs-heartbeat-block.md and mongodb-rs0-full-restore.md.

Step 1 — Confirm it’s the bms-4 → bms-2/bms-3 link, and characterise the loss

The signature of this incident is size-correlated packet loss on bms-4’s path to the mongo peers: small ops mostly pass, large multi-packet responses black-hole. Measure it (from bms-4 host):

C="mongodb://145.239.133.104:27017/admin?directConnection=true&serverSelectionTimeoutMS=5000"
s=0; for i in $(seq 1 10); do timeout 7 mongosh --quiet "$C" -u "$AU" -p "$AP" --eval 'db.hello().ok'        >/dev/null 2>&1 && s=$((s+1)); done; echo "small hello:        $s/10"
l=0; for i in $(seq 1 10); do timeout 7 mongosh --quiet "$C" -u "$AU" -p "$AP" --eval 'db.serverStatus().ok' >/dev/null 2>&1 && l=$((l+1)); done; echo "large serverStatus: $l/10"
# Prove it is NOT bms-4's NIC / internet egress:
ip -s link show eno1 | grep -A2 'RX:'                 # expect 0 errors / 0 dropped
for i in $(seq 1 5); do curl -s -o /dev/null -w '%{size_download}\n' 'https://speed.cloudflare.com/__down?bytes=100000'; done  # expect 100000 x5

Diagnosis = this playbook when: small ≫ large (e.g. 8/10 vs 1/10), NIC is clean, and internet 100 KB downloads are 5/5. That means the degradation is on the OVH-internal path bms-4 → bms-2/bms-3 only — a real network-layer problem, but not a MongoDB, docker, firewall, or exporter bug.

What it is NOT (things that look tempting but are dead ends)

• Not an MTU black hole. An MSS clamp on the forwarded path does not fix it — even --set-mss 1000 still lost ~70% of large transfers (#7209). Do not leave TCPMSS rules behind.
• Not a container/NAT issue. A fresh mongo:7.0 container on bms-4_default shows the same size-correlated failure as the host. Restarting/recreating the exporters does nothing for the loss.
• Not the stray-iptables DROP cause (mongodb-rs-heartbeat-block.md) — check anyway (iptables -L INPUT -n -v | grep 27017), but in #7209 there were none.
• Not conntrack exhaustion — check nf_conntrack_count vs _max, but it was 252/262144.

Step 2 — There is no bms-4-side fix; scope impact and escalate

The loss is on OVH’s internal network between the dedicated servers. bms-4 cannot fix it.

• Impact is monitoring-only as long as Step 0 shows a writable primary: rs0 heartbeats are small and mostly pass, so the arbiter vote and primary stay stable; bms-2↔bms-3 replication uses a different path (check rs.status() lag is small). Production apps on bms-1 use a different path too.
• If it persists (more than an hour or two): open an OVH ticket for packet loss on bms-4 54.36.123.110 → bms-2 145.239.133.104 / bms-3 51.68.155.224, and check whether OVH VAC / anti-DDoS mitigation is engaged on bms-2’s IP (a common OVH cause of exactly this size/rate-correlated drop pattern). The MongoNoPrimary alert clears on its own once the link heals.
• Add human-action to the alert issue for the network follow-up. Do not silence-and-forget — the exporters are a real blind spot on rs0 health while the link is degraded.

Step 3 — Watch out for the exporter-password footgun (secondary bug from #7209)

bms-4/docker-compose.yml on bms-4 builds the exporter URI as mongodb://prometheus:${MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED}@<host>:27017/admin?authSource=admin, interpolated from /opt/p24-infra/bms-4/.env at docker compose up time. As of #7209 that .env was missing MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED (a secrets-sync gap). Consequence: any docker compose up/--force-recreate on bms-4 bakes an empty/wrong password into the exporter → SCRAM-SHA-256 Authentication failed — a second, unrelated way for MongoNoPrimary to fire.

• The long-lived exporter container keeps the correct password it was created with, so this only bites when someone recreates it — easy to trigger accidentally while “fixing” the alert.
• Immediate restore (operational SOPS read, no SOPS write):

  cd /opt/p24-infra/bms-4
  export MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED=$(sops -d --input-type dotenv --output-type dotenv \
    <repo>/secrets/mongodb-bms.env.sops | grep '^MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED=' | cut -d= -f2- | tr -d '"')
  docker compose up -d --force-recreate mongodb-exporter-bms2 mongodb-exporter-bms3
  unset MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED

• Durable fix (secret-manager): ensure secrets-sync deploys the mongodb-bms.env.sops keys into the bms-4 .env target so the exporters survive a recreate without a manual export.

Refs

• Incident: #7209 (2026-09-22) — first occurrence; OVH network event at ~13:08Z.
• Related: mongodb-rs-heartbeat-block.md, mongodb-rs0-full-restore.md, mongotimeout-false-positive.md.

---

Audit Log — Log to infra_operations

import sys; sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
log_op(actor="claude", op_type="investigation", resource="mongodb-rs0",
       result="success",
       detail="MongoNoPrimary false-positive: rs0 healthy (bms-2 writable primary); root cause OVH-internal packet loss bms-4->bms-2/bms-3 degrading exporter scrapes. Fixed exporter password (.env secrets-sync gap). Network follow-up escalated.",
       env="bms-4", gh_issue=7209)