p24-infra Docs

bms2-mongodb-exporter-decommission

5 min read

Playbook: Decommission the redundant bms-2 local mongodb_exporter.service

Status: ACTIVE — remediation runbook for #5859 Applies to: bms-2 (145.239.133.104, OVH ns3087638, MongoDB rs0 voting member) Requires: SSH to bms-2 as ubuntu (canonical user — never enumerate other users, fail2ban hazard per server-ssh-connectivity.md) with a key authorised on that host (developer key / /root/.ssh/id_ed25519 on bms-4 or vps-i1 root context).

TL;DR

The mongodb_exporter.service systemd unit running locally on bms-2 is redundant/legacy. It is not a Prometheus target, it is managed by no ansible role and by no secrets-sync path, and it duplicates the canonical exporter that already works. Its only observable effect today is harm: a stale, hardcoded prometheus password in its inline ExecStart makes it fail MongoDB SCRAM auth on every scrape (~240 failures/hour), which is filling mongod.log.

Fix = decommission it (stop + disable), not credential-manage it. Then add logrotate for mongod.log. rs0 metrics are unaffected — see “Why decommission, not fix” below.

Why decommission, not fix (root-cause correction for #5859)

The #5859 issue body assumed “rs0 MongoDB metrics from the bms-2 local exporter have been dark for ~3 days”. That premise is incorrect. The canonical rs0 metrics do not come from any exporter running on bms-2:

FactEvidence
Prometheus scrapes the bms-4 containers, not bms-2monitoring/prometheus/prometheus.yml job mongodb → 54.36.123.110:9216 (label role: primary, i.e. bms-2’s data) and :9217 (bms-3). The only thing scraped on bms-2 is 145.239.133.104:9100 (node_exporter).
The bms-2-data exporter is healthybms-4-mongodb-exporter-bms2-1 container UP 4+ weeks; curl localhost:9216/metrics → mongodb_up{cluster_role="mongod"} 1 and 3072 mongodb_* series.
The bms-2 local systemd exporter is scraped by nothingNo repo config references a bms-2 exporter port other than :9100.
It is managed by nothingNo ansible role references mongodb_exporter; no secrets-sync.yml job targets it. This is why the 2026-08-04 prometheus-password rotation never reached it — it was invisible to the (correct) distribution step.

So: rs0 monitoring is not degraded. The local exporter is a leftover from an earlier, now-superseded design (remote container exporters on bms-4, see mongodb-credential-rotation.md). Credential-managing it would perpetuate a duplicate service and keep two exporters hitting the same mongod. The correct action is to remove it.

Step 0 — Confirm the canonical exporter is healthy BEFORE removing anything

Run on bms-4 (no bms-2 access needed):

docker ps --filter name=mongodb-exporter --format '{{.Names}}\t{{.Status}}'
# expect: bms-4-mongodb-exporter-bms2-1  Up ...
curl -s http://localhost:9216/metrics | grep -F 'mongodb_up'   # expect: mongodb_up{...} 1

If mongodb_up is not 1, STOP — do not remove the local exporter; investigate the container first (see mongodb-exporter-uri-special-chars.md).

Step 1 — Decommission the local exporter on bms-2

Secret-safety: the credential is inline in the unit’s ExecStart/Environment. Never run systemctl cat, systemctl status, cat the unit file, or anything that prints the process cmdline/env — those dump the password. The commands below never print the unit body.

# PLAYBOOK: bms2-mongodb-exporter-decommission.md
ssh ubuntu@145.239.133.104 'sudo systemctl disable --now mongodb_exporter.service && \
  sudo systemctl is-active mongodb_exporter.service; \
  sudo systemctl is-enabled mongodb_exporter.service'
# expect: inactive / disabled (is-active exits non-zero → "inactive")

Then remove the unit fragment so the stale credential is not left at rest on disk. Find the path without printing contents, then delete:

# PLAYBOOK: bms2-mongodb-exporter-decommission.md
ssh ubuntu@145.239.133.104 'FRAG=$(systemctl show mongodb_exporter.service -p FragmentPath --value); \
  echo "fragment: $FRAG"; \
  sudo rm -f "$FRAG" && sudo systemctl daemon-reload && echo "unit removed"'

If the exporter binary was installed only for this unit, it can be left in place (harmless) or removed separately — not required for this fix.

Step 2 — Verify the mongod auth-failure spam stops

# PLAYBOOK: bms2-mongodb-exporter-decommission.md
# Count prometheus storedKey-mismatch failures in the last 2 min AFTER decommission (should trend to 0).
ssh ubuntu@145.239.133.104 'sudo journalctl -u mongod --since "-2min" 2>/dev/null | grep -c "prometheus" || echo 0'

The bms-4 container exporter still authenticates fine (it uses the current MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED from secrets/mongodb-bms.env.sops), so mongod.log will now only show the (successful) bms-4 exporter connections.

Step 3 — Add logrotate for mongod.log (disk-fill hardening)

bms-2 /var/log/mongodb/mongod.log was ~11 GB and unrotated — a disk-fill risk on the rs0 PRIMARY. Install a logrotate policy that uses MongoDB’s logRotate reopen (so mongod keeps writing to the same path after rotation):

# PLAYBOOK: bms2-mongodb-exporter-decommission.md
ssh ubuntu@145.239.133.104 'sudo tee /etc/logrotate.d/mongodb >/dev/null <<'"'"'CONF'"'"'
/var/log/mongodb/mongod.log {
    daily
    rotate 14
    size 100M
    compress
    delaycompress
    missingok
    notifempty
    create 0600 mongodb mongodb
    sharedscripts
    postrotate
        /bin/kill -SIGUSR1 $(pidof mongod) 2>/dev/null || true
    endscript
}
CONF
sudo chmod 0644 /etc/logrotate.d/mongodb && sudo logrotate --debug /etc/logrotate.d/mongodb'

SIGUSR1 tells mongod to reopen its log file (equivalent to db.adminCommand({logRotate:1})), which is the correct rotation signal when systemLog.logRotate is the default rename. Verify the config parses (--debug above prints the plan without rotating). To force a first rotation of the existing 11 GB file immediately:

# PLAYBOOK: bms2-mongodb-exporter-decommission.md
ssh ubuntu@145.239.133.104 'sudo logrotate --force /etc/logrotate.d/mongodb && \
  sudo ls -la /var/log/mongodb/ && df -h /'

Do the same on bms-3 (51.68.155.224, ubuntu) if it also lacks a mongodb logrotate policy — check with ssh ubuntu@51.68.155.224 'ls /etc/logrotate.d/ | grep -i mongo || echo none'.

Step 4 — Log the operation

import sys; sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
log_op("claude", "config_change", "bms2-mongodb_exporter.service", "success",
       "Decommissioned redundant local mongodb_exporter + added mongod.log logrotate", "bms-2", 5859)

Distribution/rotation checklist note

Do NOT add the bms-2 local exporter to the MONGODB_RS0_PROMETHEUS_PASSWORD distribution list — it is being decommissioned. The complete consumer list for that credential remains the two bms-4 containers (mongodb-exporter-bms2, mongodb-exporter-bms3), fed by secrets-sync.yml into /opt/p24-infra/bms-4/.env and restarted per mongodb-credential-rotation.md Step 2d. Once Step 1 here is done, the credential-liveness probe class will no longer diverge from actual consumers.

Graph View

Backlinks