p24-infra Docs

mongotimeout-false-positive

3 min read

Playbook: [MongoTimeoutError] recurring false-positive GitHub issues

Trigger: GitHub issues titled [MongoTimeoutError] <host> (label bug) that recur / keep re-firing, auto-generated by the mezmo-alert-router n8n workflow (Sc2AKCdslinnSvWs) from the Mezmo MongoTimeoutError alert view (1ce14a4838). History: #1407/#1410/#1428/#1429/#1430/ 1433.

Key distinction — two completely different sources produce these:

Host on the issueSourceVerdict
bms-1pm2_v42-prod_production_err.log — Pinbox24 Node.js mongojs driver failing to reach rs0 (Server selection timed out after 30000 ms)GENUINE client-side connectivity error
bms-4 / vps-i1hourly-devops-triage.log, dev-issue-<N>.log — AI-agent transcripts that quote [MongoTimeoutError] verbatimFALSE POSITIVE self-referential feedback loop

bms-4 is an rs0 arbiter + n8n host and vps-i1 runs no Pinbox24 app — neither produces genuine mongojs timeouts. Any [MongoTimeoutError] bms-4 / vps-i1 issue is always self-ref noise.

Step 1 — Confirm source via Mezmo (from bms-4 or vps-i1)

cd /tmp/<worktree> # or /opt/p24-infra
export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
MEZMO_KEY=$(sops --decrypt --input-type dotenv --output-type dotenv \
  secrets/monitoring.env.sops | sed -n 's/^P24_INFRA_MEZMO_SERVICE_KEY=//p')
export MEZMO_KEY
python3 - <<'PY'
import os, requests, datetime as dt
from datetime import datetime, timezone
from collections import Counter
H={"Authorization":f"Token {os.environ['MEZMO_KEY']}"}
now=datetime.now(tz=timezone.utc); frm=int((now-dt.timedelta(hours=48)).timestamp()*1000); to=int(now.timestamp()*1000)
j=requests.get("https://api.mezmo.com/v2/export",headers=H,
  params={"from":frm,"to":to,"size":10000,"query":"MongoTimeoutError"},timeout=90).json()
c=Counter((l.get("_host"),l.get("_app") or l.get("_file")) for l in j.get("lines",[]))
for (h,a),n in c.most_common(): print(f"{h:8} {a:40} {n}")
PY
unset MEZMO_KEY
  • All lines from bms-1 / pm2_*-prod_*_err.log → genuine. Go to Step 3.
  • Lines from bms-4 / vps-i1 agent logs → false positive. Go to Step 2.

Step 2 — False positive (agent self-ref): verify the durable guards are in place

The durable fix (#1433) has two layers — verify both, re-apply if drifted:

  1. Alert-view host filter — the view query must exclude agent hosts:

    # GET /v1/config/view -> view 1ce14a4838 query must be:
#   "MongoTimeoutError -host:bms-4 -host:vps-i1"

    If it reverted to bare MongoTimeoutError, re-apply via PUT (see mezmo-operations.md §10: bodyTemplate must be an object, drop alertid, preserve both channel URLs). Wildcards do not work in Mezmo app: queries, so per-filename ingestion exclusions cannot catch dev-issue-<N>.log — the host filter is the only durable mechanism.

  2. Ingestion exclusions active — triage-log-self-ref (and the rest) must be active:true:

    P24_INFRA_MEZMO_SERVICE_KEY="$MEZMO_KEY" python3 scripts/mezmo-manage.py exclusions apply
# then confirm via GET /v1/config/ingestion/exclusions -> every p24 rule active:true

    Rules are created inactive by Mezmo; apply reconciles them to active (#1433).

Then close the FP issue with an evidence comment, no code change needed.

Step 3 — Genuine bms-1 timeout

Real Pinbox24 → rs0 connectivity failure. The view now requires a sustained burst (triggerlimit=10 / 5m) before opening a GH issue, so a live issue means a real, recurring problem. Agents cannot SSH bms-1 (root key is human-workstation only — see #1270). Diagnose from Mezmo (rs0 health on bms-2/bms-3, election storms, TLS) and escalate to the operator with human-action if rs0 itself is unhealthy. Do not add an ingestion exclusion for genuine bms-1 timeouts — that would hide a real outage.

Prevention (shipped in #1433)

  • MongoTimeoutError view query: -host:bms-4 -host:vps-i1 (drops agent self-ref).
  • n8n GH-issue channel: triggerlimit=10 / 5m (no issue on single transient).
  • mezmo-exporter metric pinbox24_mongo_timeouts_5m_total: query excludes agent hosts.
  • Pinbox24MongoTimeouts Prometheus alert: for: 10m on the now-genuine-only metric.
  • mezmo-manage.py exclusions apply reconciles inactive rules to active:true.

Graph View

Backlinks