p24-infra Docs

mailgun-mongodb-stale-credential-hang

11 min read

Playbook: mailgun-v42-prod — Stale MongoDB Credential Causes Silent Webhook Hang

Incident date: 2026-07-10 Resolved (emergency): 2026-07-10 — container recreated via docker-compose (rename old + docker-compose up -d) Resolved (permanent): 2026-07-10, same day — mailgun-v42-prod switched from the rs0 admin credential to the w4_app credential (same user v42-prod already uses), CI restart step fixed to use --force-recreate, and the two containers’ rotation is now coupled in one operation. See “Permanent Fix” section below. Severity: P0 — 100% inbound email → AI-Logs pipeline data loss, zero error visible to the sender Services affected: mailgun-v42-prod (bms-1, 94.23.26.113) Symptom reported by user: sent ~450 emails with file attachments to rechnung-AI@integrations-eu.pinbox24.com, zero AI-Logs records created, no error surfaced anywhere

Symptoms

  • Mailgun received and stored every email correctly (events?event=stored shows 100% accepted)
  • Every webhook POST /api/v1/integration/add to mailgun-api.w4.pinbox24.com times out — nginx-proxy logs upstream timed out (110: Connection timed out) → 504 to Mailgun
  • Inside mailgun-v42-prod, PM2’s own request log shows the request entering but never completing: POST /api/v1/integration/add - - ms - - (no status code, no duration)
  • mailgun-v42-prod container itself looks perfectly healthy: docker ps shows Up, PM2 shows both cluster workers online, low CPU/mem, RestartCount=0, not OOM-killed
  • No new error appears in the app’s error log for each failed request — only a handful of MongoNetworkError: ... Authentication failed entries right at container startup, then silence, because the driver’s connection pool enters a dead state and every subsequent request just hangs waiting on a connection that will never be granted (no per-request timeout on the DB call)

This combination — container “healthy” by every naive check (docker ps, PM2 status, resource usage) but 100% of real traffic silently hanging — is why it went undetected: Grafana/uptime-style health checks (GET /health, GET /) still return 200 because they don’t touch MongoDB.

Root Cause

  1. mailgun-v42-prod reads MONGODB_URL from /root/mailgun-prod/mailgun-environment.env once, at container start (baked into the container’s process environment).
  2. At some point after the container started (2026-07-09 00:27 UTC), the w4_app MongoDB password was rotated / corrected and the on-disk env file was updated (2026-07-09 23:11 UTC) — but nothing restarted the container.
  3. From that point on, the container was running with a MongoDB credential that no longer matched what MongoDB rs0 (bms-2 PRIMARY) expects → every DB connection attempt fails Authentication failed.
  4. The mailgun app code has no timeout/circuit-breaker around the MongoDB call in the integration/add handler, so each request just hangs on a dead connection pool forever instead of failing fast.
  5. nginx-proxy’s proxy_read_timeout eventually expires (~60s) and returns 504 to Mailgun — but by then the request has already been silently dropped; nothing was ever written to w4_db.regRecords.

Root cause in one line: MongoDB credential rotated on disk, container never recreated to pick it up → every webhook silently hangs → zero records, zero errors, zero alerts.

How to confirm (read-only, no secret values printed)

# On bms-1 — compare the env file on disk vs what's actually loaded in the running container.
# Hashes differing = stale credential confirmed. This never prints the secret itself.
grep "^MONGODB_URL=" /root/mailgun-prod/mailgun-environment.env | sha256sum
docker exec mailgun-v42-prod sh -c "cat /proc/1/environ | tr '\0' '\n' | grep '^MONGODB_URL=' | sha256sum"
# If these two hashes differ -> container is running a stale credential.
 
# Confirm the hang pattern in nginx-proxy:
docker logs nginx-proxy --since 1h 2>&1 | grep "mailgun-api.w4.pinbox24.com" | grep -c "upstream timed out"
 
# Confirm requests enter but never complete in the app itself:
docker exec mailgun-v42-prod pm2 logs mailgun-backend --lines 30 --nostream
# Look for lines like "POST /api/v1/integration/add - - ms - -" (no status/duration = never returned)
 
# Confirm the actual auth failure (only visible near container start, in the err log):
docker exec mailgun-v42-prod sh -c "grep -c MongoNetworkError /var/log/mailgun-v42-prod/pm2/pm2_mailgun-v42-prod_production_err.log"

Also use scripts/mailgun-pipeline-analysis.ps1 -Date <yyyy-MM-dd> for a fast three-stage check (Mailgun events accepted → PM2 webhook call count → MongoDB record count). Note: as of 2026-07-10 this script has a PowerShell parse bug (embedded || inside an escaped bash string, plus a mojibake em-dash) that prevents it from running as-is — the manual commands above were used instead. Fix the script before relying on it again.

Fix

docker restart is not sufficient — it reuses the environment baked in at container creation and does not re-read the host env file. You must recreate the container.

# On bms-1 — recreate via docker-compose so the fresh env file + bind-mounted patches are applied.
# Rename (not remove) the old container first — avoids any name conflict without touching prod data.
docker stop mailgun-v42-prod
docker rename mailgun-v42-prod mailgun-v42-prod-stale
cd /root/mailgun-prod
CONTAINER_NAME=mailgun-v42-prod IMAGE_NAME=mailgun-v42-prod:latest CLUSTER_NETWORK=test-net docker-compose up -d
 
# Verify the new container loaded the CORRECT credential (hashes must now match):
grep "^MONGODB_URL=" /root/mailgun-prod/mailgun-environment.env | sha256sum
docker exec mailgun-v42-prod sh -c "cat /proc/1/environ | tr '\0' '\n' | grep '^MONGODB_URL=' | sha256sum"
 
# Verify the duplicate-check bind-mounted patch survived recreation (see
# pinbox24-mailgun-duplicate-check-fix.md):
docker exec mailgun-v42-prod sh -c "grep -c 'PATCH v3' /app/helper/integration.helper.js"
docker inspect mailgun-v42-prod --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'
 
# Watch it actually process the backlog:
docker logs nginx-proxy --since 2m 2>&1 | grep "mailgun-api.w4.pinbox24.com"
# Expect 200s (after ~10s of PM2 cluster warm-up, a handful of transient 502s are normal)
docker exec mailgun-v42-prod pm2 logs mailgun-backend --lines 20 --nostream

Once the old mailgun-v42-prod-stale container is confirmed unnecessary, remove it in a later session (renamed + stopped, so it carries zero risk sitting there in the meantime).

Mailgun’s own retry behavior

Mailgun retries a failed store(notify=...) webhook automatically for several hours, so most of the backlog from a multi-hour outage self-heals once the container is fixed — no need to manually resend every email. Verify backlog drain with:

// Inside v42-prod container: count today's records for the affected inbound address
db.regRecords.countDocuments({createdAt: {$gte: ISODate("2026-07-10T00:00:00Z")}, "recordData.ai-email-address": "rechnung-AI@integrations-eu.pinbox24.com"})

Why this was invisible until the user noticed 450 missing emails

  • docker ps / container uptime looked fine — the process never crashed or restarted
  • PM2 process list looked fine — both cluster workers online, low CPU
  • GET /health and GET / (used by any naive uptime check) never touch MongoDB, so they kept returning 200 throughout the entire outage
  • Mailgun itself received every email successfully (its job ends at store()) — nothing on the Mailgun side ever indicated a problem
  • The only visible symptom was absence — no new AI-Logs records — which nobody was actively counting until the user manually noticed after sending a large batch

Prevention implication: a “records created per hour” metric/alert for the mailgun pipeline would have caught this within the first hour instead of after ~450 emails. See the follow-up issue for making container restart mandatory after any MongoDB credential rotation that touches w4_app/w3_app.

Detection Alert (issue #3688, added 2026-07-11)

This blind spot is now closed. mailgun-pipeline-exporter (monitoring/exporters/mailgun-pipeline-exporter/) runs on bms-1 alongside the containers it watches and independently counts, on a 30-minute rolling window:

  • mailgun_pipeline_inbound_accepted{address} — Mailgun’s own “accepted” events for the inbound domain (integrations-eu.pinbox24.com), queried directly against the Mailgun Events API. This is ground truth for “an email arrived” and does not depend on mailgun-v42-prod being healthy.
  • mailgun_pipeline_records_created{address} — regRecords documents created in w4_db for the same tracked address in the same window. Ground truth for “the pipeline actually wrote something”.

Tracked addresses: rechnung-AI@, ai-standard-docs@, gutschrift_AI@integrations-eu.pinbox24.com (the 3 configured AI-doc inbound addresses — see pinbox24-mailgun-duplicate-check-fix.md).

Alert: MailgunPipelineNoRecordsCreated (monitoring/prometheus/rules/pinbox24.yml) fires severity: critical when Mailgun accepted mail (sum(...accepted_total) > 0) but MongoDB created nothing (sum(...records_created_total) == 0) for the same window, sustained for: 30m — the exact signature of this incident. It is deliberately root-cause-agnostic: it fires for a stale credential (this incident), a crashed process, a MongoDB outage, or a code regression alike, because the detection doesn’t need to know why output stopped, only that it did.

How to interpret it firing:

  1. It is gated on the exporter’s own error counter (mailgun_pipeline_exporter_api_errors_total) being quiet — if MailgunPipelineExporterErrors (warning) is also firing, the exporter itself can’t reach Mailgun or MongoDB and the _total gauges are unreliable this cycle; fix the exporter first (bad/rotated MAILGUN_ADMIN_API_KEY or MAILGUN_PIPELINE_MONGODB_URI), don’t chase a phantom pipeline outage. This mirrors the Pinbox24NoLogs/MezmoExporterApiErrors gating pattern already in the same rule file, added after a ~20h misdiagnosis (#2492) of exactly this exporter-vs-pipeline confusion.
  2. If the exporter is healthy and the alert fires standalone, treat it exactly like this incident: start at the “How to confirm” section above (env-file vs running-container credential hash comparison), and if that’s clean, broaden to PM2 logs / MongoDB connectivity / application error logs.
  3. MailgunPipelineExporterDown (warning, up{job="mailgun_pipeline"} == 0) means the detector itself is offline — the pipeline could be silently failing right now with nothing watching.

Deployment status: LIVE on bms-1 since 2026-07-11 (deployed #3688; env-wiring fix #3817). The exporter runs in the mezmo-agent compose project at /opt/mezmo-agent/ and is scraped by Prometheus on vps-i1 at 94.23.26.113:9251. Its three env keys (MAILGUN_ADMIN_API_KEY, MAILGUN_PIPELINE_MONGODB_URI, MAILGUN_INBOUND_DOMAIN) live in the single shared /opt/mezmo-agent/.env — NOT a separate per-exporter file (see mailgun-pipeline-exporter-deploy.md). See bms-1/.env.example and bms-1/docker-compose.yml for requirements, and add-new-monitoring-exporter.md for the general pattern.

#3817 (2026-07-11) — MailgunPipelineExporterErrors fired for 50m; env never wired. The exporter was standing but its keys had been dropped into a stray staging file (/opt/p24-infra/bms-1/mailgun-pipeline-exporter.env) instead of merged into the compose’s /opt/mezmo-agent/.env, so the container started with only PORT set → both mailgun_pipeline_exporter_api_errors_total{source="mailgun"} and {source="mongodb"} incremented every 5-minute cycle (”… not set — skipping”). Fix: merged the three keys into /opt/mezmo-agent/.env and --force-recreated. Two corrections vs. the earlier assumption above: (1) MAILGUN_MONGODB_URL in secrets/pinbox24-backends.env.sops is empty, so the Mongo URI is instead the least-privilege w4_app connection string mailgun-v42-prod uses live (matching this playbook’s Permanent-Fix guidance — never the rs0 admin credential); (2) neither exporter key is wired into secrets-sync.yml yet, so the on-disk .env is manually maintained until the secret-manager follow-up lands (populate MAILGUN_MONGODB_URL in SOPS + add a sync-bms-1 step).

Permanent Fix (2026-07-10, same day)

The emergency fix above (recreate the container) restored service but left two structural problems in place: (1) mailgun-v42-prod was using the rs0 admin (root) credential just to write into one database — unnecessary privilege exposure for a webhook-processing container — and (2) it was rotated on a separate, easy-to-forget code path from v42-prod, which is exactly how it drifted onto a stale value in the first place.

What changed:

  1. mailgun-v42-prod now uses the w4_app credential (readWrite on w4_db only) — the exact same user and the exact same multi-host replicaSet URI shape as v42-prod’s own NEW_MONGODB_URI. Confirmed live: mongodb://w4_app:<password>@145.239.133.104,51.68.155.224/w4_db?replicaSet=rs0&authSource=w4_db, sourced from mongodb_w4_app_password in secrets/bms-servers.env.sops (the same key w4-mongodb-credential-rotation.md already used for v42-prod). It no longer touches MONGODB_RS0_ADMIN_PASSWORD at all.
  2. secrets-sync.yml’s “Sync mailgun-environment.env keys” step rebuilds MONGODB_URL from mongodb_w4_app_password instead of mongodb_rs0_admin_password on every CI run.
  3. secrets-sync.yml’s “Restart mailgun-prod” step now uses --force-recreate — it was missing this flag since the step was added (2026-07-08), which meant every rs0 admin password rotation since then silently left mailgun-v42-prod on a stale credential until someone manually force-recreated it (this incident). This is the exact anti-pattern w4-mongodb-credential-rotation.md already documented and fixed for v42-prod after the 2026-07-05 outage — it just hadn’t been carried over to this later-added mailgun-specific step.
  4. docs/playbooks/w4-mongodb-credential-rotation.md updated — mailgun-v42-prod is now listed as a second w4_app consumer, Step 3e updates its env file whenever w4_app is rotated, Step 4 recreates both containers in the same operation, and Step 5d verifies both via the same sha256 hash-comparison pattern used to diagnose this incident. Rotating w4_app from now on means rotating and recreating both containers together, every time — never one without the other.

Net effect: the credential this container holds is now least-privilege, and the two containers that share it can no longer drift onto different values — the exact drift that caused this incident is now structurally prevented, not just fixed once.

  • pinbox24-mailgun-duplicate-check-fix.md — the bind-mounted integration.helper.js / mailgunFileHandler.helper.js patches this container depends on; confirmed to survive docker-compose up -d
  • pinbox24-mailgun-s3v2-stabilization.md — broader pipeline stabilization history
  • mailgun-flow-fix-plan.md — upstream GitLab bug fixes needed (no timeout/error-handling on the DB call is the same class of bug flagged there for the S3 side)
  • scripts/mailgun-pipeline-analysis.ps1 — three-stage manual diagnostic script (Mailgun events / PM2 log / MongoDB count); PowerShell parse errors fixed (#3702) — this playbook’s original reference to it being broken is now stale
  • monitoring/exporters/mailgun-pipeline-exporter/ — continuous, automated version of the same three-stage check, wired to the MailgunPipelineNoRecordsCreated alert (issue #3688; see “Detection Alert” section above)
  • add-new-monitoring-exporter.md — general pattern for adding a new Prometheus exporter in this repo

Graph View

Backlinks