p24-infra Docs

mailgun-e2e-probe-operations

6 min read

Mailgun/W4 e2e liveness probe — operations (#3752)

Service: mailgun-pipeline-exporter (bms-1, port 9251) — extended in place, not a new service. Plan: docs/plans/plan-3752-mailgun-mongodb-e2e-liveness.md Sibling passive detector (#3688): docs/playbooks/mailgun-pipeline-exporter-deploy.md Status: code + wiring only, both legs OFF by default — not yet enabled live. This doc is the checklist to follow before flipping either flag.

1. What this adds

The passive mailgun-pipeline-exporter (#3688) can only detect a broken pipeline when organic traffic is flowing (accepted > 0 AND created == 0). During a quiet period it can never fire even if the write path is dead. #3752 closes that by manufacturing its own traffic on a schedule:

LegMechanismCadenceDefault
1 — direct_apiThis exporter calls POST /api/reg/{regId}/new → POST /api/offices/files/upload → PUT /api/reg/{regId}/{id} directly against api.w4.pinbox24.com, bypassing n8n/Drive entirely. Auth token minted via the p24-auth CF Worker (GET https://p24-auth.radieu.workers.dev/token, Authorization: Bearer P24_AUTOMATION_KEY) — never a raw stored Pinbox24 credential.15 minOFF (MAILGUN_E2E_LEG1_ENABLED=false)
2 — n8n execution healthNo new code. Cross-references the pre-existing N8nWorkflowPersistentlyFailing / N8nHighFailureRatio alerts (monitoring/prometheus/rules/n8n.yml) for workflow_id="MUjApqruo6H88ebw".continuous (existing scrape)always on (nothing to enable)
3 — n8n webhook canaryPOSTs a synthetic file to an isolated webhook trigger added to the production n8n workflow by scripts/n8n-3752-add-e2e-canary-trigger.ps1 (NOT applied automatically — see §3).dailyOFF (MAILGUN_E2E_LEG3_ENABLED=false, and the trigger itself doesn’t exist until the script runs)

Both legs write a real record into production w4_db.regRecords (~11.1M docs), tagged recordData._synthetic: true + recordData._probeRunId, and deliberately stop before POST /api/reg-process/{id}/318 (the “advance” call the real pipeline makes next) — per the #3752 human decision (2026-08-05): minimize footprint, do not advance the synthetic record through the full production pipeline.

2. Metrics + alerts

New series on the existing /metrics endpoint (no Pushgateway, no new scrape job):

  • p24_mailgun_e2e_success{leg="direct_api"|"n8n_webhook"} — 1 if the last probe’s record was verified in Mongo within the deadline, else 0.
  • p24_mailgun_e2e_latency_seconds{leg=...} — write→verified latency of the last successful run.
  • p24_mailgun_e2e_last_success_timestamp{leg=...} / p24_mailgun_e2e_last_run_timestamp{leg=...}.
  • p24_mailgun_e2e_synthetic_backlog — count of un-purged synthetic docs past the TTL (purge visibility, not a safety mechanism — see §4).
  • mailgun_pipeline_exporter_api_errors_total{source="p24auth"|"w4api"|"n8n_webhook"|"mongo_purge"} — new source labels on the existing counter.

New alerts (monitoring/prometheus/rules/pinbox24.yml):

  • MailgunPipelineE2EFailed — Leg 1 ran and its record did not land within the deadline, sustained ~16 min (>1 retry cycle). Pipeline broken. Scoped to leg="direct_api" only — Leg 3’s daily cadence doesn’t fit a short for: window; its breakage shows up via MailgunPipelineE2EStale instead.
  • MailgunPipelineE2EStale — the probe itself stopped attempting cycles (exporter down, flag flipped off, host issue). Probe broken, not pipeline broken — same p4_claude_auth_valid/#4429 lesson this repo has already learned once: a dead probe must never read as a healthy (or a failing) pipeline.
  • Leg 2 gets no new alert — see the cross-reference comment in n8n.yml next to N8nWorkflowPersistentlyFailing.

3. Rollout checklist (do this in order — do not skip steps)

3.1 Leg 1 (direct-API)

  1. Confirm secrets-sync.yml’s sync-bms-1 job has delivered P24_AUTOMATION_KEY to /opt/mezmo-agent/.env on bms-1 (it’s wired in the same PR as the code — verify with docker inspect mailgun-pipeline-exporter --format '{{.Config.Env}}' looking for the key name only, never its value).
  2. Manually verify the CF Worker path works before trusting the schedule: from any machine with the automation key, curl -s -H "Authorization: Bearer $P24_AUTOMATION_KEY" https://p24-auth.radieu.workers.dev/token should return {"token": "...", "expiresAt": "..."}. Never print the key or the returned token in a shared terminal/log.
  3. Set MAILGUN_E2E_LEG1_ENABLED=true in /opt/mezmo-agent/.env on bms-1, then docker-compose up -d --force-recreate mailgun-pipeline-exporter (env_file is only re-read on recreate — see docs/playbooks/mailgun-pipeline-exporter-deploy.md).
  4. Watch docker logs -f mailgun-pipeline-exporter for the first leg1: log line (up to 15 min). Confirm leg1: verified in ...s, not a warning.
  5. Confirm in Mongo (read-only): a w4_db.regRecords doc with recordData._synthetic: true and a recent createdAt exists, and does NOT show any reg-process/318 advance side effect.
  6. Confirm the new Prometheus series are scraping: p24_mailgun_e2e_success{leg="direct_api"} on vps-i1’s Prometheus should read 1.

3.2 Leg 3 (n8n canary) — read scripts/n8n-3752-add-e2e-canary-trigger.ps1’s header in full first

  1. pwsh scripts\n8n-3752-add-e2e-canary-trigger.ps1 -WhatIf — review the exact node/connection diff. No live change happens.
  2. Run it for real (no -WhatIf) to add the 14 e2e_canary_*_rechnungen nodes. This does not change the workflow’s active state or touch any existing node — the new trigger is a webhook, so nothing runs until it is explicitly called.
  3. Manually trigger it once via the n8n UI (“Execute workflow” on the new webhook node) or curl -X POST https://n8n.srv1072950.hstgr.cloud/webhook/e2e-canary-w4-rechnungen -H "Content-Type: application/json" -d '{"filename":"_e2e_canary_manual-test.pdf","fileBase64":"<base64 of a tiny PDF>","probeRunId":"manual-test"}'. Watch the execution in the n8n UI end-to-end:
    • Drive file appears in the production folder with the reserved name, then disappears (cleanup node ran).
    • A regRecords doc appears tagged _synthetic: true / _probeRunId: "manual-test".
    • No call was made to .../reg-process/.../318 — check the execution’s node list.
    • The Redis lock (w4ingest:lock:rechnungen) is released promptly (not held for the full 600s TTL) — lock_del node ran.
  4. Only after that manual run succeeds cleanly, set MAILGUN_E2E_LEG3_WEBHOOK_URL and MAILGUN_E2E_LEG3_ENABLED=true in bms-1’s .env, then recreate the exporter as in §3.1 step 3.

Known verification gaps a human should close during step 2/3, not assume: the Google Drive node’s queryString search-filter parameter and the moveBinaryData (base64→binary) node’s parameter shape were written from general n8n schema knowledge, not confirmed against this specific n8n version’s UI. Both fail safe if wrong (the node errors visibly in the execution log; it does not silently fall back to the unfiltered search this design exists to avoid) — but confirm they actually work as intended during the manual dry-run before relying on the schedule.

4. Synthetic-record isolation + purge

  • Marker: recordData._synthetic: true + recordData._probeRunId (uuid). Reserved ai-email-address value e2e-canary@probe.internal (not a real tracked address).
  • Inertness (the actual safety mechanism): the probe never calls the reg-process/.../318 advance step, per the #3752 human decision. If the Pinbox24 owner later confirms a _synthetic record is fully inert even if advanced, that call could be added — do not add it without that confirmation.
  • Purge: every hour (MAILGUN_E2E_PURGE_INTERVAL_S, independent of whether either leg is enabled), the exporter deletes _synthetic: true docs older than MAILGUN_E2E_PURGE_TTL_S (default 24h). A purge failure (e.g. the Mongo credential turns out to be read-only) increments mailgun_pipeline_exporter_api_errors_total{source="mongo_purge"} and is visible via p24_mailgun_e2e_synthetic_backlog — it is tidiness, not safety; an un-purged record is still inert per the point above.
  • If the Pinbox24 owner has NOT yet confirmed synthetic-record safety (plan §8.2, still open at implementation time): this is exactly why both legs ship disabled. Get that confirmation before flipping MAILGUN_E2E_LEG{1,3}_ENABLED.

5. Silencing / rollback

  • Disable a leg without touching code: set MAILGUN_E2E_LEG1_ENABLED=false / MAILGUN_E2E_LEG3_ENABLED=false in bms-1’s .env, force-recreate the exporter. Purge keeps running regardless, cleaning up anything already written.
  • Remove the n8n canary trigger entirely: pwsh scripts\n8n-3752-add-e2e-canary-trigger.ps1 -Remove — deletes the 14 e2e_canary_*_rechnungen nodes, leaves everything else untouched.
  • Silence the alerts (e.g. during a known Pinbox24 maintenance window): standard Alertmanager silence on alertname=~"MailgunPipelineE2E.*", same as any other Pinbox24 alert.

Graph View

Backlinks