p24-infra Docs

plan-3752-mailgun-mongodb-e2e-liveness

20 min read

Plan #3752 — Phase 3: synthetic W4 ingestion e2e liveness monitoring

Status: Design v2 (Code-change-design ready for /review-plan) — REVISION, see §0 Owner-issue: #3752 Branch: plan/3752-mailgun-mongodb-e2e-liveness Parent (Phase 1+2, closed): #2547 — shipped PR #3748 Sibling service (passive detector, shipped, now partly inert): #3688 — monitoring/exporters/mailgun-pipeline-exporter/ Originating incident: #2546 (2026-07-02, 4-year-expired cert on mailgun-api.w4.pinbox24.com) — the general failure class (“endpoint up, TLS valid, records never reach MongoDB”) this plan closes, not the specific dead leg any more (see §0). v1 plan (superseded by this doc): shipped in PR #4808, reviewed/APPROVED in the 2026-08-01 /review-plan comment on this issue — approval stands for everything except §4–§6 of that version, which assumed a now-dead ingestion path (see §0). Triggering investigation: #5440 (closed) — established the 2026-07-11 pipeline cutover; PR #5452 disabled the resulting false-positive alert and deferred the permanent fix to this issue.

Plan only — do NOT implement. The deliverable is this document. Implementation is gated on /review-plan and the human decisions in §8. Authored by a dev-coder worker: writing the design is in scope; the credential and server/DB/n8n actions it enumerates (§6) are delegated — see §9.

0. Why this revision exists (read this first)

The v1 plan (PR #4808, approved 2026-08-01) designed a synthetic probe around Option A — a real Mailgun send to test-monitor@integrations-eu.pinbox24.com as the primary daily leg, on the premise that Mailgun→endpoint routing was the untested leg that broke in #2546.

The #5440 investigation (infra-task worker bms4-cw-1, read-only Mailgun Events API + DNS + GitLab + n8n evidence) then established that AI-doc ingestion for this exact register (AI-Logs, regId 67597cb44eefb1002c9847d4) was cut over on 2026-07-11 from email → Mailgun inbound to Google Drive → n8n workflow MUjApqruo6H88ebw (“W4 Direct API Ingestion”) → direct POST /api/reg/67597cb44eefb1002c9847d4/new. Mailgun inbound for integrations-eu.pinbox24.com is decommissioned as a side-effect (accepted=0 every day since, confirmed via stats/total). Sending to test-monitor@integrations-eu.pinbox24.com today would land on a dead route and prove nothing — v1’s primary leg would silently monitor an empty mailbox forever.

This revision replaces the probe mechanism (v1 §4–§6) with one that exercises the actual live path. Confirmed by the issue owner (2026-08-05):

  1. Probe path: n8n → direct-API, not the retired email leg — exercise whatever combination of triggering the n8n workflow / calling the direct API accurately represents real production traffic (this issue’s own open decision — resolved below in §5).
  2. Synthetic records in production regRecords remain acceptable, tagged + purged on a schedule, with purge-failure behaviour specified (inert, not merely tidy) — this requirement is unchanged from v1 and is carried forward verbatim in §7.

Everything not touched by the path change is carried over from v1 with only the necessary edits: synthetic-record tagging/purge (§7, unchanged), build-location decision to extend the bms-1 mailgun-pipeline-exporter rather than build a new vps-i1 service (§4, unchanged — reused credentials there are unaffected by which upstream trigger produces the record), metric-exposure approach (native gauges, no Pushgateway — §5d/§5e unchanged), compliance steps (§10, unchanged).

What changed, concretely:

v1 (superseded)v2 (this doc)
Probe triggerReal Mailgun sendn8n workflow execution health (existing, free) + direct POST /api/reg/.../new (new, cheap)
Whole-chain legDaily real send to test-monitor@…Deferred, optional — full n8n webhook + synthetic Drive file (§5, Leg 3)
New credential neededMailgun send-scoped key (likely already had it)None required for the two mandatory legs (§6)
Verified target endpointmailgun-api.w4.pinbox24.com (now dead traffic)api.w4.pinbox24.com (confirmed live, already TLS/endpoint-monitored — §2)

1. Goal (one sentence)

Add liveness coverage that, on a schedule, verifies the live AI-Logs ingestion path — n8n (MUjApqruo6H88ebw) driving Google Drive → api.w4.pinbox24.com → MongoDB regRecords — is actually landing records, closing the failure class Phase 1+2 and the passive #3688 detector structurally cannot catch: the trigger/endpoint looks fine, yet the write path is dead, 24/7, independent of organic file volume.

2. What each existing layer covers, and the exact gap this fills (updated)

Sources read for this revision (2026-08-05, this repo + live read-only queries): monitoring/prometheus/prometheus.yml, monitoring/prometheus/rules/{synthetic,pinbox24,n8n}.yml, monitoring/exporters/mailgun-pipeline-exporter/app.py, n8n REST API (GET /workflows/MUjApqruo6H88ebw, GET /executions?workflowId=...) via the existing N8N_API_KEY/BMS4_N8N_HOST (secrets/n8n-bms4.env.sops) — read-only, no values displayed.

LayerShipped inCatchesBlind to
Phase 1+2 — blackbox/blackbox_ssl probes of api.w4.pinbox24.com (confirmed present in both jobs — prometheus.yml:261,292 — this is the live-path target, no gap here)PR #3748cert-expiry, EndpointDown, /api/i18n/langs functional checkendpoint 200-OK but no record written
Generic n8n monitoring — n8n-bms4-exporter + monitoring/prometheus/rules/n8n.yml (N8nWorkflowPersistentlyFailing, N8nHighFailureRatio) — already covers MUjApqruo6H88ebw by workflow_id, no new work neededpre-existingsustained status="error" executions (>3/1h for 30m), workflow-failure ratio >30%deliberately excludes status="crashed" (in-flight runs reaped by an n8n process restart — see rule comment, n8n.yml:122-125, added to avoid false-tripping on a single restart); live-verified today (2026-08-05) the last 20 executions of this exact workflow are ALL status="crashed", not error — i.e. right now this generic alert structurally cannot see the workflow’s current instability (see §2a)
#3688 passive detector — mailgun-pipeline-exporter on bms-1, alert MailgunInboundPipelineStalled/MailgunPipelineNoRecordsCreated#3688(nothing, on this path — gated on mailgun_pipeline_inbound_accepted > 0, permanently 0 since the cutover; correctly non-paging per #5452, but provides zero coverage of the live path)everything on the live path
#3752 (this plan) — direct-API synthetic write+verify (Leg 1, mandatory)—write-path dead regardless of organic traffic or n8n’s own health, 24/7n8n/Drive-side regressions upstream of the API call (deliberately — see §5)
#3752 (this plan) — n8n execution-health cross-reference (Leg 2, mandatory, reuses existing metric)—sustained n8n-side error failures (already alerted); adds visibility into the crashed-only gap found in §2aa healthy-looking crashed-free window with 0 real executions (quiet folder) — same blind-to-quiet-periods issue #3688 had; Leg 1 covers this independently

Rule of thumb, updated: Leg 1 asks “if I write a record right now via the same API n8n calls, does it land?” — manufactured traffic, works 24/7, independent of n8n’s health. Leg 2 asks “is the thing that’s supposed to be calling that API actually running and succeeding?” — real traffic, cross-references monitoring that already exists. Together they close the full “trigger is fine but write is dead” and “trigger itself is dead” gap without needing to safely reproduce Google Drive polling from a probe (see §5’s Leg 3 discussion for why that’s deferred, not omitted).

2a. New finding this revision surfaces — not actioned here

Live query of GET /executions?workflowId=MUjApqruo6H88ebw&limit=20 (2026-08-05) shows the most recent 20 executions all status="crashed", clustered around two stoppedAt timestamps (08:06:16Z, 08:47:31Z) — consistent with the n8n main process on bms-4 restarting and bulk-reaping in-flight executions, not per-execution logic failures. Per n8n.yml:122-125 this is deliberately excluded from N8nWorkflowPersistentlyFailing to avoid a single-restart false trip — but a repeated crash-loop pattern across multiple restarts in one morning is itself a health signal with no current alert. Out of scope for this plan (it’s an n8n/bms-4 process-stability question, not an ingestion-pipeline-liveness question) — flagged here so it isn’t lost; worth a follow-up issue titled along the lines of “n8n bms-4 process restarting repeatedly — status=crashed execution bursts uncorrelated with deploys” if it recurs. Leg 1 (§5) is unaffected by this either way, which is exactly why it is the mandatory leg, not Leg 2.

3. Current coverage baseline (confirmed 2026-08-05, this repo + live n8n read)

  • monitoring/exporters/mailgun-pipeline-exporter/app.py (#3688) still runs on bms-1, holds MAILGUN_ADMIN_API_KEY and MAILGUN_PIPELINE_MONGODB_URI (read access to w4_db.regRecords). This baseline is unchanged from v1 — the build-location decision (§4) still holds regardless of which upstream leg produces the record being verified.
  • New for this revision — the n8n workflow MUjApqruo6H88ebw (“W4 Direct API Ingestion (Rechnungen/Gutshrifts/Standard)”) was inspected read-only via the n8n REST API (secrets/n8n-bms4.env.sops → N8N_API_KEY/BMS4_N8N_HOST, already-provisioned, no new credential): 78 nodes, active, 3 category branches (rechnungen/gutshrifts/standard), each: scheduleTrigger (hourly, :20/:30/:40) → Redis singleton lock (#4008) → googleDrive search (resource: fileFolder, fixed production folderId) → splitInBatches loop → wait → googleDrive download → httpRequest chain: POST /api/auth (custom credential, n8n-internal) → POST /api/reg/67597cb44eefb1002c9847d4/new → POST /api/offices/files/upload → PUT /api/reg/.../{id} (link) → POST /api/reg-process/{id}/318 (advance) → size-based Drive archive/move.
  • Notable discovery: each of the 3 categories already has a parallel test_webhook_* + test_search_* node pair (e.g. test_webhook_rechnungen, webhook path test-w4-ingest-rechnungen) that shares the same Redis lock gate and the same production Drive folderId as the scheduled path, feeding into the same loop_rechnungen → download → create → upload → link → advance chain. This is pre-existing on-demand test tooling (most likely added alongside the #4008 singleton-lock work), not something this plan is proposing to build — but it is exactly the entry point a full-chain synthetic leg would need, and its current wiring (unfiltered limit:1 search against the shared production folder) is not safe to reuse as-is for an unattended recurring canary — see §5, Leg 3.
  • V42_GOOGLE_CREDENTIALS_JSON_B64 already exists in secrets/pinbox24-w4.env.sops — plausibly the same credential the workflow’s googleDrive nodes use (unconfirmed scope/folder-restriction). Relevant only to the optional Leg 3 (§5); not required for the two mandatory legs.
  • Candidate existing W4-API credentials that may already authorize a direct write (avoiding a new credential request for Leg 1): V42_monitoringToken, V42_INVOICE_AUTH_TOKEN (secrets/pinbox24-w4.env.sops) — names only, not evaluated here; confirm-before-request is the action item (§6.1), same pattern v1 used for the Mailgun key.

4. Design decision — where the probe runs (unchanged from v1)

Still recommend extending the bms-1 mailgun-pipeline-exporter (#3688) rather than a new vps-i1 service. Nothing about the path change affects this: the exporter already holds a working w4_db.regRecords read path on bms-1 (§3), the write-verification loop for Leg 1 is identical in shape to what v1 designed (send/POST → poll Mongo for the tagged marker), and co-locating the two detector halves (passive #3688 + active #3752) in one service, one ops doc, one alert group is still the right trade-off. See v1 §4 (PR #4808 history) for the full comparison table — it is not reproduced here because none of its inputs changed.

The exporter’s name (mailgun-pipeline-exporter) is now slightly stale given the live path has no Mailgun leg — not renaming it in this plan (churn on a working, alerted-on service name for a cosmetic reason is not worth it; note it in the ops doc instead, per §10).

5. Design decision — probe mechanism (REVISED — this is the core of this revision)

5a. The three candidate legs

LegWhat it exercisesCadenceNew credential?Verdict
1. Direct-API synthetic write+verify — POST /api/auth → POST /api/reg/67597cb44eefb1002c9847d4/new (+ upload/link/advance to mirror a real record shape) with a synthetic, marked payload, then poll regRecords for the tagged marker within a deadlineapi.w4.pinbox24.com + MongoDB write path — the leg that must never be silently dead (the 2546-class guarantee)15 minLikely none — reuse/confirm an existing W4 API credential (§6.1)Mandatory
2. n8n execution-health cross-reference — read n8n_bms4_executions_1h{workflow_id="MUjApqruo6H88ebw"} (already scraped, already alerted via N8nWorkflowPersistentlyFailing/N8nHighFailureRatio)Is the thing that’s supposed to call Leg 1’s endpoint actually running and succeeding on real trafficContinuous (existing scrape cadence)None — reuses existing n8n-bms4-exporter metricsMandatory (cross-reference only — no new component)
3. Full-chain via n8n webhook — drop a marked synthetic PDF into a dedicated, non-production Drive subfolder, POST the corresponding test_webhook_*-style trigger, verify via the same Mongo poll as Leg 1The Drive-search/webhook wiring itself (folder-ID drift, OAuth-token expiry masquerading as “0 results = success”) — the one failure class Legs 1+2 cannot seeDaily (if approved)Yes — Drive write scoped to one dedicated subfolder (§6.4); candidate reuse of V42_GOOGLE_CREDENTIALS_JSON_B64, scope unconfirmedDeferred / optional — human decision, §8

5b. Why this replaces v1’s “Option A daily send + Option B 15-min POST” 1:1 justification

v1 justified real-send-Option-A over direct-POST-Option-B because “the untested leg is precisely the one that broke” (Mailgun→endpoint routing, #2546). Applying the same rule to the live path: the leg most recently shown to be unstable is the n8n hop itself (§2a — live-verified status=crashed burst on 2026-08-05), which argues for testing it, not skipping it. But unlike Mailgun (which required a genuinely new send credential), n8n’s own execution telemetry for this exact workflow already exists and is already alerted on (n8n.yml, pre-existing, N8nWorkflowPersistentlyFailing) — building a second, bespoke way to ask “is n8n succeeding” would duplicate monitoring that is already live, for a cost (new Drive credential, workflow changes, cadence tuning to avoid colliding with the production Redis lock) that the original A/B trade-off didn’t have to pay. Leg 2 gets that coverage for free by cross-referencing, not rebuilding.

What Leg 2 alone cannot prove — and what actually motivated Option A originally — is “if a document is dropped right now, does a record land.” That is precisely Leg 1: it doesn’t wait for n8n, doesn’t touch Drive, and calls the exact downstream API n8n itself calls (api.w4.pinbox24.com/api/reg/{regId}/new), so it verifies the part of the chain most analogous to “the record actually gets written” — the same guarantee v1’s Option A was chasing, minus the dependency on a trigger mechanism (Mailgun, and now n8n/Drive) that has changed once already and could change again. This is deliberately a stronger invariant than “test whatever today’s trigger happens to be.”

Leg 3 is where the full chain (Drive polling included) gets tested, and it is real — the test_webhook_* discovery (§3) shows the production workflow already has an on-demand entry point for exactly this. It is marked deferred/optional, not omitted, for two reasons documented so the cheaper-looking shortcut isn’t silently taken instead of a real decision:

  1. Safety of the existing test_webhook_* wiring for automation. test_search_rechnungen (and the gutshrifts/standard equivalents) search the same production folderId with limit: 1 and no filename filter — calling it on an unattended schedule could process a real customer file out of order instead of (or as well as) the probe’s synthetic one, which is not a “probe fails safe” outcome. Reusing this for an automated canary needs either (a) a small n8n change to filter the search by the probe’s reserved filename pattern, or (b) pointing a dedicated probe-only search node at a dedicated, non-production subfolder — both are workflow edits, not just exporter code, and are out of scope for a “write the plan” pass.
  2. Credential cost. Placing a file into Drive requires write access somewhere. §3 found a plausibly-reusable credential (V42_GOOGLE_CREDENTIALS_JSON_B64), but its actual scope (read-only vs. write, which folder(s)) is unconfirmed — verifying or requesting this is a secret-manager/human-adjacent action this plan should not assume the answer to.

Recommendation: implement Legs 1+2 now (both are free of new credentials and low-risk); bring Leg 3 back as a follow-up decision once Legs 1+2 have run for a baseline period — record this as an explicit §8 human decision rather than a silent scope cut.

5c. Verification (Leg 1)

Poll w4_db.regRecords for the synthetic record with a ~5-minute deadline, using the exporter’s existing _fetch_mongo_record_counts query shape, keyed on the synthetic marker (§7), not the production TRACKED_ADDRESSES (those are Mailgun-era email addresses and are not meaningful for a direct-API-created record — the synthetic record instead carries recordData._synthetic: true + recordData._probe_run_id, matched directly). Success = record present within deadline; latency emitted as a gauge, mirroring v1 exactly.

5d. Metric exposure — native gauge, no Pushgateway (unchanged from v1)

Same rationale as v1: extend the exporter’s existing /metrics, no new infra.

  • p24_mailgun_e2e_success{leg="direct_api"} — 1 if the last Leg-1 write+verify landed within SLA, else 0. (leg="n8n_webhook" reserved for if/when Leg 3 ships.)
  • p24_mailgun_e2e_latency_seconds{leg="direct_api"} — send→record latency.
  • p24_mailgun_e2e_last_success_timestamp{leg="direct_api"} / p24_mailgun_e2e_last_run_timestamp{leg="direct_api"} — drive the staleness alert exactly as v1 designed.
  • Leg 2 needs no new metric — it is a PromQL cross-reference (§5e) against the existing n8n_bms4_executions_1h{workflow_id="MUjApqruo6H88ebw"} series, not a new exporter output.
  • Reuse the exporter’s existing mailgun_pipeline_exporter_api_errors_total{source=...} counter, adding source="w4api" for Leg 1’s own call failures.

5e. Alert rules (updated targets, same structure as v1)

Distinguish “pipeline broken” from “probe broken” — a dead probe must not read as a dead pipeline (the p4_claude_auth_valid/#4429 failure mode; do not repeat it. Same principle n8n.yml:122-125 already applies to crashed vs error, reused here deliberately):

  • MailgunPipelineE2EFailed — p24_mailgun_e2e_success{leg="direct_api"} == 0 for 2 consecutive evaluations. Meaning: Leg 1 ran, the record did not land → write path broken. Route to the existing Pinbox24 alert path.
  • MailgunPipelineE2EStale — time() - p24_mailgun_e2e_last_run_timestamp{leg="direct_api"} > 90m, or the series is absent. Meaning: the probe itself is not running — distinct, lower urgency, never conflated with pipeline failure.
  • No new alert for Leg 2 — N8nWorkflowPersistentlyFailing / N8nHighFailureRatio already cover it generically. Add a one-line comment cross-reference in pinbox24.yml next to the new rules pointing at n8n.yml’s alerts by name, so the three layers (Phase 1+2, n8n-generic, this plan) read as one system from either file.
  • Keep the Phase 1+2 rules in synthetic.yml and #3688’s MailgunPipelineNoRecordsCreated / MailgunInboundPipelineStalled untouched (still inert-but-harmless on the dead email leg, per #5452 — retiring them is a separate cleanup decision, not blocking this plan).

6. Access / credentials to request — with owners (raise before implementation)

All items below are out of dev-coder scope and are delegated per §9. This plan only enumerates them.

6.1 Direct-API write credential (Leg 1) — secret-manager, likely no new secret

  • Action: confirm whether an existing W4-API credential already authorizes POST /api/reg/67597cb44eefb1002c9847d4/new — candidates found in secrets/pinbox24-w4.env.sops: V42_monitoringToken, V42_INVOICE_AUTH_TOKEN (names only; scope/validity unconfirmed by this plan). Test with a scoped, clearly-synthetic call before requesting anything new.
  • If neither works: request a narrowly-scoped write credential (ideally restricted to the AI-Logs regId only, if the API supports register-scoped tokens) into secrets/pinbox24-w4.env.sops, Tier 1 (no human) unless the API only issues broad tokens, in which case escalate per secret-rotation-access-matrix.md.

6.2 MongoDB regRecords read+delete — unchanged from v1, already satisfied

  • Reuses MAILGUN_PIPELINE_MONGODB_URI (or its MAILGUN_MONGODB_URL successor per #3850). No new user, no firewall change. Delete scope (for purge, §7) still needs to be limited to synthetic-marked docs only, per v1’s own recommendation — carried forward unchanged.

6.3 n8n API read access (Leg 2) — already satisfied, zero new access

  • N8N_API_KEY + BMS4_N8N_HOST already exist in secrets/n8n-bms4.env.sops and were used read-only to produce this plan’s evidence. The bms-1 exporter needs the same key delivered to it (or, simpler, Leg 2 can be a pure Prometheus cross-reference/recording rule with no exporter code change at all — see §5e; recommend the latter unless a future need arises to expose a combined single metric).

6.4 Google Drive write credential (Leg 3, deferred) — secret-manager, do not action until §8 approves Leg 3

  • If/when Leg 3 is approved: confirm whether V42_GOOGLE_CREDENTIALS_JSON_B64 (secrets/pinbox24-w4.env.sops) is reusable and scoped to write into a dedicated probe subfolder; if not, request a new Drive credential scoped to exactly one folder. Given the Google-OAuth provisioning pattern already seen elsewhere in this ecosystem (gmail-tools), this may require a human step (GCP console / Drive folder-sharing UI) rather than a pure Tier 1 SOPS add — flag this explicitly to whoever picks up Leg 3, do not assume Tier 1.

7. Synthetic-record isolation + purge (mandatory — unchanged in substance from v1)

Carried forward from v1 verbatim in substance; the mechanism does not depend on which upstream leg produced the synthetic call.

  • Isolation: stamp an unmistakable marker on the payload — recordData._synthetic: true and a recordData._probe_run_id (uuid) — on every Leg-1 write. (Adjust marker naming only if the direct API rejects unknown fields under recordData; confirm the schema tolerates extra fields before implementation, per the API-contract note in docs/pinbox24/ai-processing-register-workflow.md.) The verification query keys on the marker, so it can never match an organic record. No email address is involved any more (Leg 1 bypasses email entirely) — drop v1’s test-monitor@… reserved address concept; it is moot for the direct-API leg and only relevant if/when Leg 3 ships (in which case its equivalent is a reserved filename pattern in the probe’s dedicated Drive subfolder, not a reserved address).
  • Purge: the exporter deletes synthetic records older than a short TTL (e.g. 24h) each cycle, matched by the marker. This is the only place the probe needs Mongo write (delete) access — call it out explicitly in the access request (§6.2 is read + delete on synthetic-marked docs only, ideally enforced by the least-privilege user’s scope).
  • Purge-failure behaviour (must be inert, not merely tidy): the marker must make a synthetic record harmless even if the purge never runs — confirm with the Pinbox24 owner that a _synthetic:true record is excluded from all reporting/billing/reg-process flows (the aiProcessingLog process advances the record to activity 318 as part of the write — confirm this does not trigger any downstream side effect, e.g. billing or a customer-visible notification, for a synthetic record). If it cannot be guaranteed inert, do not advance the process for synthetic records (stop after create+upload+link, skip the advance call) — verifying the record landed does not require advancing its workflow state. Emit p24_mailgun_e2e_synthetic_backlog (count of un-purged synthetic docs) and alert if it grows.

8. Human decisions required before implementation (this issue is in Review)

  1. Probe mechanism — approve Leg 1 (direct-API write+verify) + Leg 2 (n8n execution-health cross-reference) as the mandatory pair (recommended, §5), with Leg 3 (full n8n-webhook chain) deferred pending a separate go/no-go once Legs 1+2 have a baseline (recommended), or approve building Leg 3 now alongside 1+2.
  2. Synthetic-record safety — Pinbox24 owner confirms a _synthetic:true record in regRecords is inert w.r.t. reporting/billing, and whether advancing it to reg-process activity 318 is safe or should be skipped (§7). Carried over from v1, still open.
  3. Build location — reconfirm extending the bms-1 mailgun-pipeline-exporter (§4, unchanged recommendation) — no new decision needed here unless Leg 3 changes the calculus (a Drive-write credential living on bms-1 vs. elsewhere may warrant reconsideration if Leg 3 is approved).
  4. New — direct-API credential — approve testing V42_monitoringToken / V42_INVOICE_AUTH_TOKEN reuse before requesting a new W4-API write credential (§6.1).

9. Role boundary (dev-coder authoring note, unchanged from v1)

Per worker-issue-scoped.md Step R2 and dev-coder.md capabilities, this worker:

  • May (and did) write this plan doc, including the read-only n8n REST API inspection used to ground §3/§5 (workflow structure and recent execution status only — no credential values, no workflow mutation).
  • Must delegate the credential actions in §6 to secret-manager, and any bms-1/bms-4 deploy or n8n-workflow-editing action (including Leg 3’s search-filter change, if approved) to sys-admin. Referencing key names to confirm existence, and reading workflow JSON structure via the n8n API, are allowed and are not delegation triggers.

10. Compliance (per CLAUDE.md §Compliance — at implementation time, not now)

  • Register the extended probe capability against the existing mailgun-pipeline-exporter row in Supabase dev_r_services (unchanged from v1).
  • Ops doc: extend docs/playbooks/mailgun-pipeline-exporter-deploy.md (or add docs/mailgun-e2e-probe-operations.md) covering: what Leg 1 writes, how to silence it, synthetic record tagging/purge, the purge-failure runbook, and a note that the exporter’s name is now a historical artifact of the retired Mailgun leg (not renamed, per §4).
  • Also update docs/pinbox24/ai-processing-register-workflow.md (currently describes only the email pipeline, last updated 2026-07-02, now stale per the #5440 finding) to document the current n8n → direct-API architecture — flagged in #5440’s own recommendation #4, folding it into this plan’s implementation checklist rather than leaving it as a dangling suggestion.
  • Not an EU AI Act system (no scoring/ranking/employment decisions) — no dev_r_ai_systems entry.

11. Acceptance (mirrors the #3752 triage checklist)

  • Plan doc written under docs/plans/ (this revision)
  • Probe mechanism chosen with reasoning recorded (§5 — Leg 1 + Leg 2 mandatory, Leg 3 deferred)
  • Synthetic-record tagging + purge strategy specified, incl. purge-failure behaviour (§7)
  • Metric + alert rule drafted, incl. probe-liveness (staleness) distinction (§5d–5e)
  • All credential/access requests enumerated with target SOPS files (§6)
  • Compliance registration steps included (§10)
  • Pinbox24 owner confirms synthetic records are safe + advance-or-not decision (§8.2 — human)
  • Leg 3 go/no-go recorded (§8.1 — human)
  • /review-plan passed
  • Parent (Phases 1+2, closed): #2547 · shipped PR #3748 · originating incident #2546
  • Sibling passive detector (now inert on this path): monitoring/exporters/mailgun-pipeline-exporter/ (#3688)
  • Path-change investigation: #5440 (closed) · false-positive-alert fix PR #5452
  • v1 plan (superseded): PR #4808, /review-plan approval 2026-08-01
  • n8n workflow: MUjApqruo6H88ebw (“W4 Direct API Ingestion (Rechnungen/Gutshrifts/Standard)”), bms-4
  • Redis singleton-lock precedent (shared by the test_webhook_* entry points Leg 3 would reuse): #4008, docs/playbooks/n8n-redis-singleton-lock.md
  • Existing generic n8n alerting: monitoring/prometheus/rules/n8n.yml (N8nWorkflowPersistentlyFailing, N8nHighFailureRatio)
  • Mailgun flow fix plan (historical, email-era): docs/pinbox24/mailgun-flow-fix-plan.md
  • AI-Logs pipeline architecture doc (stale, needs update per §10): docs/pinbox24/ai-processing-register-workflow.md

Design generated by Claude Code (dev-coder worker). Review before starting implementation.

Graph View

Backlinks