p24-infra Docs

mailgun-flow-fix-plan

6 min read

Pinbox24 — Mailgun Inbound Email Flow Fix Plan

Created: 2026-06-29 Source issue: #2048 Status: Plan — code fixes execute in external GitLab repos (see “Scope & ownership”) Affected microservices: p24-ms-mailgun (→ mailgun-v42-prod), pinbox24-ms-s3-v2 (→ s3-v2-v42-prod) Production host: bms-1 (OVH ns367522, 94.23.26.113) Image registry: AWS ECR 563740926945.dkr.ecr.eu-central-1.amazonaws.com (region eu-central-1)

TL;DR

The inbound email pipeline (Mailgun → p24-ms-mailgun / pinbox24-ms-s3-v2 → MongoDB + Wasabi S3) is silently broken. Both microservices return a success HTTP status to Mailgun before attachments finish processing, so Mailgun receives a 2xx and never retries — files are lost with no error surfaced. There are also several await-less async bugs and undefined-variable bugs in the mailgun service, plus production credentials hardcoded in docker-deploy-prod.sh in both repos (a deploy blocker that must be rotated first).

This document is the engineering plan. The line-level code fixes land in the two external GitLab repos; this infra repo owns the plan, the deploy playbook (docs/playbooks/pinbox24-ms-deploy.md), and the credential rotation tracking.

Scope & ownership

Work itemRepo / systemOwner
pinbox24-ms-s3-v2 async-bug fixesGitLab pinbox24-ms-s3-v2App dev (external repo)
p24-ms-mailgun async + undefined-var fixesGitLab p24-ms-mailgunApp dev (external repo)
Credential rotation (hardcoded prod secrets)both repos + SOPS + GitLab CI/CD varsHuman — security blocker
Staging verificationbms-3 staging stacksApp dev + infra
Production deploy (s3-v2 first, then mailgun)bms-1 via ECR / GitLab CIHuman — see deploy playbook
This plan + deploy playbookp24-infra (this repo)infra agent

Why the code fixes are not in this PR: the source for both microservices lives in separate GitLab repositories, not in p24-infra. A p24-infra worker has no checkout of those repos. The infra-repo deliverable is this plan and the deploy/rollback playbook; the code changes below are the exact spec for whoever edits the GitLab repos.

Root causes

A. pinbox24-ms-s3-v2 — CRITICAL (silent data loss)

  1. Promise.all() not awaited — scanque.controller.ts:43. The handler responds 201 before the upload promises resolve. The HTTP 201 races ahead of the actual S3/Mongo writes.
  2. forEach + async callback — storage.controller.ts:115. Array.prototype.forEach does not await async callbacks, so the response fires after the first file and the rest are a fire-and-forget race.
  3. Outer catch returns 201 on any error — failures are swallowed and reported to Mailgun as success, so Mailgun never retries a genuinely failed webhook.

B. p24-ms-mailgun — HIGH

  1. mailConfirmation.create() not awaited — integration.js:32 and integration.js:64. Dedup record is written fire-and-forget, so duplicate-detection is unreliable.
  2. bodyMail.emailSender is undefined — Mailgun posts the sender as from / sender, not emailSender. The field-name mismatch means sender is always undefined.
  3. checkAddressbook() called but never defined — throws ReferenceError for addressbookupdate actions.
  4. No try/catch around JSON.parse(attachments) — a malformed attachments payload throws and takes down the request handler.
  5. No timeout / circuit-breaker on the MongoDB call in the integration/add handler (surfaced by incident #3688, 2026-07-10). When the MongoDB connection pool is dead — e.g. mailgun-v42-prod running a stale w4_app credential because the container was docker restarted (which does not re-read the env file) instead of fully recreated after a rotation — every webhook hangs forever on a connection that will never be granted, instead of failing fast with a 5xx that Mailgun would retry. 100% of ~450 inbound emails produced zero MongoDB records with no error surfaced anywhere. This is the same class of bug as the “return 500 not 201 so Mailgun retries” fix in §A/§B: the DB layer must fail fast (bounded timeout → 5xx) rather than hang. The infra/process side of #3688 (make container recreate mandatory after any w4_app/w3_app rotation, plus a “webhooks received but zero regRecords created” alert) is handled in p24-infra; the app-level bounded-timeout fix belongs in the external p24-ms-mailgun GitLab repo alongside the fixes above. Full incident + root cause: docs/playbooks/mailgun-mongodb-stale-credential-hang.md.

C. Security — BLOCKER for deploy

  • Production credentials hardcoded in docker-deploy-prod.sh in both repos.
  • These must be rotated and moved to GitLab CI/CD variables before any deploy. Treat the exposed values as compromised (they are in git history). See “Credential rotation” below.

Fix specification

pinbox24-ms-s3-v2

  • Add await before Promise.all() in the scanque controller (line ~43) so the response only fires after all uploads resolve.
  • Replace forEach with for...of in the storage controller (line ~115) so each async upload is awaited sequentially (or collect promises and await Promise.all).
  • Return 500 from catch blocks instead of 201, so Mailgun retries failed webhooks.
  • Fix Mailgun URL parsing — replace slice(8) with a real URL parser.
  • Add /tmp file cleanup after upload, and filename sanitization before writing to S3.

p24-ms-mailgun

  • Add await on mailConfirmation.create() (lines ~32 and ~64).
  • Fix sender resolution: bodyMail['from'] || bodyMail.sender instead of bodyMail.emailSender.
  • Guard the checkAddressbook() call (function is not implemented — either implement or no-op guard so addressbookupdate does not throw).
  • Wrap JSON.parse(attachments) in try/catch.
  • Add input validation for required Mailgun fields before processing.

Both repos

  • Move credentials out of docker-deploy-prod.sh into GitLab CI/CD variables.
  • Rotate every exposed key (see rotation playbook reference below).

Dependency graph

Parallel (independent):
  [A] s3-v2:   await Promise.all, for..of, 500-on-error, URL parser, tmp cleanup, sanitize
  [B] mailgun: await create(), sender field, guard checkAddressbook, JSON.parse guard, validation
  [C] Credential rotation (hardcoded secrets removed from git, moved to GitLab CI/CD vars)

Then (A + B + C all done):
  [D] Staging verification
  [E] Production deploy — s3-v2 FIRST, then mailgun

[A], [B], [C] touch disjoint repos/systems and can proceed in parallel. [D]/[E] are gated on all three.

Credential rotation (security blocker)

Hardcoded production credentials in docker-deploy-prod.sh (both repos) are exposed in git history and must be treated as compromised. Rotation is a human action and follows the established incident-rotation playbook: docs/playbooks/static-api-key-incident-rotation.md.

Distribution on rotation is mandatory — update the new values in all stores: SOPS (secrets/*.env.sops), GitLab CI/CD variables, the live container env on bms-1, and any dependent service. Never echo a secret value into chat, a commit, or a log — reference the key name only.

A dedicated tracking issue is opened for this work: #2052 (human-action, security).

Rollback strategy

ServiceRegistryRollback
p24-ms-mailgun (mailgun-v42-prod)AWS ECRECR preserves an old- image tag → instant rollback to the previous image
pinbox24-ms-s3-v2 (s3-v2-v42-prod)private registry (note: private-registry.dev.pinbox24.com is currently DOWN — confirm before relying on it)No automatic old- tag — must manually tag the current image before each deploy
BothGitLab pipeline historyShows the exact image digest for any past deploy — recoverable reference

The asymmetry is important: s3-v2 has no automatic previous-image tag, so the deploy operator must tag the running image before replacing it. The deploy playbook enforces this as a pre-flight step.

Staging endpoints

ServiceStaging URL
p24-ms-mailgunmailgun-v42-stage.dev.pinbox24.com
pinbox24-ms-s3-v2s3-v2-test.dev.pinbox24.com

Staging stacks run on bms-3 (51.68.155.224).

Verification (staging, before prod)

  1. Send a test inbound email through the staging Mailgun route to a *.dev.pinbox24.com recipient that lands on the staged services.
  2. Confirm the attachment is written to Wasabi S3 and the metadata record to MongoDB.
  3. Force an error path (e.g. an unreachable S3 bucket) and confirm the service now returns 500 (so Mailgun would retry) instead of 201.
  4. Confirm no ReferenceError in addressbookupdate and that the sender field is populated.

Graph View

Backlinks