p24-infra Docs

w4-s3v2-mailgun-password-verify-first

6 min read

W4 s3-v2-v42-prod — mailgunPassword/MAILGUN_PASSWORD verify-first investigation (#2468)

Status: investigation complete — STOPPED, no rotation performed (dead-credential finding, not a clean match/no-match case; flagged for human/dev-coder decision) Owner: secret-manager (this pass); follow-up needs dev-coder or human input on which code path is authoritative before any write. Applies to: s3-v2-v42-prod container on bms-1 (94.23.26.113), secrets/pinbox24-w4.env.sops.

1. Context

Issue #2468 asks to rotate the Mailgun SMTP/API password originally hardcoded in docker-deploy-prod.sh (incident #2401, closed 2026-07-01). Two 2026-08-04 comments on #2468 narrowed the target: SOPS holds both mailgunPassword (exact name match to the #2401 literal) and MAILGUN_PASSWORD in secrets/pinbox24-w4.env.sops, plus MAILGUN_USER_NAME, V42_MAILGUN_API_KEY, V42_MAILGUN_EU_DOMAIN/V42_MAILGUN_US_DOMAIN. The task was to verify-first (same method as w3-pm2-jlist-exposure-verify-first.md / w4-v42-jwt-exposure-verify-first.md) whether this credential is actually in live use before rotating — the task anticipated two outcomes: (a) nothing live matches SOPS → stale, stop, no rotate; (b) a live match → proceed to rotate via Mailgun’s API. Neither outcome cleanly applied — see §4.

2. Verify-first method (never prints a value)

Same pattern as the W3/W4 precedents: key names + sha256[:12] digests only, computed inline in PowerShell/python, never echoed. HTTP auth probes use the value only as a -Headers Basic-Auth argument (never written to a variable that gets printed) and report only HTTP status codes.

3. Finding — 2026-08-04

3a. SOPS ↔ live container: exact match (not stale)

docker exec s3-v2-v42-prod env (mailgun-related keys only, remote python digest) vs sops -d secrets/pinbox24-w4.env.sops (same digest method), all sha256[:12]:

KeyLive containerSOPS pinbox24-w4.env.sopsMatch
mailgunPassword561268eb287e561268eb287e✅
MAILGUN_PASSWORD561268eb287e561268eb287e✅
mailgunUsername / MAILGUN_USER_NAME14c2529eb44914c2529eb449✅
V42_MAILGUN_API_KEYf6169bcf017bf6169bcf017b✅
V42_MAILGUN_EU_DOMAIN, V42_MAILGUN_US_DOMAIN, V42_MAILGUN_EU_HOST, V42_MAILGUN_BASE_URL, V42_MAILGUN_SUFIX——✅ (all match)

mailgunPassword and MAILGUN_PASSWORD are confirmed duplicates of the same value (not drifted from each other), and both are byte-identical to what is running live in s3-v2-v42-prod right now. This container is a genuine, active consumer — unlike the W4 v42-prod finding in w4-v42-jwt-exposure-verify-first.md, which found MAILGUN_PASSWORD absent from that container’s process env entirely. s3-v2-v42-prod is a different container from v42-prod and does wire this key.

MAILGUN_USER_NAME / mailgunUsername = literal "api" (verified by boolean digest comparison against sha256("api"), value never printed). Confirmed in app code (/app/dist/globalHelpers/mailgun.helper.js inside the container): mailgunUsername/mailgunPassword are used as HTTP Basic-Auth credentials (request({url, auth:{user,pass}})) to fetch a Mailgun-stored attachment URL in getFileFromMailGun(), called from mailgunFileHandlerHelper — this is Mailgun’s standard api:<key> Basic-Auth convention for its HTTP API/storage endpoints, not an SMTP login/password pair.

3b. Provider-side: the live value is invalid (401) everywhere tested

Methodology validated with a known-good control first (V42_MAILGUN_API_KEY, which the app uses for outbound sending): GET /v3/domains/{domain} with V42_MAILGUN_API_KEY → 200 on both api.eu.mailgun.net/services.pinbox24.com and api.mailgun.net/post.pinbox24.com. Same exact request shape repeated with mailgunPassword:

ProbeResult
GET /v3/domains/services.pinbox24.com (EU host) with mailgunPassword401
GET /v3/domains/post.pinbox24.com (US host) with mailgunPassword401
GET /v3/domains (account-level list) with mailgunPassword401
GET /v3/domains/{domain}/credentials with MAILGUN_ADMIN_API_KEY — does "api" appear as a listed SMTP login?No (EU domain logins: postmaster@, p24-infra@; US domain logins: postmaster@, info@, problems@)

The live, SOPS-matching mailgunPassword value is not a valid Mailgun credential at all — not as an API key (401 everywhere, including the account-level endpoint that a fully-scoped key should reach) and not as an SMTP credential ("api" is not a registered SMTP login on either W4 domain).

3c. Application-level signal

mailgunFileHandlerHelper’s Mailgun-download code path (§3a) has no error handling — the request() callback writes body to disk and resolves unconditionally, regardless of HTTP status. A 401 response body would be silently written as if it were the file. Container logs (/var/log/s3-v2-v42-prod/pm2/pm2_s3-v2-v42-prod_production_err.log) show only 7 mailgun-related lines total, all from 2026-04-22, and the error text (addScanFromMailgun attachments not found in [object Object]) is a different failure class (missing attachment metadata), not an auth error — consistent with the code’s silent-failure pattern (a 401 would never surface as a distinct log line here) and with this path being exercised rarely.

4. Why this doesn’t fit either anticipated outcome

  • Not “nothing live matches” (§3 outcome a): the SOPS value is exactly what’s wired into the running container — this is a genuine active consumer, not stale-deploy clobber (#4169 class) or an unused legacy duplicate (the w4-v42-jwt-exposure-verify-first.md v42-prod finding).
  • Not a clean “live match → rotate” (§3 outcome b) either: the live value is already dead at the provider. Rotating via PUT /v3/domains/{domain}/credentials/{login} doesn’t apply — "api" isn’t a registered SMTP login. Rotating via /v1/keys means minting a brand-new key from scratch, not replacing a value the app currently trusts, because the current value was never confirmed to authenticate against either W4 domain during this investigation.

This means:

  1. The original #2401 exposure is already moot for this specific value — the credential now printed in docker-deploy-prod.sh’s history is already worthless to an attacker; it doesn’t authenticate anywhere. Re-rotating it closes no live exposure.
  2. Minting a new working key would be a functional change, not a pure security rotation — it would plausibly fix a currently-broken (or already-dead-and-unused) attachment-download code path, which is outside #2468’s stated scope (rotate an exposed secret) and carries its own regression risk that deserves a deliberate decision, not an automatic side effect of a rotation task.
  3. It’s also possible this code path is genuinely dead/deprecated (matches the W3/W4 pattern of legacy, rarely-exercised integrations found stale elsewhere in this repo — see V32_REDIS_PASSWORD housekeeping note in w3-pm2-jlist-exposure-verify-first.md and the MAILGUN_PASSWORD-absent finding in w4-v42-jwt-exposure-verify-first.md) — in which case the right fix might be removing the dead SOPS keys and the dead code path instead of minting a replacement.

Per this task’s own stop condition (“if you can’t cleanly determine which container/domain is authoritative, stop and comment rather than guessing”) and the Agent Safety Rules (never take an action with regression risk without human sign-off), no rotation, no SOPS write, no container recreate was performed.

  1. dev-coder / human (required before any write): decide whether s3-v2-v42-prod’s mailgunFileHandlerHelper Mailgun-attachment-download feature is still wanted. If yes → secret-manager mints a fresh /v1/keys sending key (or, if the app genuinely needs a domain SMTP credential, a new login via POST /v3/domains/{domain}/credentials) scoped to whichever of services.pinbox24.com/post.pinbox24.com the feature targets, updates both mailgunPassword and MAILGUN_PASSWORD in secrets/pinbox24-w4.env.sops (confirmed duplicates of one value, §3a), and recreates (not restarts) s3-v2-v42-prod — Friday-window prod-container rules apply per secret-rotation-access-matrix.md. If no → remove the dead mailgunPassword/MAILGUN_PASSWORD/ MAILGUN_USER_NAME keys from SOPS as housekeeping and let the code path fail closed (already does).
  2. Separate from #2468: the same silent-failure pattern (auth errors written to disk as if they were the file, §3c) is worth a code-level fix regardless of the outcome of (1) — same class of bug already flagged for the mailgunFileHandler/S3 side in mailgun-flow-fix-plan.md.
  3. Confirm whether mailgunPassword’s original (now-dead) value was ever valid, or has been dead since before #2401 — not pursued here (would require GitLab history on the app repo, out of scope for a verify-first pass).

Graph View

Backlinks