p24-infra Docs

mailgun-credential-rotation

4 min read

Playbook: Mailgun & SMTP Credential Rotation

Covers: MAILGUN_API_KEY · MAILGUN_ADMIN_API_KEY · SMTP_PASSWORD · PINBOX24_SMTP_PASSWORD

Overview

KeySOPS file(s)Purpose
MAILGUN_API_KEYmonitoringDomain-level sending key for p24-infra transactional email
MAILGUN_ADMIN_API_KEYmonitoring, role-secret-managerAccount-level admin key; returns 401 when expired (confirmed 2026-07-06)
SMTP_PASSWORDmonitoringAlertmanager SMTP relay credential (Mailgun EU)
PINBOX24_SMTP_PASSWORDbms-serversPinbox24 SMTP credential (external provider)

All are ❌ Tier 3 Manual — Mailgun dashboard UI only; no key creation API.

Note on SMTP_PASSWORD: brief Alertmanager downtime is acceptable during this rotation per user confirmation. Schedule rotations during low-alert periods.

When to rotate

  • MAILGUN_ADMIN_API_KEY: When it returns 401 (no expiry warning is sent — confirmed). Immediately on exposure.
  • MAILGUN_API_KEY: Annually or on exposure.
  • SMTP_PASSWORD: Annually or on exposure.
  • PINBOX24_SMTP_PASSWORD: When Pinbox24 email delivery failures appear in logs.

Rotation steps

MAILGUN_API_KEY

1. Create new key (human required)

https://app.eu.mailgun.com → Settings → API Keys → Add Sending Key → copy immediately. (Alternatively: Sending → Domain Settings → select domain → SMTP credentials)

2. Distribute (Claude handles)

Update secrets/monitoring.env.sops — key: MAILGUN_API_KEY. Follow docs/playbooks/sops-edit-operations.md.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }

Restart any exporters that use this key after secrets-sync.yml deploys.

MAILGUN_ADMIN_API_KEY

This key appears in two SOPS files — update both atomically.

1. Create new admin key (human required)

  1. https://app.mailgun.com (main, not EU subdomain) → User Settings → API Security → API Keys
  2. Delete old private key → Create new private key → copy immediately (starts with key-)

2. Distribute to both files (Claude handles)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Update secrets/monitoring.env.sops — key: MAILGUN_ADMIN_API_KEY
# Update secrets/role-secret-manager.env.sops — key: MAILGUN_ADMIN_API_KEY
# Canary both before git add:
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
sops --decrypt --input-type dotenv --output-type dotenv secrets\role-secret-manager.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }

3. Verify (exit code only — do not print response body)

$env:K = (sops -d --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
    Select-String "^MAILGUN_ADMIN_API_KEY=").ToString().Split("=",2)[1]
try {
    $r = Invoke-WebRequest "https://api.mailgun.net/v3/domains" -UseBasicParsing `
        -Headers @{Authorization=("Basic " + [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes("api:$($env:K)")))}
    Write-Host "Status: $($r.StatusCode)"  # Expected: 200
} catch { Write-Host "Status: $($_.Exception.Response.StatusCode.value__)" }
$env:K = ""

SMTP_PASSWORD (Alertmanager)

Schedule during low-alert period. Alertmanager will not send emails while restarting (~30 seconds).

1. Reset SMTP credential (human required)

https://app.eu.mailgun.com → Sending → Domains → select domain → SMTP credentials → find the SMTP username used by Alertmanager → Edit → Reset Password → copy new password.

2. Distribute (Claude handles)

Update secrets/monitoring.env.sops — key: SMTP_PASSWORD. After secrets-sync.yml deployment, restart Alertmanager:

ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart alertmanager"

3. Verify

ssh root@217.154.82.162 "docker logs --tail 20 alertmanager | grep -i smtp"

Expected: no auth error in logs; test alert email arrives.

PINBOX24_SMTP_PASSWORD

1. Obtain new credential (human required)

Log into the SMTP provider dashboard used by Pinbox24 → locate the SMTP account → reset password.

2. Distribute (Claude handles)

Update secrets/bms-servers.env.sops — key: PINBOX24_SMTP_PASSWORD. After secrets-sync.yml deployment to bms-1, recreate (not docker restart) every Pinbox24 container that reads this credential from its env file.

Recreate, do not docker restart. docker restart reuses the environment baked into the container at creation time and does not re-read the on-disk env file, so a restarted container keeps the OLD PINBOX24_SMTP_PASSWORD even after secrets-sync.yml updated the file. Only docker-compose up -d --force-recreate (after the old container is stopped/removed) picks up the new value. This is the same stale-credential class that caused the 2026-07-10 silent mailgun outage — see mailgun-mongodb-stale-credential-hang.md.

This file is for mailgun/SMTP API keys, not the Pinbox24 MongoDB credential. If you are here to rotate a w4_app / w3_app MongoDB password (which mailgun-v42-prod also consumes via MONGODB_URL), use w4-mongodb-credential-rotation.md / w3-mongodb-credential-rotation.md, which recreate mailgun-v42-prod (and v42-prod / v32-prod) together in one operation.

Log rotation

| YYYY-MM-DD HH:MM UTC | #ISSUE | MAILGUN_ADMIN_API_KEY | reason | radieu | monitoring.env.sops + role-secret-manager.env.sops |

Recovery

  • Alertmanager not sending after rotation: docker logs alertmanager for SMTP auth errors; verify SMTP_PASSWORD in SOPS matches the dashboard value.
  • MAILGUN_ADMIN_API_KEY still 401 after update: confirm the Private Key type was copied (not the Public Key) from the Mailgun dashboard.

References

  • secret-rotation-access-matrix.md — Tier 3 classification
  • docs/playbooks/sops-edit-operations.md — SOPS write pattern
  • docs/secrets-rotation-log.md — audit trail
  • docs/playbooks/mailgun-mongodb-stale-credential-hang.md — the silent-hang incident: why a docker restart after a credential change is never sufficient (recreate the container)
  • docs/playbooks/w4-mongodb-credential-rotation.md — recreates mailgun-v42-prod + v42-prod together when the w4_app MongoDB credential rotates

Graph View

Backlinks