Mailgun API Key Rotation & Email Delivery Playbook

What this covers

MAILGUN_API_KEY — used by infra-src/report-scheduler/report_scheduler.py to send daily/weekly reports
Domain: services.pinbox24.com (Mailgun EU, https://api.eu.mailgun.net) — read from MAILGUN_EU_DOMAIN in secrets/monitoring.env.sops; services.pinbox24.com is also the hardcoded default in report_scheduler.py. Always probe the domain the consumer actually uses.
MAILGUN_ADMIN_API_KEY — separate account-level admin key (consumed by monitoring/exporters/mailgun-pipeline-exporter). Rotate independently; it is also the key used to mint replacement sending keys (see §Fix).
SMTP fallback: smtp.eu.mailgun.org:587, user = full email address (e.g. raporty@services.pinbox24.com)

History: an earlier revision of this playbook named ai.pinbox24.com. That domain is not what the report scheduler sends from — probing it produced misleading results during #4407.

Trigger

Alert: ReportNotGenerated or WeeklyReportNotGenerated firing for >1h
Report log shows: email-api error 403: Your request was blocked.
Mailgun dashboard shows API key revoked or expired

Confirm the problem

Do NOT diagnose with GET /v3/domains. That is an account-level endpoint. A healthy sending-role key is scoped to one domain and returns 401 there regardless — so the result is the same whether the key is fine or revoked. Using it as the test wastes a rotation cycle (#4407).

The decisive test is a testmode send to the key’s own domain. Mailgun fully authenticates and validates the request but delivers nothing (o:testmode=yes), so it is safe to run in production. Pair it with the admin key as a control on the same domain and region — if the admin key succeeds where the sending key fails, the domain, region and account are healthy and the sending key itself is revoked.

# Linux worker (vps-i1 / bms-4)
export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
cd /opt/p24-infra
DEC=$(sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops)
SEND=$(printf '%s\n' "$DEC"  | grep '^MAILGUN_API_KEY='       | cut -d= -f2-)
ADMIN=$(printf '%s\n' "$DEC" | grep '^MAILGUN_ADMIN_API_KEY=' | cut -d= -f2-)
D=$(printf '%s\n' "$DEC"     | grep '^MAILGUN_EU_DOMAIN='     | cut -d= -f2-)
unset DEC
 
# Decisive: testmode send with the sending key (nothing is delivered)
curl -s -o /dev/null -w 'sending key: %{http_code}\n' -u "api:$SEND" \
  "https://api.eu.mailgun.net/v3/$D/messages" \
  -F from="raporty@$D" -F to='radieu@gmail.com' \
  -F subject='probe' -F text='probe' -F o:testmode=yes
 
# Control: admin key on the same domain
curl -s -o /dev/null -w 'admin  key: %{http_code}\n' -u "api:$ADMIN" \
  "https://api.eu.mailgun.net/v3/domains/$D"
unset SEND ADMIN D

sending	admin	Diagnosis
200	200	Healthy — no-op, do not rotate
401	200	Sending key revoked → rotate (§Fix)
401	401	Account/domain/region problem — do not mint; check billing, domain verification, EU-vs-US

# Real-world impact — has the scheduler been failing?
ssh root@217.154.82.162 "grep -c '401' /var/log/report-scheduler.log; tail -5 /var/log/report-scheduler.log"

Fix (preferred) — autonomous rotation on a Linux worker

MAILGUN_ADMIN_API_KEY authenticates against /v1/keys (US and EU), so minting is fully programmatic — no dashboard, no MFA (confirmed #4400, executed #4407). A worker can complete this end to end.

/v1/keys requires form encoding. A JSON body fails with 400 Missing mandatory parameter: role.

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
cd /opt/p24-infra
ADMIN=$(sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops \
  | grep '^MAILGUN_ADMIN_API_KEY=' | cut -d= -f2-)
D=services.pinbox24.com
 
# 1. Mint — response contains the secret; write to a 0600 file, never echo it
umask 077
curl -s -o /tmp/mint.json -w 'http=%{http_code}\n' -X POST 'https://api.eu.mailgun.net/v1/keys' \
  -u "api:$ADMIN" \
  -F role=sending -F description="p24-infra-report-scheduler-$(date -u +%Y%m%d)" -F domain_name="$D"
unset ADMIN
 
# 2. Validate BEFORE writing anywhere — testmode, nothing delivered
NEW=$(python3 -c "import json;print(json.load(open('/tmp/mint.json'))['key']['secret'])")
[ -z "$NEW" ] && { echo "[ERROR] mint failed"; exit 1; }
curl -s -o /dev/null -w 'new key: %{http_code}\n' -u "api:$NEW" \
  "https://api.eu.mailgun.net/v3/$D/messages" \
  -F from="raporty@$D" -F to='radieu@gmail.com' -F subject='probe' -F text='probe' -F o:testmode=yes
# Expect 200. Anything else — STOP, do not write to SOPS.

Write to SOPS using the Linux temp-file pattern in infra/agent-prompts/worker-secret-manager.md §General pattern — encrypt to secrets/monitoring-tmp.env.sops, canary-decrypt the temp file, assert 6/6 age recipients and an unchanged key count, and only then mv over production.
Distribute — see §Distribution below. Then shred -u /tmp/mint.json.

Distribution targets for `MAILGUN_API_KEY`

Target	How
`secrets/monitoring.env.sops`	commit → PR → merge
GH Secret `MAILGUN_API_KEY`	`gh secret set MAILGUN_API_KEY --repo radieu/p24-infra`
Live `/opt/p24-infra/monitoring/.env` (vps-i1)	automatic on merge via `secrets-sync.yml`

No container restart is needed — report_scheduler.py is a cron that reads the env at each run. This key has no W3/W4 prod consumer; W3/W4 use V42_MAILGUN_API_KEY / MAILGUN_PASSWORD, which are rotated separately and do fall under the Friday-window prod rules.

⚠️ secrets-sync.yml’s sync-vps-i1 job refuses to run if it lands on the ionos-2 runner (vps-i1 itself) — it needs bms-4 online. If bms-4 is down at merge time, deploy manually per docs/playbooks/gh-runner-assignment-policy.md. Until the live .env is updated, report emails stay broken even though SOPS is correct.

Fix (fallback) — manual rotation from the Windows dev machine

Use only if the admin key is itself unavailable.

Create new key in Mailgun dashboard → Sending → Domain Settings → services.pinbox24.com → API Keys → Add Sending Key. Store it in a temp file (e.g. Desktop/tmp.txt), never paste in chat.
Test the new key before saving:

$env:MG_KEY = (Get-Content "C:\Users\konar\Desktop\tmp.txt" -Raw).Trim()
$r = Invoke-RestMethod `
    -Uri "https://api.eu.mailgun.net/v3/services.pinbox24.com/messages" `
    -Method POST `
    -Credential ([System.Management.Automation.PSCredential]::new("api", (ConvertTo-SecureString $env:MG_KEY -AsPlainText -Force))) `
    -Body @{ from="p24-infra <raporty@services.pinbox24.com>"; to="radieu@gmail.com"; subject="Mailgun key test"; text="OK" }
$env:MG_KEY = ""
# Expect: Queued. Thank you.

Update SOPS:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:MG_KEY = (Get-Content "C:\Users\konar\Desktop\tmp.txt" -Raw).Trim()
$plain = sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops
$updated = ($plain | Where-Object { $_ -notmatch "^MAILGUN_API_KEY=" }) + "MAILGUN_API_KEY=$($env:MG_KEY)"
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring-tmp.env.sops", ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\monitoring-tmp.env.sops
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS canary FAILED — do not commit" }
[System.IO.File]::Delete("$PWD\secrets\monitoring-tmp.env.sops")
$env:MG_KEY = ""

Push to vps-i1 live env:

$env:MG_KEY = (Get-Content "C:\Users\konar\Desktop\tmp.txt" -Raw).Trim()
ssh -i C:\Users\konar\.ssh\id_ed25519 root@217.154.82.162 "sed -i 's|^MAILGUN_API_KEY=.*|MAILGUN_API_KEY=$($env:MG_KEY)|' /opt/p24-infra/monitoring/.env && echo OK"
$env:MG_KEY = ""

Delete the temp key file from Desktop manually (PowerShell Remove-Item may be denied).
Commit and push:

git add secrets/monitoring.env.sops
git commit -m "fix: rotate MAILGUN_API_KEY"
git push origin HEAD   # branch → PR → main (never push to main directly)

Verify next report run (scheduled daily, check /var/log/report-scheduler.log) or trigger manually:

ssh root@217.154.82.162 "cd /opt/p24-infra && python3 infra-src/report-scheduler/report_scheduler.py przeglady-hu-sp-uvv"

Send email via Mailgun API — quick reference

# PowerShell (Windows dev machine)
$env:MG_KEY = "key-..."   # never hardcode — read from SOPS
Invoke-RestMethod `
    -Uri "https://api.eu.mailgun.net/v3/services.pinbox24.com/messages" `
    -Method POST `
    -Credential ([System.Management.Automation.PSCredential]::new("api", (ConvertTo-SecureString $env:MG_KEY -AsPlainText -Force))) `
    -Body @{
        from    = "p24-infra <raporty@services.pinbox24.com>"
        to      = "radieu@gmail.com"
        subject = "Subject here"
        text    = "Body here"
    }
$env:MG_KEY = ""

# bash (Linux / vps-i1)
MG_KEY=$(grep "^MAILGUN_API_KEY=" /opt/p24-infra/monitoring/.env | cut -d= -f2-)
curl -s --user "api:$MG_KEY" \
  https://api.eu.mailgun.net/v3/services.pinbox24.com/messages \
  -F from='p24-infra <raporty@services.pinbox24.com>' \
  -F to='radieu@gmail.com' \
  -F subject='Subject here' \
  -F text='Body here'
unset MG_KEY

SMTP settings (for n8n / Alertmanager / other services)

Field	Value
Host	`smtp.eu.mailgun.org`
Port	`587` (STARTTLS) or `465` (SSL)
Username	full email, e.g. `raporty@services.pinbox24.com`
Password	`SMTP_PASSWORD` from `secrets/monitoring.env.sops`

SMTP password rotation is separate from the API key — rotate via Mailgun dashboard → Domain Settings → SMTP credentials.

Escalation

Check Mailgun EU dashboard for sending quota, domain verification, or billing block
403: Your request was blocked = key revoked or domain suspended
401 Forbidden = key revoked, OR wrong region (always api.eu.mailgun.net), OR an account-level endpoint probed with a domain-scoped sending key — confirm with the testmode + admin-control matrix above before rotating

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="MAILGUN_API_KEY",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — Mailgun sending key regenerated and SOPS updated",
    env="bms-1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'MAILGUN_API_KEY', 'success', 'Scheduled rotation — Mailgun sending key regenerated and SOPS updated', 'bms-1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

mailgun-api-key-rotation

Mailgun API Key Rotation & Email Delivery Playbook

What this covers

Trigger

Confirm the problem

Fix (preferred) — autonomous rotation on a Linux worker

Distribution targets for `MAILGUN_API_KEY`

Fix (fallback) — manual rotation from the Windows dev machine

Send email via Mailgun API — quick reference

SMTP settings (for n8n / Alertmanager / other services)

Escalation

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

mailgun-api-key-rotation

Mailgun API Key Rotation & Email Delivery Playbook

What this covers

Trigger

Confirm the problem

Fix (preferred) — autonomous rotation on a Linux worker

Distribution targets for MAILGUN_API_KEY

Fix (fallback) — manual rotation from the Windows dev machine

Send email via Mailgun API — quick reference

SMTP settings (for n8n / Alertmanager / other services)

Escalation

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

Distribution targets for `MAILGUN_API_KEY`