p24-infra Docs

mailgun-pipeline-exporter-deploy

8 min read

Playbook: Deploy mailgun-pipeline-exporter to bms-1

Status: STABLE (executed successfully 2026-07-11, issue #3688)

Trigger

Standing up mailgun-pipeline-exporter (issue #3688, PR #3705) on bms-1 for the first time, or redeploying it after a code change to monitoring/exporters/mailgun-pipeline-exporter/.

Pre-conditions

  • PR adding/changing the exporter is merged to main.
  • bms-1/.env.example in this repo lists the required env keys.
  • MAILGUN_ADMIN_API_KEY, MAILGUN_PIPELINE_MONGODB_URI, and (since #3752) P24_AUTOMATION_KEY are delivered by CI — do not place them by hand (#4657, #3752). secrets-sync.yml’s sync-bms-1 job decrypts them from SOPS and upserts them into /opt/mezmo-agent/.env on every secrets push:
    KeySOPS source
    MAILGUN_ADMIN_API_KEYsecrets/monitoring.env.sops
    MAILGUN_PIPELINE_MONGODB_URIsecrets/pinbox24-w4.env.sops → V42_NEW_MONGODB_URI (fallback; the job prefers MAILGUN_MONGODB_URL in secrets/pinbox24-backends.env.sops automatically once #3850 adds it there)
    P24_AUTOMATION_KEY (#3752, Leg 1/Leg 3 CF Worker token minting)secrets/pinbox24-w4-auth.env.sops
    A secret-manager operation is only needed to rotate a value in SOPS, never to place one
    on the host. If a key ever needs restoring on bms-1, re-run the sync rather than editing the
    file: gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=bms-1.
    The MAILGUN_E2E_LEG{1,3}_ENABLED / MAILGUN_E2E_LEG3_WEBHOOK_URL flags are plain
    (non-secret) config in .env, NOT synced by CI — see docs/mailgun-e2e-probe-operations.md.

Key fact: repo path vs. live server path

bms-1/docker-compose.yml in this repo is the source of truth for what’s deployed live at /opt/mezmo-agent/docker-compose.yml on bms-1 (94.23.26.113) — NOT /opt/p24-infra/bms-1/ (that directory on bms-1 is just a staging area for per-service .env files dropped by secrets-sync.yml/manual placement, not a compose project root).

The compose file’s build: ../monitoring/exporters/<name> and env_file: - .env are both relative to /opt/mezmo-agent/ on the live host, so:

  • exporter source must land at /opt/monitoring/exporters/<name>/ (sibling of mezmo-agent/)
  • all env keys (including pre-existing LOGDNA_AGENT_KEY for mezmo-agent) share ONE /opt/mezmo-agent/.env file — do not create a separate per-exporter env file here.
  • that shared file is written by CI. secrets-sync.yml’s sync-bms-1 job rewrites it on any secrets push. Since #3850 it merges — it upserts LOGDNA_AGENT_KEY and leaves every other key alone. Before #3850 it >-truncated the file to that single line, so any hand-placed exporter key was deleted on the next run; because the job recreates mezmo-agent only, the exporter kept its old env until it was next recreated, making the loss invisible for as long as nothing touched the container.
  • Since #4657 the job also upserts the exporter’s own two keys from SOPS, and recreates mailgun-pipeline-exporter (not just mezmo-agent) when they change. That closes the gap that made this recurrent: the #3850 merge preserved unrelated keys correctly, but nothing ever put the exporter keys there, so once the pre-#3850 truncation had removed them they never came back. The tell is the job’s own log line — env merged (49 bytes, 1 keys) means only LOGDNA_AGENT_KEY is present and the exporter is running blind; a healthy sync reports 3 keys. A key that fails to decrypt is skipped, never written empty, so a SOPS miss cannot blank a working on-host value.
  • LOGDNA_AGENT_KEY must stay the last line and must not end with a newline (compose v1 folds it into the value → 33-char key). The sync job maintains that ordering; preserve it if you edit by hand.

Known issue: bms-1’s Docker Engine is severely outdated (20.10.2)

Confirmed 2026-07-11: docker version --format '{{.Server.Version}}' on bms-1 returns 20.10.2 (released ~Feb 2021). That version’s default seccomp profile predates support for the clone3 syscall, which glibc >= 2.34 (Debian bookworm/trixie) uses internally. The symptom is a build failure at apt-get update inside any container build using a bookworm/trixie-based image (python:3.13-slim, python:3.13-slim-bookworm, even bare debian:trixie-slim all reproduce it identically):

E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb ...'
E: Sub-process returned an error code

Running the same rm -f ... command directly (outside apt’s internal script-runner) succeeds fine — the failure is specific to how apt’s Post-Invoke hook is spawned, consistent with a blocked clone3 syscall under the container’s seccomp profile.

Workaround (in place for this exporter): pin the exporter’s Dockerfile FROM to python:3.11-slim-bullseye (glibc 2.31, predates the clone3 dependency). Confirmed working on bms-1 2026-07-11.

This affects every future build on bms-1, not just this exporter — any new/changed Dockerfile targeting bms-1 needs a bullseye-or-older base until bms-1’s Docker Engine is upgraded. See issue tbd (filed alongside this playbook) for the tracked upgrade decision — this is a separate, carefully-scheduled maintenance task on a production host (v42-prod, v32-prod, mailgun-v42-prod all run there) and should not be done opportunistically.

Steps

  1. Copy updated compose file:

    scp -i <ssh-key> bms-1/docker-compose.yml root@94.23.26.113:/opt/mezmo-agent/docker-compose.yml

  2. Copy exporter source (mkdir -p first; /opt/monitoring/exporters/<name>/ is a sibling of /opt/mezmo-agent/):

    ssh ... "mkdir -p /opt/monitoring/exporters/mailgun-pipeline-exporter"
scp monitoring/exporters/mailgun-pipeline-exporter/{app.py,Dockerfile,requirements.txt} \
  root@94.23.26.113:/opt/monitoring/exporters/mailgun-pipeline-exporter/

  3. Merge the new exporter’s env keys into the existing /opt/mezmo-agent/.env (append, don’t overwrite — LOGDNA_AGENT_KEY must survive). Never cat/print the file to your own terminal.

    Since #4657, prefer CI for mailgun-pipeline-exporter specifically — run gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=bms-1 and skip this step. The manual merge below remains the pattern for a new exporter whose keys sync-bms-1 does not yet deliver; wire such keys into that job in the same PR, or they will be lost on the next sync exactly as 4657 were.

    Pitfall hit during execution: if the destination file has no trailing newline (check with wc -l vs wc -c — a non-zero byte count with wc -l returning 0 means no trailing newline), a plain cat new >> existing concatenates the first appended line onto the end of the last existing line, corrupting BOTH (the existing key’s value gets garbage appended, and the new key becomes unparseable). Always rebuild explicitly instead:

    { cat existing.env; printf '\n'; cat new-keys.env; printf '\n'; } > merged.env.new \
  && mv merged.env.new existing.env && chmod 600 existing.env

    Verify with wc -l, grep -c '=', and cut -d= -f1 (key names only) — never print values. Take a timestamped backup of the destination file before any merge.

  4. Open ufw for the exporter’s port, source-restricted to vps-i1 only:

    ufw allow from 217.154.82.162 to any port 9251 proto tcp comment 'mailgun-pipeline-exporter <- vps-i1'

  5. Build + start (from /opt/mezmo-agent/):

    docker-compose up -d --build mailgun-pipeline-exporter

    Note: even naming the specific service, docker-compose may still recreate mezmo-agent too if the shared .env file’s content changed (it hashes the resolved config per service). This happened during execution — mezmo-agent restarted cleanly with zero data loss (verified via docker inspect --format '{{.RestartCount}}' == 0, i.e. it wasn’t crash-looping, and its logs showed successful log-shipping resuming immediately). Treat this as an accepted, low-risk side effect, not a bug to prevent.

  6. Verify:

    docker ps --filter name=mailgun-pipeline-exporter --format '{{.Names}} {{.Status}}'
docker inspect <container> --format '{{.State.Health.Status}}'   # wait for "healthy"
curl -s http://localhost:9251/metrics | head -5   # from bms-1 itself

    From vps-i1, confirm Prometheus can reach it:

    curl -s 'http://localhost:9090/api/v1/targets?state=active' | \
  python3 -c "import sys,json; d=json.load(sys.stdin); [print(t['labels']['job'], t['health']) for t in d['data']['activeTargets'] if t['labels'].get('job')=='mailgun_pipeline']"

    Expect mailgun_pipeline up.

  7. Log the operation (see .claude/task-playbooks/server-operation.md §Mandatory post-flight).

Common mistakes to avoid

  • Assuming /opt/p24-infra/bms-1/ is the compose project root — it isn’t; it’s an env-staging directory only.
  • Creating a separate .env file for the new exporter instead of appending to the existing shared /opt/mezmo-agent/.env. This actually happened (#3817, 2026-07-11): the keys were dropped into /opt/p24-infra/bms-1/mailgun-pipeline-exporter.env (the staging dir), never merged into /opt/mezmo-agent/.env, and the compose still had env_file: - .env, so the container ran with only PORT set and MailgunPipelineExporterErrors fired on both sources. Symptom to recognise: docker inspect <container> --format '{{.Config.Env}}' shows the exporter’s keys absent while a populated file exists elsewhere on disk. Fix = merge into the shared .env + docker-compose up -d --force-recreate (env_file is only re-read on recreate).
  • Appending to a shared env file with plain cat >> without checking for a missing trailing newline first — see the pitfall in Step 3.
  • Hand-placing a key that no CI job delivers. The keys then live in exactly one place — a file that CI rewrites — with no source of truth to restore them from. This is what made #3817 recur as #4657 on 2026-07-30: the keys were gone from /opt/mezmo-agent/.env for at least a day (four consecutive syncs logged 1 keys) before a routine docker-compose up -d recreated the exporter and surfaced it. Because the exporter’s “env not set” path returns {} rather than zeros, its data series went absent, which also silently disarmed the pinbox24.yml pipeline-integrity alert that depends on them — a monitoring blind spot, not a noisy alert. Always wire a new key into sync-bms-1 in the same PR that introduces it.
  • Assuming the current base image (python:3.13-slim or similar) will build on bms-1 without checking Docker Engine compatibility first — see “Known issue” above.
  • docs/playbooks/add-new-monitoring-exporter.md — general “add a new exporter” pattern
  • docs/playbooks/mailgun-mongodb-stale-credential-hang.md — incident this exporter detects

Graph View

Backlinks