p24-infra Docs

mailgun-deploy-missing-mongodb-url

5 min read

Playbook: p24-ms-mailgun deploy job — silent 6ms exit-1 = a required env key is missing

Investigated: 2026-08-01 (issue #5027) Repo (code): pinbox24/p24-ms-mailgun (GitLab, project id 13489062), .gitlab-ci.yml Runner: mailgun-bms1-autodeploy (shell executor on bms-1, 94.23.26.113) Service: mailgun-v42-prod (bms-1, test-net, port 3000) Severity: low — no production impact (the container is maintained by secrets-sync.yml, not by this CI job) Status: root-caused (deterministic). Value fix routed to secret-manager; CI robustness fix in a GitLab MR.

Symptom

A merge to master on pinbox24/p24-ms-mailgun auto-triggers build → deploy (both only: master, no pre-merge CI gate). The build job succeeds; the deploy job fails with a distinctive signature:

  • The step_script stage lasts ~6 ms (e.g. 09.608964Z → 09.672331Z).
  • It emits exactly one line — GitLab’s collapsed-multi-line-command marker ($ # Fail fast if SOPS-deployed env file is missing # collapsed multi-line command) — then ERROR: Job failed: exit status 1.
  • None of the deploy script’s own messages appear: not the test -f guard’s ERROR: … not found, not any test -n … || echo "ERROR: … missing or empty" line, not the unconditional echo "All required env keys found".

Because the near-instant, zero-output failure doesn’t match any of the script’s own echo+exit branches, it looks like a shell/executor hiccup that “just needs a clean re-run to confirm it was transient.” It is not transient. It recurs identically on every deploy.

Root cause

The deploy job’s script: is a single multi-line YAML block. GitLab’s bash shell executor runs that block with set -e -o pipefail. The block begins:

test -f "${SOPS_ENV_FILE}" || { echo "ERROR: ${SOPS_ENV_FILE} not found …"; exit 1; }
 
MAILGUN_MONGODB_URL=$(grep "^MAILGUN_MONGODB_URL=" "${SOPS_ENV_FILE}" | head -1 | cut -d= -f2-)
MAILGUN_AUTH_TOKEN=$(grep "^MAILGUN_AUTH_TOKEN=" "${SOPS_ENV_FILE}" | head -1 | cut -d= -f2-)
…
test -n "${MAILGUN_MONGODB_URL}" || { echo "ERROR: MAILGUN_MONGODB_URL missing or empty"; exit 1; }
…
echo "All required env keys found"
  1. SOPS_ENV_FILE = /opt/p24-infra/bms-1/pinbox24-backends.env exists → the test -f guard passes silently. (This is the only guard the earlier investigation reproduced by hand.)
  2. The first extraction runs grep "^MAILGUN_MONGODB_URL=" …. MAILGUN_MONGODB_URL is absent from that file, so grep matches nothing and exits 1.
  3. Under pipefail, the pipeline grep | head | cut inherits grep’s non-zero status; the assignment VAR=$(… failing pipeline …) therefore has a non-zero exit status; under set -e the shell aborts the entire job on that line — before reaching any test -n … || echo … guard.
  4. Result: the diagnostic guards designed to name the missing key are dead code in the missing-key case. The job dies during extraction with zero output in a few milliseconds.

Why the key is missing. MAILGUN_MONGODB_URL is not present in secrets/pinbox24-backends.env.sops (that file carries only MAILGUN_AUTH_TOKEN, MAILGUN_S3_SERVER_URL, MAILGUN_S3_V2_SERVER_URL). The sync-pinbox24-backends job in secrets-sync.yml deploys that SOPS file verbatim to bms-1, so the on-host file lacks the key too. The CI deploy script requires four MAILGUN_* keys but the source only supplies three.

Evidence captured 2026-08-01 (all read-only)

CheckResult
Key names in secrets/pinbox24-backends.env.sopsMAILGUN_AUTH_TOKEN, MAILGUN_S3_SERVER_URL, MAILGUN_S3_V2_SERVER_URL — no MAILGUN_MONGODB_URL
grep -q "^MAILGUN_MONGODB_URL=" /opt/p24-infra/bms-1/pinbox24-backends.env on bms-1ABSENT
Failed deploy job 15659825381 tracestep_script 6.4 ms, one collapsed-command line, exit status 1, no script output
Prior failed deploy 15255729031 (2026-07-09)Different bug — old script sourced the env file → bash syntax error on special chars + MAILGUN_MONGODB_URL is empty; a re-run 6 min later succeeded. The grep
mailgun-v42-prod containerhealthy, RestartCount=0, running current mailgun-v42-prod:latest (= daeabc43), recreated 20:58 by secrets-sync.yml (#4948 rotation), with a working MONGODB_URL — maintained outside this CI, which is why the perpetually-red deploy job has no production effect

Fix (two parts, in this order)

Part A — value (secret-manager, blocking). Add MAILGUN_MONGODB_URL to secrets/pinbox24-backends.env.sops = the w4_db Mongo URI mailgun-v42-prod uses (same value as V42_NEW_MONGODB_URI in secrets/pinbox24-w4.env.sops; secrets-sync.yml’s sync-bms-1 already uses that exact fallback for mailgun-pipeline-exporter). Then run secrets-sync.yml (target=pinbox24-backends) so the on-host file gains the key. This unblocks the deploy.

Part B — robustness (sys-admin, GitLab MR on .gitlab-ci.yml). Make the extraction survivable so the existing diagnostic guards can actually fire. Append || true to each extraction (grep may legitimately match nothing), e.g.:

MAILGUN_MONGODB_URL=$(grep "^MAILGUN_MONGODB_URL=" "${SOPS_ENV_FILE}" | head -1 | cut -d= -f2- || true)

With that, a missing key falls through to test -n … || { echo "ERROR: MAILGUN_MONGODB_URL missing or empty"; exit 1; }, which names the culprit in the trace instead of dying silently. Merge Part B after Part A lands (a merge to master triggers a real deploy; if the key is still missing the new guard will simply report it clearly rather than fail silently — either way, land A first so the deploy goes green).

How to recognise a recurrence

A 6 ms, single-collapsed-line, zero-script-output exit status 1 on any GitLab shell-executor job = a VAR=$(cmd) extraction died under pipefail/errexit before the script’s own guards could run. Do not chase the executor or the runner host. Instead:

  1. Read the deploy script and list every VAR=$(grep "^KEY=" …) extraction.
  2. On the target host, grep -q "^KEY=" <env-file> (name only — never print values) for each KEY. The first one that returns non-zero is the killed line.
  3. The missing key is either absent from the SOPS source or not yet synced to the host. Fix the source (secret-manager) and/or add || true + a test -n guard so future misses are loud.
  • docs/playbooks/mailgun-mongodb-stale-credential-hang.md — different failure (runtime hang, not deploy).
  • docs/playbooks/mailgun-pipeline-exporter-deploy.md — the MAILGUN_MONGODB_URL → V42_NEW_MONGODB_URI fallback in secrets-sync.yml.
  • docs/w3-w4-stack-operations.md — ownership + GitLab MR permission matrix for pinbox24/*.

Graph View