p24-infra Docs

pinbox24-mailgun-deploy-script-bug

4 min read

Playbook: Pinbox24 Mailgun Deploy Script Argument Bug

Service: p24-ms-mailgun on bms-1 (94.23.26.113) File: docker-deploy-stage.sh in the mailgun build directory Last updated: 2026-06-29

Trigger

Manually running docker-deploy-stage.sh for p24-ms-mailgun (staging) produces a broken mailgun-environment.env file — VIRTUAL_HOST is empty, SERVER_URL contains garbage, and nginx-proxy never routes requests to the container.

Also triggered when CI calls the script with a different argument order than the script expects.

Symptoms

  • VIRTUAL_HOST= is empty in mailgun-environment.env
  • SERVER_URL=http://<full-ecr-image-path>:3000 (ECR registry path where hostname should be)
  • S3_SERVER_URL similarly garbled
  • nginx-proxy does not create a vhost entry for the mailgun container
  • Container starts but is unreachable; all mailgun API calls return 502 or connection refused
  • docker logs mailgun-v42-stage shows the app started successfully — misleading, the routing layer is broken

Root Cause

The script has a positional argument mismatch between how CI calls it and how the script reads its arguments.

CI invocation (from the GitLab CI job):

$S_EXE_FILE $PRIVATE_REGISTRY/$S_IMAGE_NAME $S_MAIN_CONTAINER $S_BACK_DOMAIN

This passes 3 arguments: $1 = full image path, $2 = container name prefix, $3 = back-end domain.

Script variable assignments:

MAIN_CONTAINER=$1      # actually receives the full image path
PRIVATE_REGISTRY=$3    # actually receives the domain
BACK_DOMAIN=$4         # 4th arg — never provided by CI; always empty

Consequences:

  • BACK_DOMAIN is always empty because CI never passes a 4th argument
  • MAIN_CONTAINER holds the full ECR path, so SERVER_URL=http://${MAIN_CONTAINER}:3000 becomes a garbage URL
  • VIRTUAL_HOST=${BACK_DOMAIN} is empty, so nginx-proxy ignores the container

This is a latent bug in the script. CI may have worked at some historical point with a different calling convention, but the current CI pipeline and script are out of sync.

Immediate Fix (Workaround — use when you need staging up now)

The script only writes the env file; it does not run docker itself. Use this two-step approach:

Step 1 — Generate the env file (even if wrong):

bash docker-deploy-stage.sh <image-path> mailgun-v42-stage mailgun-v42-stage.dev.pinbox24.com

Step 2 — Manually correct the env file:

# Open /root/<build-dir>/mailgun-environment.env and set:
VIRTUAL_HOST=mailgun-v42-stage.dev.pinbox24.com
SERVER_URL=http://mailgun-v42-stage:3000
S3_SERVER_URL=http://s3-v2-v42-stage:3000

Verify no trailing whitespace or CRLF line endings in the env file (file mailgun-environment.env should report ASCII or UTF-8, not CRLF).

Step 3 — Start the container:

cd /root/<mailgun-build-dir>
CLUSTER_NETWORK=test-net \
CONTAINER_NAME=mailgun-v42-stage \
IMAGE_NAME=mailgun-v42-stage \
docker-compose up -d

Step 4 — Verify routing:

docker ps | grep mailgun-v42-stage
curl -s -o /dev/null -w "%{http_code}" http://mailgun-v42-stage.dev.pinbox24.com/health

Expect 200 or 404 (app-level), not 502.

Recovery Steps

If the container is running but unreachable after a deploy:

  1. Check the env file: cat /root/<build-dir>/mailgun-environment.env — look for empty VIRTUAL_HOST
  2. Stop the container: docker stop mailgun-v42-stage && docker rm mailgun-v42-stage
  3. Fix the env file manually (Step 2 above)
  4. Restart: CLUSTER_NETWORK=test-net CONTAINER_NAME=mailgun-v42-stage IMAGE_NAME=mailgun-v42-stage docker-compose up -d
  5. nginx-proxy auto-detects the VIRTUAL_HOST label within ~5 seconds; no nginx reload needed

Known Limitation — Staging MongoDB

The staging mailgun container is configured to use the old artnet.pl MongoDB cluster, which is defunct. The container will start and respond to HTTP but any operation that writes to or reads from MongoDB will fail silently or return errors.

Do not use staging mailgun for end-to-end DB testing until the staging Mongo connection string is updated to point to the bms-2/bms-3 replica set.

Permanent Fix Required

Create a GitLab MR in pinbox24/p24-ms-mailgun to fix docker-deploy-stage.sh:

  • BACK_DOMAIN should read $3, not $4
  • MAIN_CONTAINER should be derived from $2 (container name prefix), not $1 (full image path)
  • PRIVATE_REGISTRY should read $1 with the image path stripped, or the script should accept it as a separate argument

Until the MR is merged and CI re-runs, all manual deploys require the workaround above.

Track in GitLab: open an MR or issue in pinbox24/p24-ms-mailgun titled fix: docker-deploy-stage.sh argument variable mismatch (BACK_DOMAIN always empty).

Prevention

  • Before any manual mailgun deploy, always inspect the generated env file before running docker-compose
  • Add a validation step to the deploy script: [ -z "$BACK_DOMAIN" ] && { echo "ERROR: BACK_DOMAIN empty"; exit 1; }
  • Cross-reference with pinbox24-bms1-manual-deploy.md before starting
  • pinbox24-bms1-manual-deploy.md — full manual deploy procedure for bms-1 services
  • pinbox24-docker-compose-staging-prod-collision.md — how running compose from a shared dir kills production
  • pinbox24-s3-wasabi-bms1.md — s3-v2 service deploy on bms-1

Graph View

Backlinks