p24-infra Docs

pinbox24-ms-deploy

4 min read

Playbook: Pinbox24 microservice deploy — p24-ms-mailgun & pinbox24-ms-s3-v2

Created: 2026-06-29 Source issue: #2048 Scope: Deploy and roll back the two inbound-email microservices on bms-1 production. Related plan: docs/pinbox24/mailgun-flow-fix-plan.md

Human-execution required. This playbook drives production deploys to bms-1. The microservice source and the docker-deploy-prod.sh script referenced here live in external GitLab repos (p24-ms-mailgun, pinbox24-ms-s3-v2), not in p24-infra. Steps that call those repo scripts are marked [GitLab repo] — run them from a checkout of the respective repo, or via its GitLab CI pipeline. Confirm the exact script/pipeline-stage names against the repo before running.

What this covers

ServiceContainer (bms-1)Public URLStaging URL
p24-ms-mailgunmailgun-v42-prodmailgun-api.w4.pinbox24.commailgun-v42-stage.dev.pinbox24.com
pinbox24-ms-s3-v2s3-v2-v42-prods3-v2-api.w4.pinbox24.coms3-v2-test.dev.pinbox24.com
  • Production host: bms-1 (OVH ns367522, 94.23.26.113)
  • Staging host: bms-3 (51.68.155.224)
  • Image registry: AWS ECR 563740926945.dkr.ecr.eu-central-1.amazonaws.com (eu-central-1)
    • Secondary private-registry.dev.pinbox24.com is currently DOWN (OVH) — do not rely on it for s3-v2 rollback; tag manually instead (see “Rollback”).

When to use

  • Shipping the inbound-email flow fixes from #2048 (async/await bugs, 500-on-error, sender field).
  • Any subsequent code change to either microservice that needs to reach bms-1 production.

Pre-flight — BLOCKERS (do not skip)

  1. Credentials rotated and out of git. The fixes in #2048 are blocked until the hardcoded production credentials are removed from docker-deploy-prod.sh (both repos), rotated, and moved to GitLab CI/CD variables. Follow static-api-key-incident-rotation.md. Verify the deploy script no longer contains literal secret values: 
    # [GitLab repo] — run in each repo checkout; expect NO matches
grep -nE '(PASSWORD|SECRET|KEY|TOKEN)=[^$]' docker-deploy-prod.sh || echo "clean — no literal secrets"
  2. Staging verified. The fix must pass staging verification (see the fix plan, “Verification”) on *.dev.pinbox24.com before any production deploy.
  3. Deploy order is fixed: s3-v2 FIRST, then mailgun. The mailgun service forwards to s3-v2; deploying the storage service first avoids a window where mailgun posts to an old storage API.

Deploy — s3-v2 first

Step 1 — Tag the currently-running image (MANDATORY for s3-v2)

s3-v2 has no automatic old- tag. Capture the running image digest so rollback is possible.

# On bms-1 — record the digest currently serving s3-v2-v42-prod
ssh root@94.23.26.113 "docker inspect --format '{{index .RepoDigests 0}}' s3-v2-v42-prod"
# Save the printed digest somewhere durable (deploy log / issue comment).

Optionally re-tag the running image as a manual rollback point:

# [GitLab repo / bms-1] — tag the current prod image as old-<date> in ECR before replacing it.
# Confirm the exact image repo path from docker-deploy-prod.sh before running.

Step 2 — Deploy s3-v2

# [GitLab repo] — trigger the s3-v2 production deploy via its CI pipeline or:
#   ./docker-deploy-prod.sh
# (Now credential-free; secrets injected from GitLab CI/CD variables.)

Step 3 — Verify s3-v2

ssh root@94.23.26.113 "docker ps --filter name=s3-v2-v42-prod --format '{{.Status}} {{.Image}}'"
# Health/endpoint check:
curl -sS -o /dev/null -w '%{http_code}\n' https://s3-v2-api.w4.pinbox24.com/health   # confirm 200

Confirm an inbound test writes the attachment to Wasabi S3 and metadata to MongoDB, and that a forced error now returns 500 (not 201).

Deploy — mailgun second

Step 4 — Deploy mailgun

# [GitLab repo] — trigger the mailgun production deploy via CI or ./docker-deploy-prod.sh
# AWS ECR retains an `old-` tag automatically for this service (instant rollback available).

Step 5 — Verify mailgun

ssh root@94.23.26.113 "docker ps --filter name=mailgun-v42-prod --format '{{.Status}} {{.Image}}'"
curl -sS -o /dev/null -w '%{http_code}\n' https://mailgun-api.w4.pinbox24.com/health   # confirm 200

End-to-end: send a real inbound email through the production Mailgun route and confirm the attachment lands in Wasabi S3 + MongoDB, the sender field is populated, and addressbookupdate actions do not throw.

Rollback

ServiceMethod
mailgun-v42-prodAWS ECR keeps the old- tag → redeploy the old- image (instant).
s3-v2-v42-prodNo auto old tag → redeploy the digest captured in Step 1. The private-registry.dev.pinbox24.com mirror is DOWN, so rely on ECR + the recorded digest.
EitherGitLab pipeline history shows the exact image digest of any past deploy — use it to pin a rollback target.
# s3-v2 rollback — redeploy the digest recorded in Step 1
ssh root@94.23.26.113 "docker pull <recorded-digest> && docker tag <recorded-digest> <s3-v2-prod-image-tag> && docker compose up -d s3-v2-v42-prod"
# Confirm exact compose service name / image tag from bms-1 before running.

Error reporting (per repo standard)

If a deploy step fails, report it — do not silently work around it:

  1. Discord embed via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL.
  2. GitHub issue in radieu/p24-infra labelled bug.

Escalation

Graph View

Backlinks