pinbox24-docker-compose-staging-prod-collision

Playbook: Pinbox24 Docker-Compose Staging/Production Collision

Services: pinbox24-ms-s3-v2, p24-ms-mailgun on bms-1 (94.23.26.113) Last updated: 2026-08-07

---

Trigger

Running any docker-compose command (including via docker-deploy-stage.sh or docker-deploy-prod.sh) from a shared build directory when both staging and production containers were previously started from that same directory.

---

Symptoms

• Production container disappears from docker ps immediately after a staging deploy
• Production endpoint (*.pinbox24.com) returns 502
• docker ps -a | grep <service> shows the old production container in Exited state
• No error in the deploy script output — compose reports success
• The staging container is running normally; only production is affected

---

Root Cause

docker-compose.yml for these services defines a single service with a variable container name:

services:
  app:
    container_name: ${CONTAINER_NAME}

Docker Compose tracks which containers it “owns” based on the working directory of the compose project (the directory from which docker-compose was last run). When the compose project directory is the shared build directory:

1. First run: CONTAINER_NAME=s3-v2-v42-prod — compose creates and owns s3-v2-v42-prod
2. Second run: CONTAINER_NAME=s3-v2-v42-stage — compose sees it owns a container from this project but the name no longer matches. Compose stops s3-v2-v42-prod and creates s3-v2-v42-stage

This is standard compose behavior — it reconciles actual containers against its project state. It is not a bug in Docker; it is a misuse of a single compose file for two environments.

Affected services (both follow the same pattern):

• pinbox24-ms-s3-v2
• p24-ms-mailgun

---

Immediate Fix

Stop sharing the build directory between environments. Create isolated per-environment directories and run all compose commands exclusively from those directories.

For s3-v2:

mkdir -p /root/s3v2-prod
mkdir -p /root/s3v2-stage
cp /root/<s3v2-build-dir>/docker-compose.yml /root/s3v2-prod/
cp /root/<s3v2-build-dir>/docker-compose.yml /root/s3v2-stage/

For mailgun:

mkdir -p /root/mailgun-prod
mkdir -p /root/mailgun-stage
cp /root/<mailgun-build-dir>/docker-compose.yml /root/mailgun-prod/
cp /root/<mailgun-build-dir>/docker-compose.yml /root/mailgun-stage/

From this point forward, NEVER run docker-compose from the shared build directory for these services. All production deploys use the -prod directory; all staging deploys use the -stage directory.

---

Recovery Steps

If production was killed by a staging deploy:

Step 1 — Confirm production is stopped:

docker ps -a | grep s3-v2-v42-prod
# Should show "Exited" — not "Up"

Step 2 — Check whether ECR pull is needed: The production image is usually already cached locally. ECR credentials expire; if the deploy was recent, the cached image is current. Try docker images | grep s3-v2-v42-prod to confirm.

Step 3 — Start production from its isolated directory:

cd /root/s3v2-prod
CLUSTER_NETWORK=test-net \
CONTAINER_NAME=s3-v2-v42-prod \
IMAGE_NAME=s3-v2-v42-prod \
docker-compose up -d

If the image is cached locally, docker-compose uses it without pulling from ECR. If ECR pull fails with “access denied”, this is expected — the cached image is used automatically.

Step 4 — Verify production is back:

docker ps | grep s3-v2-v42-prod
curl -s -o /dev/null -w "%{http_code}" http://s3-v2-v42-prod.pinbox24.com/health

Expect 200. nginx-proxy detects the container within ~5 seconds of it starting.

Step 5 — Verify staging is still running:

docker ps | grep s3-v2-v42-stage

Both containers should now appear in docker ps.

---

Repeat the same steps for mailgun if affected:

cd /root/mailgun-prod
CLUSTER_NETWORK=test-net \
CONTAINER_NAME=mailgun-v42-prod \
IMAGE_NAME=mailgun-v42-prod \
docker-compose up -d

---

Prevention

Before any deploy on bms-1:

1. Confirm you are in the correct isolated directory (pwd), not the shared build directory
2. Confirm the target container name matches the directory (/root/s3v2-prod -> s3-v2-v42-prod)
3. Run docker ps before the deploy — note which containers are running
4. Run docker ps immediately after the deploy — confirm no production container disappeared

Add to your deploy checklist in pinbox24-bms1-manual-deploy.md:

[ ] Confirmed working from /root/<service>-prod or /root/<service>-stage (NOT the build dir)
[ ] docker ps before deploy noted — verified no prod container missing after deploy

---

Escalation — Permanent Fix

The correct long-term fix is one of the following (escalate via GitLab MR in the affected repos):

Option A — Separate compose files per environment:

docker-compose.prod.yml   # container_name: s3-v2-v42-prod (hardcoded or env-specific)
docker-compose.stage.yml  # container_name: s3-v2-v42-stage

Each file is a separate compose project even from the same directory.

Option B — Docker Compose profiles:

services:
  app-prod:
    profiles: [prod]
    container_name: s3-v2-v42-prod
  app-stage:
    profiles: [stage]
    container_name: s3-v2-v42-stage

Deploy with docker-compose --profile prod up -d or --profile stage.

Option C — Isolated directories (already applied as workaround, codify in CI): Update CI to clone into environment-specific directories rather than a shared build path.

Until the permanent fix is in place, the isolated-directory workaround is mandatory for all manual deploys.

---

Case Study 2026-07-08 — Undefined ${CLUSTER_NETWORK} Deleted v42-prod + s3-v42-prod

Services: v42-prod, s3-v42-prod, wkhtml-v42-prod in /home/p24-server-scripts/v4/v42/ Issue: #3218

Same root cause family (shared compose project, one service targeted, whole project reconciled) but a different trigger: an undefined environment variable in the top-level networks: block, not a container_name mismatch.

What happened

1. docker-compose.yml in /home/p24-server-scripts/v4/v42/ defines three services in one project: backend (→ v42-prod), s3 (→ s3-v42-prod), wkhtml (→ wkhtml-v42-prod).
2. An unrelated fix (adding restart: unless-stopped to wkhtml, for #3193) also added test-net to the top-level networks: block — this touches the whole project’s config, not just wkhtml.
3. The fix command was CONTAINER_NAME=v42-prod IMAGE_NAME=v42-prod docker-compose up -d wkhtml — scoped to wkhtml by name, but Compose still evaluates and reconciles the entire project against the new top-level config before touching any single service.
4. backend/s3 reference networks: [${CLUSTER_NETWORK}] — this var was never set anywhere (no .env, no shell export). Compose resolved it to an empty string, decided backend/s3 needed recreating against the (invalid) new network topology, stopped + removed both containers, and then failed to recreate them — leaving v42-prod and s3-v42-prod fully deleted (not “Exited”: gone).
5. nginx-proxy had no live backend → 503 on every /api/* endpoint → cascading CORS errors on the frontend. restart: unless-stopped did not help — that policy recovers a crashed process, not a container that Compose itself removed.

Recovery

Both containers were recreated from the separate, correct isolated directory (/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts, the actual CI/CD build path — see pinbox24-w3-w4-outage-diagnosis.md Fix B) using cached local images (*-prod:backup-20260707), not the broken shared directory. No ECR pull was required.

Fix applied

Added /home/p24-server-scripts/v4/v42/.env with CLUSTER_NETWORK=prod-v-4-net so any future docker-compose invocation from that directory (read-only or not) resolves the network correctly instead of silently defaulting to an empty string.

Verified 2026-07-08 (read-only, no container touched):

• .env did not previously exist in /home/p24-server-scripts/v4/v42/ — created (not overwritten)
• docker-compose config / docker-compose ps from that directory now resolve prod-v-4-net cleanly — the “undefined network” warning is gone
• v42-prod and s3-v42-prod confirmed untouched throughout — StartedAt timestamp identical before and after the .env write (2026-07-07T23:48:41.28...Z)

Scope check: the same ${CLUSTER_NETWORK} pattern (referenced with no confirmed .env providing a value) also exists in v2/v21, v2/v22, v3/v31, v3/v32, v3/v3reso, v3/v3socket, v4/v41, and cron/ on bms-1 — tracked for audit + fix in #3222 (not fixed yet, scoping only).

Audit results — Issue #3222 (2026-07-08)

Audited all 8 directories flagged in the scope check above, using the same read-only check as the v42 fix: docker-compose config 2>&1 | grep -iE "undefined|WARN" run standalone from each directory (no launcher script, no exported env vars — the worst case for an ad hoc/manual invocation).

Result: all 8 directories were already OK — no .env fix needed, nothing was modified.

Directory	Check result	Top-level networks: (bms-1, hardcoded in the compose file)	.env action	Live container(s) confirmed untouched
v2/v21	ALREADY_OK	prod-v-2-net	none	yes (read-only checks only)
v2/v22	ALREADY_OK	prod-v-2-net	none	yes
v3/v31	ALREADY_OK	prod-v-3-net	none	v31-prod — Up 2 days at audit time
v3/v32	ALREADY_OK	prod-v-3-net	none	v32-prod, s3-v32-prod — running (recent uptime from an unrelated scheduled redeploy; only docker-compose config/docker inspect/docker ps were run against this directory during the audit)
v3/v3reso	ALREADY_OK	prod-v-3-net	none	v32-prod-reso, s3-v32-prod-reso, cron-v32-prod-reso — running
v3/v3socket	ALREADY_OK	prod-v-3-net	none	v32-prod-socket, s3-v32-prod-socket, cron-v32-prod-socket — running
v4/v41	ALREADY_OK	prod-v-4-net	none	v41-prod — Up 2 days
cron	ALREADY_OK	test-net (its cron-job.env also already sets CLUSTER_NETWORK=test-net, correctly matching)	none	p24-cron-job

No directory had an existing .env. Three of them (v3/v32, v3/v3reso, v3/v3socket) have a cron-job.env with a CLUSTER_NETWORK= key present but empty — this file is not auto-loaded by docker-compose (only a literal .env in the project directory is), so it has no effect on compose’s own variable resolution and was left untouched.

Why these 8 didn’t need the same fix as v42, even though the same bare $CLUSTER_NETWORK reference exists in each compose file’s per-service networks: list: v42’s specific failure was that the top-level networks: block itself referenced ${CLUSTER_NETWORK} — an undefined value there corrupts the whole project’s network topology and triggers Compose to reconcile (and delete) every service in the project. In all 8 audited directories, the top-level networks: block is hardcoded to a real external network name (prod-v-2-net / prod-v-3-net / prod-v-4-net / test-net), so that specific project-wide trigger is not present. Only the per-service network-attachment list references the (unset, standalone) variable, which is a narrower, lower-severity gap — and empirically docker-compose config (v1.25.3, the version installed on bms-1) does not emit an undefined/WARN line for it when run standalone.

Residual recommendation (not auto-fixed, needs no immediate action): an ad hoc docker-compose up or restart invoked directly from one of these 8 directories — bypassing the launcher scripts below — would still see an empty per-service network attachment. Lower severity than the v42 project-wide reconciliation bug, but the standing prevention rule still applies: always deploy via the launcher scripts or docker-deploy-prod.sh, never call docker-compose up/restart bare from these directories.

Launcher scripts (/home/p24-server-scripts/start-*.sh) — already set CLUSTER_NETWORK themselves:

Script	CLUSTER_NETWORK set inline	Directories it drives
start-v3.sh	"prod-v-3-net"	v3/v31 (frontend), v3/v32 (backend)
start-v4.sh	"prod-v-4-net"	v4/v41 (frontend), v4/v42 (backend)
start-reso.sh	"prod-v-3-net"	v3/v3reso (backend)
start-socket.sh	"prod-v-3-net"	v3/v3socket (backend)

Each script exports CLUSTER_NETWORK as an inline prefix immediately before invoking docker-compose (e.g. CLUSTER_NETWORK=$CLUSTER_NETWORK CONTAINER_NAME=$B_CONTAINER_NAME docker-compose -f ... up -d), scoped to that single command’s environment. This matches the per-directory hardcoded top-level network values above and is redundant-but-safe belt-and-suspenders with the .env approach used for v42 — not conflicting. v2/v21/v2/v22’s docker-deploy-prod.sh also accepts CLUSTER_NETWORK as a positional argument with a fallback default, same pattern.

Generalized lesson

Any undefined variable referenced in a shared docker-compose project is a latent trigger for this bug — not just container_name. Before running any docker-compose command (including read-only ones like ps or config) in a shared multi-service directory:

docker-compose config 2>&1 | grep -i "undefined\|WARN"

If this reports anything, stop — do not run up, restart, or down until every referenced env var resolves to a real value. A missing var does not fail loudly; Compose silently treats it as empty and proceeds to reconcile (and potentially delete) unrelated containers in the same project.

---

Case Study 2026-08-06 — v42-prod clobbered from the GitLab-CI build dir itself (#5792)

Services: v42-prod, s3-v42-prod in /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ Issue: #5792

The 2026-07-08 case study above treated /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ (the actual GitLab-CI build path) as the safe, correctly-isolated recovery target — as opposed to the shared /home/p24-server-scripts/v4/v42/ directory that caused that incident. This case shows the CI build dir itself is not environment-isolated either: it is reused across both development-branch prod pipelines and staging pipeline runs for p24-back-ts, so it is exactly the “shared build directory” this playbook’s root-cause section describes — just one level further upstream than previously documented.

What happened

1. docker-compose.yml’s backend service in this directory uses the same dynamic pattern as the original root cause: container_name: $CONTAINER_NAME, image: $IMAGE_NAME.
2. At 2026-08-06 17:23:09 UTC, a deploy — since confirmed as the development-branch stage-back-end-deploy job of pipeline 2738147490 (see Root cause CONFIRMED below) — executed docker-compose from this same directory with CONTAINER_NAME=v42-stage (and the sibling s3/wkhtml services similarly targeted -stage names).
3. Compose reconciled the whole project against the new container names: stopped + removed v42-prod and s3-v42-prod entirely (not “Exited” — gone from docker ps -a), then created v42-stage, s3-v42-stage, wkhtml-v42-stage in their place.
4. nginx-proxy had no backend for api.w4.pinbox24.com → served its self-signed letsencrypt-nginx-proxy-companion placeholder cert (top-level cert symlinks are only written on an actual issuance/renewal event — see bms1-nginx-proxy-missing-cert-symlinks.md) + 503 on every /api/* request. Same cascading symptom as the 2026-07-08 case.
5. Discovered when a user reported “W4 backend is down”; root-caused via docs/playbooks/pinbox24-w3-w4-health-verification.md §2.1–2.2 (HTTP 503 with cert validation bypassed + docker ps -a showing the container missing, not stopped).

Recovery

Scoped, additive recreate from the same directory, using the cached local image the prod pipeline had just built 29 minutes before the clobber (v42-prod:latest, built 16:54 UTC — safe to assume it was the intended, currently-live prod image):

# PLAYBOOK: pinbox24-docker-compose-staging-prod-collision.md
cd /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/
CLUSTER_NETWORK=prod-v-4-net CONTAINER_NAME=v42-prod \
  IMAGE_NAME=563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod:latest \
  docker-compose up -d --no-deps backend

--no-deps + naming only the backend service kept the -stage siblings (s3-v42-stage, wkhtml-v42-stage) untouched — confirmed via docker ps -a before/after. v42-prod came up with PM2 showing 3/3 instances online, 0 restarts. The missing cert symlinks self-healed within ~1 minute of the container starting (docker-gen picked up the new container event and triggered nginx-proxy-letsencrypt to write them) — no manual fix from bms1-nginx-proxy-missing-cert-symlinks.md was needed this time.

Root cause CONFIRMED (GitLab CI job traces — added post-recovery, #5792)

The recovery above, and the “almost certainly a staging pipeline run” wording in What happened, were written before the trigger was confirmed. A follow-up read of the GitLab CI job traces (job logs only — checked for and free of secret values before reading) for both repos confirmed the exact mechanism:

Both p24-back-ts (W4) and p24-v-3.2 (W3) run the identical pipeline shape on every push — back-end-build → docker-image-build → <env>-back-end-deploy → test-after-build. The deploy job name differs by branch, but both branches call the same generically-named script, docker-deploy-prod.sh (its log even prints “Starting deployment with the Production environment” for a stage deploy), against the same shared Docker Compose project directory — differing only in the CONTAINER_NAME/IMAGE_NAME args each job passes:

• master push → job prod-back-end-deploy → docker-deploy-prod.sh … v42-prod …
• development push → job stage-back-end-deploy → docker-deploy-prod.sh … v42-stage …

Confirmed W4 timeline (from GitLab job traces):

Time (UTC)	Event
16:50:10–16:55:50	master pipeline 2738084716 → prod-back-end-deploy succeeds → v42-prod freshly (re)deployed, healthy
17:18:56–17:23:15	development pipeline 2738147490 → stage-back-end-deploy succeeds → repoints the same shared compose project to v42-stage; Compose reconciliation silently tears down v42-prod/s3-v42-prod
17:23:19–17:24:21	Same pipeline’s test-after-build fails — stage smoke test curl https://v42-stage-test.dev.eat.pl/… → Connection refused (separate, pre-existing stage-environment issue, not the prod-outage cause)
—	No later master push arrived to re-run prod-back-end-deploy, so v42-prod stayed deleted until the manual recovery above

Root cause, one sentence: the shared GitLab-CI build directory (and the Compose project inside it) is used by both the master-branch prod-deploy job and the development-branch stage-deploy job, so any development push landing after a master push silently deletes whatever prod deployment just happened — no safeguard, no automatic recovery. This is a design flaw in Pinbox24’s own .gitlab-ci.yml / deploy-script setup (repo-owned by the Pinbox24 team, not p24-infra) and is exactly what Option C in the Escalation section above prescribes against — never implemented for either W3 or W4.

Why prod stayed down for hours specifically on 2026-08-06: both repos show a wall of failing pipelines across 15:00–19:00+ UTC on both branches — many consecutive development pushes (each re-clobbering prod via stage-back-end-deploy) with no intervening successful master push to restore it. The test-after-build failures are a separate, likely pre-existing stage smoke-test connectivity issue, not the trigger for the prod outage; they merely co-occur as later stages of the same pipelines.

Companion finding — v32-prod (W3) was independently down too, confirmed same root-cause family

While verifying W4, v32-prod (W3’s main backend) was found completely absent from prod-v-3-net as well (confirmed via docker network inspect prod-v-3-net — not an alias trap like s3-v32-prod-renamed, genuinely gone; only v32-prod-reso/v32-prod-socket were up). Recovered the same way regardless (plain docker-compose up -d --no-deps backend from that directory, cached v32-prod:latest built 17:48 UTC) — PM2 1/1 online, 0 restarts, cert symlinks self-healed the same way.

Initial recovery-time hypothesis (since superseded): because W3’s backend service in /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/docker-compose.yml has a hardcoded container_name: v32-prod and image: (no $CONTAINER_NAME/$IMAGE_NAME substitution), the exact container_name-swap mechanism the top of this playbook documents could not have applied, so this was first recorded as a different, unknown-trigger failure.

Confirmed (GitLab job trace 15753387347): W3 is in fact the same shared-script / shared-build-dir root-cause family as W4, not a separate mechanism. p24-v-3.2’s stage-back-end-deploy job explicitly invokes ./docker-deploy-prod.sh v32-stage v32-stage v32-stage.dev.eat.pl v31-stage.dev.eat.pl — the same generically-named prod script, run from the same shared Compose project directory, with the same silent-teardown effect on v32-prod. The hardcoded container_name only changes the precise teardown step (it is not a name-swap as in W4); the trigger — a development-branch stage deploy running the shared prod script against the shared build dir — is identical to W4’s.

Status / follow-ups

• ✅ Which pipeline clobbered W4 — identified. development pipeline 2738147490’s stage-back-end-deploy job at 17:23 UTC (see the confirmed timeline above).

• ✅ v32-prod (W3) disappearance — root-caused. Same shared-script / shared-build-dir family as W4 (job trace 15753387347), not a separate mechanism (see the companion finding above).

• ✅ Permanent-fix MRs opened (not yet merged, pending review):

  • W4 — pinbox24/p24-back-ts MR !810: https://gitlab.com/pinbox24/p24-back-ts/-/merge_requests/810
  • W3 — pinbox24/p24-v-3.2 MR !80: https://gitlab.com/pinbox24/p24-v-3.2/-/merge_requests/80

  Both add COMPOSE_PROJECT_NAME isolation per environment — the actual collision fix: a distinct Compose project per env so a development/stage deploy can no longer reconcile (and tear down) the prod project — and strip hardcoded credentials from docker-deploy-stage.sh (mirrors the SECURITY FIX already applied to each repo’s docker-deploy-prod.sh, per #5812). Syntax/YAML-validated. GitLab merge on pinbox24/* is autonomous per docs/w3-w4-stack-operations.md §2 but was held pending review given the expanded scope (2 repos, 2 collision mechanisms, credential removal).

• ⬜ Still open: audit whether the same shared-CI-dir pattern threatens p24-ms-s3transport (W3’s separate old-s3 repo) or other Pinbox24 GitLab-CI build directories.

• ⬜ Separate, lower severity: fix the stage smoke-test target (v42-stage-test.dev.eat.pl / v32-stage.dev.eat.pl connection refused) — pre-existing stage-environment issue, worth its own ticket.

---

Related Playbooks

• pinbox24-bms1-manual-deploy.md — full manual deploy procedure for bms-1 services
• pinbox24-mailgun-deploy-script-bug.md — mailgun deploy script argument mismatch bug
• pinbox24-s3-wasabi-bms1.md — s3-v2 service deploy and Wasabi configuration on bms-1
• pinbox24-w3-w4-outage-diagnosis.md — Fix B: isolated build-directory recovery path used in the 2026-07-08 case study above
• pinbox24-w3-w4-health-verification.md — the checklist used to root-cause the 2026-08-06 incidents
• bms1-nginx-proxy-missing-cert-symlinks.md — the cascading TLS symptom both incidents produced (self-healed here once the backend container came back, but won’t always)