p24-infra Docs

w3-v32-prod-clobbered-by-stage-deploy-recovery

8 min read

W3 v32-prod clobbered by a stage deploy — recovery

Status: STABLE — executed and verified 2026-07-29 17:26–17:29 UTC on bms-1 (incident #4565)

What triggers this

api.w3.pinbox24.com returns HTTP 503 with the self-signed letsencrypt-nginx-proxy-companion placeholder cert (blackbox EndpointDown, probe_ssl_earliest_cert_expiry absent), while every other W3/W4 vhost is healthy (w3.pinbox24.com on v31-prod, socket.w3 on v32-prod-socket, w3.reso-integration-… on v32-prod-reso — all 200 with valid LE certs).

Root cause (incident #4565): the v32-prod main-backend container (VIRTUAL_HOST=api.w3.pinbox24.com, port 3000) is completely absent — docker inspect v32-prod → container-not-found, and no server block for api.w3.pinbox24.com remains in nginx-proxy’s generated config, so the vhost falls through to the proxy default (503 + placeholder cert). This is the “staging deploy killed prod” gotcha (pinbox24-bms1-manual-deploy.md §Gotchas): the tracked docker-compose.yml in the live build dir /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2 was overwritten with a stage definition (backend → container_name: v32-stage, image .../v32-stage) and backend-environment.env was rewritten with staging credentials. A docker-compose reconcile of the backend service removed the prod v32-prod container. The still-running v32-prod-reso / v32-prod-socket siblings (created 7 days earlier) survived because they are no longer named in the stage compose (no --remove-orphans).

Pre-conditions

  • Root SSH access to bms-1 (94.23.26.113) — see docs/playbooks/server-ssh-connectivity.md
  • At least one sibling container (v32-prod-reso or v32-prod-socket) is confirmed currently running and serving traffic — the whole recovery clones its env and mounts from whichever sibling is healthy. If both siblings are also down, this playbook does not apply; escalate instead of improvising a from-scratch recreate.
  • A checkout of this repo with the current infra-src/pinbox24/w3/persistent-patches/*.js is reachable (bms-4 or your local dev checkout) — needed for the patch-restore step, independent of whether the build-dir copies survived.
  • Confirmed this is genuinely the “stage clobbered prod” failure mode (503 + self-signed placeholder cert, docker inspect v32-prod → container-not-found) and not a different W3 outage class — see docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md for general triage first if unsure.

Why you must NOT just docker-compose up from the build dir

  • The build-dir docker-compose.yml now defines stage (v32-stage), not v32-prod.
  • backend-environment.env on disk now holds stage creds (verify: compare each key against the running v32-prod-reso env — in #4565, 14 keys DIFFered incl. NODE_ENV, MONGODB_URL, PMONGODB_URL, REDIS_HOST, frontendUrl). Using it would point prod at stage MongoDB/Redis.

The safe source of prod config is the known-healthy running sibling v32-prod-reso, whose env was baked at create time from the prod env and is currently serving traffic fine.

Recovery — reclone v32-prod from the v32-prod-reso sibling

v32-prod, -reso, -socket run the identical image + pm2-runtime start ecosystem.config.js (same ./app-backend/app.js), same networks (test-net+prod-v-3-net), same restart policy (unless-stopped), same 6 persistent-patch mounts + /var/log. They differ ONLY in VIRTUAL_HOST/LETSENCRYPT_HOST/CONTAINER_NAME. So v32-prod = the sibling with api.w3 vhost.

  1. Restore persistent-patches/*.js in the build dir from their git source — do NOT assume the build-dir copies survived. The same CI git clean that overwrote docker-compose.yml/ backend-environment.env also deletes this directory (that’s exactly what happened in #4565 — see Execution record below). The canonical, git-tracked source is infra-src/pinbox24/w3/persistent-patches/*.js in this repo (committed docs/pinbox24/w3-persistent-patches-committed-2026-07-08.md), not the build dir or a sibling container’s live filesystem — reuse the exact restore step secrets-sync.yml’s sync-pinbox24-w3 job already runs on every secrets sync:

    ssh root@94.23.26.113 "mkdir -p /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/persistent-patches"
scp infra-src/pinbox24/w3/persistent-patches/*.js \
  root@94.23.26.113:/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/persistent-patches/

    Run this from a checkout that has the current infra-src/pinbox24/w3/persistent-patches/ (the p24-infra repo on bms-4, or your local dev checkout) — never from the bms-1 build dir itself, since that’s the thing being restored.

    Only the 5 persistent-patches/*.js files need this restore. The 6th mount source in the docker run below — <build>/app-backend/.../excel_import.js — is not a p24-infra-managed patch: it is a stock, git-tracked file in pinbox24/p24-v-3.2 (verified #4708). A CI git clean removes only untracked files (the 5 patches), so excel_import.js survives, and a fresh checkout restores it regardless; its content is also byte-identical to the copy baked into the v32-prod image, so the mount is a redundant no-op. Do not try to restore it from git in p24-infra — there is no p24-infra copy, and there does not need to be. If the whole build dir is wiped, the GitLab checkout that repopulates it brings excel_import.js back automatically.

    Fallback, only if the git checkout is unavailable and a healthy sibling (v32-prod-reso/v32-prod-socket) is confirmed running: extract the 5 persistent-patch files from the sibling’s live filesystem instead (docker exec v32-prod-reso cat <path> > <build>/persistent-patches/<file> for each) — this is what the 2026-07-29 execution actually did, before this restore step existed in this playbook (see Execution record). (excel_import.js needs no such fallback — it comes back with the checkout, or can be copied from the sibling only if the build dir’s own tracked copy is somehow missing.)

  2. Dump the sibling’s env to a root-only temp file (never printed), strip the per-container keys:

    TMPENV=$(mktemp)
docker inspect v32-prod-reso --format '{{range .Config.Env}}{{println .}}{{end}}' \
  | grep -vE '^(VIRTUAL_HOST|LETSENCRYPT_HOST|CONTAINER_NAME|HOSTNAME)=' > "$TMPENV"

  3. Recreate v32-prod, overriding the vhost keys, cloning mounts from the sibling:

    docker run -d --name v32-prod --restart unless-stopped --network test-net \
  --env-file "$TMPENV" \
  -e VIRTUAL_HOST=api.w3.pinbox24.com -e LETSENCRYPT_HOST=api.w3.pinbox24.com \
  -e CONTAINER_NAME=v32-prod \
  -v /var/log:/var/log \
  -v <build>/persistent-patches/crudService.js:/app/app-backend/helperFunction/crudService.js:ro \
  -v <build>/persistent-patches/twilioService.js:/app/app-backend/services/twilioService.js:ro \
  -v <build>/persistent-patches/mongoose.js:/app/app-backend/config/mongoose.js:ro \
  -v <build>/persistent-patches/filesUpload.js:/app/app-backend/controllers/filesUpload.js:ro \
  -v <build>/persistent-patches/pushNotification.helper.js:/app/app-backend/helperFunction/pushNotification.helper.js:ro \
  -v <build>/app-backend/helperFunction/processHelper/tasksService/tasks/excel_import.js:/app/app-backend/helperFunction/processHelper/tasksService/tasks/excel_import.js:ro \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod
docker network connect prod-v-3-net v32-prod
rm -f "$TMPENV"

    (<build> = /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2.)

Verification

docker inspect v32-prod --format '{{.State.Status}} restarts={{.RestartCount}}'   # running, low restarts
curl -sS -o /dev/null -w '%{http_code} verify=%{ssl_verify_result}\n' https://api.w3.pinbox24.com/api/i18n/langs
# expect: 200 verify=0  (LE cert, not self-signed)

nginx-proxy re-generates the api.w3 server block automatically once the container is up; the LE companion re-issues/attaches the real cert within a minute (placeholder disappears).

Rollback

There is no prior-good v32-prod to roll back TO — by the time this playbook applies, the original container is already gone (that’s the trigger). “Rollback” here means backing out a failed recreate attempt without making things worse:

  1. If the newly-created v32-prod is unhealthy or crash-looping (check docker logs v32-prod, docker inspect v32-prod --format '{{.RestartCount}}'): 
    docker stop v32-prod && docker rm v32-prod
    This returns the vhost to its pre-recovery state — still 503 + placeholder cert, same as before you started. Not worse, just not yet fixed.
  2. Never stop or restart v32-prod-reso / v32-prod-socket while diagnosing a failed recreate — they are the source of truth this whole recovery depends on, and the only reason w3.reso-integration-… / socket.w3 traffic stays healthy throughout the incident.
  3. Common cause of a failed recreate: a stale/incomplete $TMPENV file (partial docker inspect output) or a persistent-patches/*.js restore that didn’t actually land before the docker run (verify with ls -1 <build>/persistent-patches/*.js before recreating again — see the restore step above).
  4. If a second recreate attempt also fails, stop and escalate per playbook-lifecycle.md Situation 4 rather than trying further variations blind — something about this incident differs from #4565.

Durability / follow-up (human / GitLab)

The root cause is upstream: the stage deploy must not run against the prod build dir, and the committed docker-compose.yml for p24-v-3.2 must define v32-prod (prod), not v32-stage. This hand-recloned container will be dropped again on the next CI redeploy of this dir. File a follow-up so a human fixes the deploy pipeline / build-dir isolation (see pinbox24-bms1-manual-deploy.md §Gotchas, w3-w4-redeploy-idempotency.md).

Execution record — 2026-07-29 (incident #4565, executed by infra-task worker bms4-cw-1)

  • Symptom: api.w3.pinbox24.com 503 + self-signed placeholder cert since 16:50 UTC; every other W3/W4 vhost healthy. docker inspect v32-prod → container-not-found; no api.w3 server block in nginx-proxy config.
  • Root cause: build-dir docker-compose.yml overwritten with a stage def at 16:23 (backend→v32-stage), backend-environment.env rewritten with stage creds at 16:50 (14 keys DIFFered vs the running prod sibling, incl. NODE_ENV/MONGODB_URL/PMONGODB_URL/REDIS_HOST). A compose reconcile removed the prod v32-prod. CI git clean had also deleted the host-side persistent-patches/*.js (their inodes survived only inside the still-running siblings).
  • Fix executed: extracted the 6 persistent-patch code files from the healthy v32-prod-reso container via docker exec cat into /root/v32-prod-recovery/patches/, then mounted from that path into the recreated container; recreated v32-prod cloning the sibling’s prod env (--env-file from docker inspect, per-container keys stripped) with VIRTUAL_HOST/LETSENCRYPT_HOST=api.w3.pinbox24.com, both networks, restart=unless-stopped. Recipe updated 2026-07-30: the Recovery section above now sources patches from the git-tracked infra-src/pinbox24/w3/persistent-patches/*.js via scp directly into the build-dir persistent-patches/ path (matching the docker run mount sources exactly, reusing secrets-sync.yml’s existing restore step) instead of extracting from a sibling container — more resilient if the siblings themselves are ever unavailable, and it’s the same source of truth secrets-sync.yml already keeps in sync. The sibling-extraction method used on 2026-07-29 is kept as an explicit fallback, not the primary path.
  • Verified: backend pm2 app online (0 restarts), “Mongoose connected” + “Redis is ready” (prod infra), nginx-proxy regenerated the api.w3 vhost, LE companion re-attached the real cert (CN=api.w3.pinbox24.com, LE YR1, valid→2026-09-30). curl https://api.w3.pinbox24.com/ and /api/i18n/langs → 200 verify=0. Cleared #4565 (blackbox_ssl) and #4564 (http_2xx) together.
  • NOT done (human/GitLab follow-up): repair the deploy pipeline so the stage deploy stops targeting the prod build dir and the committed p24-v-3.2/docker-compose.yml defines v32-prod. This hand-recloned container is NOT idempotent — the next CI redeploy of that dir will drop it again.
  • docs/playbooks/pinbox24-w3-w4-outage-diagnosis.md — general W3/W4 backend outage triage
  • docs/playbooks/w3-redis-v32-missing-container.md — sibling “missing container” recovery (redis-v32)
  • docs/playbooks/pinbox24-bms1-manual-deploy.md — staging-kills-prod gotcha + manual deploy
  • docs/w3-w4-stack-operations.md — ownership + permission matrix

Graph View