p24-infra Docs

v32-prod-prod-v-3-net-persistence

2 min read

Playbook: v32-prod drops off prod-v-3-net on recreate

Service: v32-prod (W3 backend, api.w3.pinbox24.com) on bms-1 (94.23.26.113) Last updated: 2026-07-11 · Issue: radieu/p24-infra#2826

Trigger / symptoms

  • After a docker compose up -d or container recreate on bms-1, v32-prod is on test-net only.
  • Intended topology (matching siblings v32-prod-reso, v32-prod-socket) is prod-v-3-net + test-net.
  • api.w3.pinbox24.com may still return 200 because nginx-proxy is also attached to test-net, so routing can survive via that path — but the documented topology is wrong and is fragile.

Root cause

The live deploy uses the on-server working-tree compose file, not GitLab’s committed version:

/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/docker-compose.yml

The committed GitLab pinbox24/p24-v-3.2 master compose (ae2981a) is stale — it has no reso/socket/redis and no prod-v-3-net. The real topology lives as uncommitted local modifications to that file. If the backend (v32-prod) service block omits prod-v-3-net while the sibling blocks list it, every recreate brings v32-prod up on test-net only.

Fix (durable)

Add prod-v-3-net to the backend service networks: in the on-server compose file (both networks are already declared external):

  backend:
    container_name: v32-prod
    ...
    networks:
      - test-net
      - prod-v-3-net      # parity with reso/socket; persists across recreate

Then apply immediately with no downtime:

# PLAYBOOK: v32-prod-prod-v-3-net-persistence.md
ssh -i ~/.ssh/vps_root_key root@94.23.26.113 'docker network connect prod-v-3-net v32-prod'

Verify:

docker inspect v32-prod --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
# expect: prod-v-3-net test-net
curl -s -o /dev/null -w '%{http_code}\n' https://api.w3.pinbox24.com   # expect 200

⚠️ Do NOT

  • Never run bare docker-compose config on this dir in an agent session — it resolves env_file (backend-environment.env) and prints all W3 prod credential values to stdout (see incident #3750). Inspect with grep -nE 'networks|test-net|prod-v-3-net' docker-compose.yml or yq instead.

Known limitation / follow-up

The committed GitLab compose diverges from the live working tree. If a CI run ever hard-resets the working tree (git checkout -f), the entire live topology — not just this network line — is lost. Reconciling the committed compose with the deployed reality is a separate task requiring a GitLab MR.

Graph View

Backlinks