pinbox24-w3-w4-outage-diagnosis

Playbook: Pinbox24 w3/w4 Backend Outage Diagnosis & Fix

Created: 2026-06-27
Incident: w4.pinbox24.com + w3.pinbox24.com returning 502/504 on all API endpoints
Issue: #1716

See also: pinbox24-w3-w4-health-verification.md — run the quick health checklist there first. Come here once you’ve confirmed something is actually broken and need the diagnosis flowchart / fixes.

---

Container map (quick reference)

Domain	Backend container	Network	DB connection
w4.pinbox24.com	v42-prod on bms-1	test-net	NEW_MONGODB_URI → bms-2/bms-3 rs0 (w4_db)
w3.pinbox24.com	v32-prod on bms-1	prod-v-3-net	v3MongoUrl → artnet.pl rs0 (pinbox24-production)
API for both	v42-prod (api.w4.*, api.w3.*)	test-net	same as above
Notifications	v42-notify-prod	prod-v-4-net, test-net	—

nginx-proxy on bms-1 routes traffic. Must share a Docker network with each container.

---

Trigger symptoms

• Browser shows untranslated keys (auth.login, auth.select_language)
• Network tab shows /api/i18n/langs as (pending) then 504 Gateway Timeout
• Login POST also hangs → CORS errors cascade from 504s
• nginx-proxy returns 502 (can’t reach container) or 504 (container reached but hangs)

---

Diagnosis flowchart

Step 1 — Is it nginx-proxy (502) or app-level (504)?

curl -o /dev/null -sw "%{http_code}" https://api.w4.pinbox24.com/api/i18n/langs

• 502 → nginx can’t reach the container → go to Step 2
• 504 → nginx reached the container but it timed out → go to Step 3
• 2xx/4xx → backend is up; check app-level logic

Step 2 — nginx-proxy network split (502 diagnosis)

ssh root@94.23.26.113
docker exec nginx-proxy cat /etc/nginx/conf.d/default.conf | grep -B5 "Cannot connect" | grep "^# " | grep -v "Cannot" | sort -u

If any virtual hosts listed → containers are on wrong Docker networks.

Fix:

# Check which network each affected container is on
docker inspect v42-prod --format '{{range $k,$v := .NetworkSettings.Networks}}{{println $k}}{{end}}'
 
# Connect to the correct user-defined network
docker network connect prod-v-4-net v41-prod   # v4 containers
docker network connect prod-v-3-net v31-prod   # v3 containers
 
# Restart nginx-proxy to regenerate config
docker restart nginx-proxy

Step 3 — App hanging (504 diagnosis)

PM2 logs are NOT in docker logs (PM2 captures them). Read from:

ssh root@94.23.26.113
docker exec v42-prod pm2 list   # check status + uptime
docker exec v42-prod tail -50 /var/log/v42-prod/pm2/pm2_v42-prod_production_out.log
docker exec v42-prod tail -30 /var/log/v42-prod/pm2/pm2_v42-prod_production_err.log

Look for these specific errors:

3a. RabbitMQ ETIMEDOUT

Error connecting to RabbitMQ: Error: connect ETIMEDOUT 54.36.123.110:5672

→ See Fix A: Restart RabbitMQ

3b. Mongoose disconnected immediately on startup

Mongoose disconnected - YYYY-MM-DD HH:MM | PID: NNN

With no preceding Mongoose connected → MongoDB auth failure. → See Fix B: MongoDB password mismatch

3c. Redis reconnecting every ~5 minutes

Redis is connected with host - kr40258-001.dbaas.ovh.net | PID: NNN
Redis is ready

Appearing repeatedly → OVH DBaaS closes idle connections. Requests that hit Redis during a reconnect gap will hang. Restart PM2 to clear queued commands:

docker exec v42-prod pm2 restart all

---

Fix A: Restart RabbitMQ on bms-4

RabbitMQ (54.36.123.110:5672) is required by v42-prod for async queue processing. Without it, ALL request handlers hang on startup.

Check if running:

ssh root@54.36.123.110 'docker ps | grep rabbit; ss -tlnp | grep 5672'

Start if missing (credentials from v42-prod env — see RABBIT_MQ_URL):

# On bms-4
docker run -d \
  --name rabbitmq \
  --restart unless-stopped \
  -p 5672:5672 \
  -p 15672:15672 \
  -e RABBITMQ_DEFAULT_USER=pinbox24 \
  -e RABBITMQ_DEFAULT_PASS=<value from RABBIT_MQ_URL in v42-prod> \
  rabbitmq:3-management
 
# Verify
docker exec rabbitmq rabbitmqctl list_users

After RabbitMQ is up, restart PM2 (app won’t auto-reconnect from ETIMEDOUT state):

ssh root@94.23.26.113 'docker exec v42-prod pm2 restart all'

Confirm in logs:

tail /var/log/v42-prod/pm2/pm2_v42-prod_production_out.log | grep -i rabbit
# Expect: Connected to RabbitMQ / Queue googleSheetUpdate is ready

---

Fix B: MongoDB password mismatch — v42-prod (NEW_MONGODB_URI)

v42-prod connects to bms-2/bms-3 rs0 via NEW_MONGODB_URI. When the rs0 admin password is rotated (SOPS), the Docker container env var is NOT automatically updated — it’s baked in at docker run time.

Symptom: Mongoose disconnected immediately on every PM2 restart, no Mongoose connected.

Diagnose:

# Check if the URI user exists in rs0 admin db
# (requires SOPS admin password from secrets/bms-servers.env.sops)
mongosh 'mongodb://admin:<pass>@145.239.133.104:27017/admin' \
  --quiet --eval 'db.getUsers().users.forEach(function(u){print(u.user)})'

Temporary fix (survives PM2 restart, NOT Docker restart):

On local Windows dev machine:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:NEW_MG_PASS = (sops --decrypt --input-type dotenv --output-type dotenv secrets\bms-servers.env.sops `
  | Select-String "^mongodb_rs0_admin_password=").ToString().Split("=",2)[1]
 
# URL-encode the password (handles @, !, =, # etc.)
$encoded = [System.Uri]::EscapeDataString($env:NEW_MG_PASS)
$newUri = "NEW_MONGODB_URI=mongodb://admin:${encoded}@145.239.133.104,51.68.155.224/w4_db?replicaSet=rs0&authSource=admin"
 
ssh root@94.23.26.113 "docker exec -e '$newUri' v42-prod pm2 restart all --update-env"
$env:NEW_MG_PASS = ""; $encoded = ""; $newUri = ""

Permanent fix (recreate container via docker-compose):

v42-prod uses docker-compose project p24-back-ts. Update backend-environment.env on bms-1, then recreate:

# 1. Use a piped Python script to update backend-environment.env on bms-1
#    (same pattern as Fix B temp-fix above — extract $encoded, build Python script, pipe via SSH stdin)
 
# 2. Recreate the container
ssh root@94.23.26.113 @'
cd /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts
CONTAINER_NAME=v42-prod IMAGE_NAME=563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod docker-compose up -d --no-build backend
'@

Key paths for v42-prod: build dir /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/, env key NEW_MONGODB_URI, image 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod.

Always URL-encode the MongoDB password. Characters like @, !, =, # break URI parsing. Use [System.Uri]::EscapeDataString() in PowerShell or urllib.parse.quote() in Python.

---

Fix C: MongoDB password mismatch — v32-prod (TWO connections)

Discovery from 2026-06-27 incident: v32-prod has two independent MongoDB connections, both of which must be updated simultaneously. Updating only one still causes 422/504 errors.

v32-prod connection map

Env var	Used by	Code path
PMONGODB_URL	Mongoose	/app/app-backend/config/mongoose.js → props.PMONGODB_URL
MONGODB_URL	mongojs	/app/app-backend/config/mongo_db.js → props.MONGODB_URL

Both are set from /app/app-backend/config/env.config.js → process.env.PMONGODB_URL / process.env.MONGODB_URL.

Symptom: /api/i18n/langs returns 422 (mongojs connection fails) or Mongoose disconnected in PM2 logs. Even if one connection appears to work, the other failing causes cascading 422s on all endpoints.

Temporary fix (both vars at once, survives PM2 restart, NOT Docker restart):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:P32_PASS = (sops --decrypt --input-type dotenv --output-type dotenv secrets\bms-servers.env.sops `
  | Select-String "^mongodb_rs0_admin_password=").ToString().Split("=",2)[1]
 
$encoded = [System.Uri]::EscapeDataString($env:P32_PASS)
$pmoUri = "PMONGODB_URL=mongodb://admin:${encoded}@145.239.133.104,51.68.155.224/w4_db?replicaSet=rs0&authSource=admin"
$moUri  = "MONGODB_URL=mongodb://admin:${encoded}@145.239.133.104,51.68.155.224/w4_db?replicaSet=rs0&authSource=admin"
 
ssh root@94.23.26.113 "docker exec -e '$pmoUri' -e '$moUri' v32-prod pm2 restart all --update-env"
$env:P32_PASS = ""; $encoded = ""; $pmoUri = ""; $moUri = ""

Verify v32 is up:

(Invoke-WebRequest "https://api.w3.pinbox24.com/api/i18n/langs" -TimeoutSec 10 -UseBasicParsing).StatusCode
# Expect: 200

PM2 logs for v32-prod: /var/log/v32-prod/pm2/pm2_v32-prod_production_out.log and _err.log

Critical: do NOT rely on Mongoose connected in PM2 logs alone — it may show connected while mongojs is still failing. Always verify with the HTTP endpoint after update.

---

Fix D: socket/reso 502 — stale image + test-net split after a compose recreate

Discovery from 2026-07-12 incident (#3924): socket.w3.pinbox24.com returned 502 while the main app (w3.pinbox24.com) and API (api.w3.pinbox24.com) stayed 200. Root cause was NOT creds — it was a bad docker-compose recreate of v32-prod-socket (and v32-prod-reso, project p24-v-32, build dir /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2) with two regressions at once. Only socket has a public blackbox probe, so only it alerts; reso fails silently the same way.

Symptom signature (how to recognise this vs Fix A/B/C):

• 502 (not 504) on socket.w3 only; main app + API are 200.
• docker exec v32-prod-socket pm2 list → app backend with a fast-climbing restart count and uptime 0–2s (pm2 listen_timeout kills the cluster worker because it never binds :3000).
• PM2 logs loop Connecting with redis in redisConfig on host redis-v32 (err) + Mongoose connecting (out) forever, never reaching Redis is ready / Mongoose connected — and no stack trace.

Two independent bugs to check (fix both):

1. Wrong image. socket/reso must run the same image as the healthy v32-prod (backend) service — the current ECR build 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod, NOT the stale private-registry.dev.pinbox24.com/v32-prod (a 2022 build whose old driver code can’t speak to the current Redis 7 / rotated Mongo). Confirm:

   for c in v32-prod v32-prod-socket v32-prod-reso; do docker inspect "$c" --format '{{.Name}} {{.Config.Image}}'; done

   Both images share an identical ecosystem.config.js (backend → ./app-backend/app.js, cluster), so swapping socket/reso to the ECR image is safe — only the app code differs.

2. Missing test-net (network split, #2826). redis-v32 lives on test-net; the healthy v32-prod is on test-net + prod-v-3-net. A recreate that drops test-net leaves socket/reso on prod-v-3-net only → getent hosts redis-v32 returns nothing → the redis loop above. Confirm:

   docker exec v32-prod-socket sh -c 'getent hosts redis-v32 || echo NO_RESOLUTION'

   (Creds are fine if docker run --rm --network test-net redis:7-alpine redis-cli -h redis-v32 -a "$PASS" PING returns PONG with the socket container’s own REDIS_PASSWORD — so don’t chase Fix C for this.)

Fix (on bms-1): correct both in p24-v-3.2/docker-compose.yml for the reso and socket services (back the file up first — cp docker-compose.yml docker-compose.yml.bak.$(date +%Y%m%d%H%M)):

• image: 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod
• add - test-net to their networks: (keep - prod-v-3-net)

cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
docker-compose up -d --no-deps --no-build reso socket   # ECR image is present locally

Verify: curl -o /dev/null -sw "%{http_code}\n" https://socket.w3.pinbox24.com/ → 200; docker inspect v32-prod-socket --format '{{.RestartCount}}' stays 0 with growing uptime.

Durability: the on-server compose is a GitLab-CI build-dir copy; the committed source pinbox24/p24-v-3.2/docker-compose.yml must get the same image + test-net fix or a CI redeploy re-breaks it (human/GitLab action). The socket/reso blocks may also be uncommitted in the build dir — a CI git clean (#3874) deletes them outright. Tracked in #3804.

---

Critical: pm2 env 0 leaks credentials

DO NOT run docker exec v42-prod pm2 env 0 without a grep filter — it prints all env vars including NEW_MONGODB_URI and v3MongoUrl in plaintext.

Safe alternative (check key names only):

docker exec v42-prod pm2 env 0 | grep -i mongo | sed 's/value: .*/value: REDACTED/'

---

MongoDB rs0 health check

# Requires password from secrets/bms-servers.env.sops key mongodb_rs0_admin_password
mongosh 'mongodb://admin:<pass>@145.239.133.104:27017/admin' \
  --quiet --eval 'rs.status().members.forEach(function(m){print(m.name,m.stateStr,m.health)})'
# Expected: bms-2 PRIMARY 1, bms-3 SECONDARY 1, bms-4 ARBITER 1

---

After fixing — verification checklist

# 1. i18n loads (200 with data)
Invoke-WebRequest "https://api.w4.pinbox24.com/api/i18n/langs" -TimeoutSec 10 -UseBasicParsing | Select-Object StatusCode
 
# 2. Login responds (401/422/500, NOT timeout)
Invoke-WebRequest "https://api.w4.pinbox24.com/api/auth" -Method POST -Body '{"email":"x","password":"x"}' -ContentType "application/json" -TimeoutSec 10 -UseBasicParsing | Select-Object StatusCode
 
# 3. w4 frontend loads
Invoke-WebRequest "https://w4.pinbox24.com" -TimeoutSec 10 -UseBasicParsing | Select-Object StatusCode
 
# 4. PM2 logs show no Mongoose disconnect
# docker exec v42-prod pm2 logs --nostream --lines 30 (check for Mongoose disconnected)

---

Prevention

1. RabbitMQ must be in a docker-compose on bms-4 (restart: unless-stopped) — not a manual docker run. See issue #1716.
2. After any rs0 admin password rotation in SOPS → update backend-environment.env for both v42-prod and v32-prod on bms-1, then recreate containers via docker-compose (see Fix B/C permanent fix sections). URL-encode the password before embedding in URIs. This is now Step 1e of the canonical rotation runbook (docs/playbooks/mongodb-credential-rotation.md) — the rotation runbook will not let you skip it.
3. Monitoring active (added #1716): api.w4.pinbox24.com/api/i18n/langs and api.w3.pinbox24.com/api/i18n/langs in Prometheus blackbox http_2xx job — EndpointDown alert fires within 2 min of failure.
4. After removing any Docker container from bms-3 or bms-4 — check dev_r_services and grep Pinbox24 container env vars to confirm nothing depended on it.