p24-infra Docs

v32-reso-socket-stale-image-redis-v5-crashloop

4 min read

Playbook: v32-prod-reso / -socket crash-loop after a redis-client (v5) cutover — stale image

Service: v32-prod-reso (w3.reso-integration-addrecords.pinbox24.com, RESO EUROPA — ~86% of W3 traffic) and v32-prod-socket, on bms-1 (94.23.26.113). First seen: radieu/p24-infra#3923 (2026-07-12), the night of the #3712 local-redis-v32 cutover.

Trigger / symptoms

  • EndpointDown (blackbox) on https://w3.reso-integration-addrecords.pinbox24.com — fast, consistent HTTP 502 (nginx-proxy up, upstream dead). May pair with the same on the socket endpoint.
  • docker exec v32-prod-reso pm2 list → backend online but restart climbing (30→100+) with uptime 0s; the app never binds port 3000.
  • App logs are a red herring: pm2 out spams Mongoose connecting, pm2 err spams Connecting with redis in redisConfig on host redis-v32 — with no explicit stack. MongoDB and Redis are actually reachable and their credentials are correct; do not chase them.

The actual root cause

The bind-mounted patch persistent-patches/crudService.js is written for the node-redis v5 API (redis.createClient({ socket: { host, port } }) + redisClient.connect()). The crash is:

TypeError: helperService.redisClient.connect is not a function
    at /app/app-backend/helperFunction/crudService.js:97

.connect() only exists in node-redis v4+. It appears when the container’s image ships an old redis npm (e.g. 2.8.0). The main v32-prod runs the ECR image (563740926945…/v32-prod, redis 5.12.1) so it works; if reso/socket are still on the old private-registry.dev.pinbox24.com/v32-prod image (redis 2.8.0) they hard-crash on boot.

Underlying cause: the cutover updated docker-compose.yml’s reso/socket image: to the ECR v5 image, but the running containers were never recreated, so they kept the stale image. (An earlier recreate had also dropped reso’s test-net attachment — see v32-prod-prod-v-3-net-persistence.md.)

Diagnose (agents CAN SSH bms-1 from bms-4 post-VPN-decommission)

ssh -i ~/.ssh/vps_root_key root@94.23.26.113
# 1. Confirm the crash (not creds): run the app once in foreground
docker exec v32-prod-reso pm2 stop backend
docker exec v32-prod-reso sh -c 'cd /app && timeout 15 node app-backend/app.js 2>&1' | grep -i 'connect is not a function'
docker exec v32-prod-reso pm2 start backend
# 2. Compare redis npm version reso vs the healthy main backend
docker exec v32-prod-reso node -e 'console.log(require("redis/package.json").version)'   # bad: 2.8.0
docker exec v32-prod       node -e 'console.log(require("redis/package.json").version)'   # good: 5.x
# 3. Confirm the compose file already declares the correct ECR image + both networks
D=/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
grep -nE 'container_name: v32-prod-(reso|socket)|image:|- (test-net|prod-v-3-net)' $D/docker-compose.yml
#   ⚠️ NEVER run bare `docker-compose config` here — it prints all W3 prod creds (incident #3750).

Fix — recreate reso/socket from the (corrected) compose file

If the compose reso/socket blocks already point at the ECR v5 image and list both prod-v-3-net + test-net (they should after the cutover), just recreate:

# PLAYBOOK: v32-reso-socket-stale-image-redis-v5-crashloop.md
D=/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
ssh -i ~/.ssh/vps_root_key root@94.23.26.113 \
  "cd $D && docker-compose up -d --no-deps --force-recreate reso socket"

If the compose blocks are NOT yet corrected, edit only the image: line (→ 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod) and ensure networks: lists both prod-v-3-net and test-net for each of reso and socket, then run the recreate above. Use targeted sed/edit on those lines only — do not touch env_file blocks, never print the file.

The ECR v5 image is already present locally (docker images | grep 976d1ac2f4ed), so no registry pull/auth is needed. It is the exact image the healthy v32-prod backend runs.

Verify recovery

docker exec v32-prod-reso pm2 list | grep backend     # online, restart 0, uptime climbing
docker exec v32-prod-reso node -e 'console.log(require("redis/package.json").version)'  # 5.x
docker inspect v32-prod-reso -f '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'  # prod-v-3-net test-net
curl -s -o /dev/null -w '%{http_code}\n' https://w3.reso-integration-addrecords.pinbox24.com   # 200

Close the EndpointDown issue once the endpoint is a stable 200 and PM2 restart stays flat.

Prevention / systemic follow-up

  • Any redis-client API bump in persistent-patches/crudService.js must land together with an image whose redis npm ≥ 4 for every container that mounts it (v32-prod, reso, socket). A patch that runs on one tier’s image but not another’s silently strands the lagging tier.
  • The committed source-of-truth compose IS reconciled by p24-infra, not GitLab. secrets-sync.yml (job sync-pinbox24-w3) scps infra-src/pinbox24/w3/docker-compose.yml to the deploy path /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/docker-compose.yml and then docker-compose up -d --force-recreate ... reso socket. So any manual on-server compose edit is transient — the next pinbox24-w3 sync overwrites it and force-recreates from the repo. The reso/socket ECR-image + test-net topology was persisted into the committed compose in #3933 (2026-07-12); an earlier redis-v5 fix that only touched the on-server file would be silently reverted on the next sync. Always land the compose fix in infra-src/pinbox24/w3/docker-compose.yml.

Graph View