p24-infra Docs

w3-redis-v32-missing-container

7 min read

W3 redis-v32 missing container — Playbook

Status: STABLE — executed and verified 2026-07-13 09:16 UTC on bms-1

What triggers this — UPDATE 2026-07-13 10:10 UTC: recurs after any selective docker-compose recreate

Confirmed recurrence: redis-v32 was fixed at 09:16 UTC, then disappeared AGAIN at ~10:08 UTC during a secret-manager worker run (#4078, w3_app rotation) that recreated v32-prod-reso/v32-prod-socket/s3-v32-prod. Root cause: redis IS defined in the tracked docker-compose.yml, but any workflow that does docker-compose down (removes ALL project containers) followed by docker-compose up -d <specific services> (recreates only the named ones) silently drops redis-v32 since it’s never in the “specific services” list — even though it’s still in the compose file. Any operator/worker recreating W3 containers must explicitly include redis in the up -d service list, or run a bare docker-compose up -d (all services) instead of down + selective up. Re-fix is the same one-liner as below; consider this a standing risk until the deploy script itself is fixed to never drop redis from its recreate scope.

Report of “upload/import files not working” on W3 (w3.pinbox24.com and/or w3.reso-integration-addrecords.pinbox24.com / socket.w3.pinbox24.com), where raw file storage to Wasabi actually succeeds (check s3-v32-prod logs — location/etag present) but the overall upload→import flow fails or hangs. Root cause found 2026-07-13: the redis-v32 container (used by persistent-patches/crudService.js, the node-redis v5 fix from PR #3607) was not running at all — not stopped, fully absent from docker ps -a. v32-prod / v32-prod-reso / v32-prod-socket logs show a continuous, non-fatal but functionally-breaking error:

Redis config error  Error: getaddrinfo EAI_AGAIN redis-v32

Pre-conditions

  • SSH access to bms-1 (94.23.26.113)
  • Confirm redis-v32 really is absent (not just stopped): 
    docker inspect redis-v32
# "No such object" == absent, needs `docker-compose up`, not `docker start`
  • Identify the REAL live build dir for the running containers — do not assume a path from an older playbook. This repo’s build dir has moved multiple times (/root/builds/pn3C9eHo/... → /home/gitlab-runner/builds/eZQeLfuJe/... → CI may create yet another checkout, e.g. an unrelated /temp/p24-v-3.2/ was found stale/empty during this investigation — do not confuse it with the live one): 
    docker inspect v32-prod-reso --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'
# Confirmed 2026-07-13: /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
  • Confirm redis-v32-environment.env exists in that build dir with today’s (or a recent) mtime — secrets-sync/CI stages it there independently of backend-environment.env (deliberate, see infra-src/pinbox24/w3/docker-compose.yml comment on the redis: service — #3712 Phase 2: this container’s password is intentionally NOT the V32_REDIS_PASSWORD/OVH-DBaaS value that lands in backend-environment.env, to avoid silently coupling the two). If this file is missing or empty, STOP — do not improvise a password; that mismatch class of bug is exactly what caused the 2026-07-09 P1 cascading outage (docs/incidents/2026-07-09-w3-cascading-outage-postmortem.md). Escalate instead. 
    ls -la /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/redis-v32-environment.env

Steps

  1. cd into the confirmed live build dir and bring up only the redis service — this reads redis-v32-environment.env automatically, no manual password handling needed:

    cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
docker-compose up -d redis

    Expected output: Creating redis-v32 ... done (or Starting if it already existed).

  2. Confirm the container is running and on the right network (test-net — not prod-v-3-net; v32-prod/-reso/-socket are on both, so test-net alone is sufficient and matches the tracked compose file exactly):

    docker inspect redis-v32 --format '{{.State.Status}} {{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
# Expected: running test-net

Verification

# 1. Container up
docker ps --filter name=redis-v32 --format '{{.Names}} {{.Status}}'
 
# 2. Auth is actually enforced (proves --requirepass took effect) WITHOUT ever typing the
#    password — an unauthenticated ping must be refused:
docker exec redis-v32 redis-cli ping
# Expected: (error) NOAUTH Authentication required.
 
# 3. The EAI_AGAIN error stops appearing in fresh backend logs (existing tail, no new hits)
docker exec v32-prod sh -c "date"   # note current time, then re-check err log after ~1 min
tail -20 /var/log/v32-prod/pm2/pm2_v32-prod_production_err.log | grep -c redis-v32
 
# 4. End-to-end: perform a real upload/import via the app UI and confirm the flow that
#    previously hung/failed now completes (this is the actual repro — container health
#    alone doesn't prove the upload flow works).

Rollback

docker-compose stop redis (or docker rm -f redis-v32) — no other service depends on redis-v32 being present in a way that a restart would break further; the prior (broken) state was simply the container’s absence.

Common mistakes

  • Guessing the password / reusing V32_REDIS_PASSWORD from backend-environment.env: that key is a different, deliberately-decoupled OVH-DBaaS credential per #3712 Phase 2 — do not wire it into a hand-rolled docker run --requirepass. Always let docker-compose up -d redis read redis-v32-environment.env itself.
  • Checking the wrong build directory: this host has multiple stale build-dir remnants from prior incidents (e.g. /temp/p24-v-3.2/ was found with a 0-byte, 2021-dated backend-environment.env during this investigation — completely unrelated to the live containers). Always confirm via docker inspect ... com.docker.compose.project.working_dir on an actually-running sibling container first.
  • Attaching redis-v32 to prod-v-3-net instead of test-net: the tracked compose file puts it on test-net only; v32-prod/-reso/-socket are already on both networks so this is sufficient — don’t “fix” this without checking the compose file first.
  • Assuming container-health = feature-health: redis-v32 being Up and answering PING with NOAUTH only proves the server started — it does not prove the app’s Redis client successfully authenticated or that the upload/import flow completes. Always do the end-to-end check (step 4).

Execution record — 2026-07-13

docker-compose up -d redis run from the confirmed live build dir at 09:16 UTC. redis-v32 came up on test-net, redis-cli ping (unauthenticated) correctly returned NOAUTH Authentication required., and v32-prod’s error log shows the last getaddrinfo EAI_AGAIN redis-v32 line at 09:16 (same minute as container start) followed immediately by Redis is connected with host - redis-v32 / Redis is ready in its stdout log — confirms the app’s Redis client authenticated successfully. No password was ever typed, printed, or reconstructed by hand.

Separate finding surfaced during verification (NOT fixed by this playbook): v32-prod-reso and v32-prod-socket are independently crash-looping on MongoError ... AuthenticationFailed (code 18) — 55,958+ occurrences each since their 07:00 UTC recreation today. This is unrelated to Redis (no redis errors/successes appear in their logs at all) and matches the exact w3_app MongoDB password-drift pattern from the 2026-07-12 incident (#3911, docs/secrets-rotation-log.md) recurring after today’s deploy. This is a credential issue — out of scope for an orchestrator session per CLAUDE.md (Never attempt SOPS operations in the current session); needs secret-manager / docs/playbooks/secret-manager-request.md, not a live fix here.

Execution record — 2026-07-30 (recurred via a FAILED deploy; alert misattributed it to S3)

Second confirmed recurrence, this time triggered by a failed sync-pinbox24-w3 CI run (run 30559731530) at 16:04:22Z whose selective docker-compose recreate of s3-v32-prod/-reso/-socket dropped redis-v32 — same “selective recreate never names redis” mechanism as the standing-risk note above, now proven to fire even when the overall deploy job errors out (it collided on the label-less v32-prod name, 4572). All three W3 backends looped on getaddrinfo EAI_AGAIN redis-v32 at ~28/min from 16:05Z, raising the Prometheus V32ProdErrorSpike alert (#4689, pinbox24_backend_error_events_5m_total{v32-prod} = 135).

Watch-out for the next responder: the alert’s first-pass triage misread this as an S3 binding failure (v32-prod → destroyed s3-v32-prod). It was not — always read the actual stderr signature before accepting a hypothesis. The EAI_AGAIN redis-v32 line is unambiguous and points straight here.

Fix identical to the 2026-07-13 record: docker-compose up -d redis from the (still-current) live build dir /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2 → Creating redis-v32 ... done. Pre-flight confirmed redis-v32-environment.env present (mtime 16:04, re-staged by the same CI run) and docker-compose config clean with no ${CLUSTER_NETWORK} undefined-var trap (this compose file’s top-level networks: are hardcoded external, so #3218 does not apply). Verified: redis-v32 up on test-net, unauth redis-cli ping → NOAUTH Authentication required., all three backends logged Redis is ready and dropped to 0 EAI_AGAIN after reconnect, api.w3.pinbox24.com/api/i18n/langs → 200. Executed by bms4-cw-1 (infra-task worker). No password typed, printed, or reconstructed.

  • docs/incidents/2026-07-09-w3-cascading-outage-postmortem.md — the P1 this system’s credential/env drift pattern originally came from
  • docs/playbooks/w3-rollback-2026-07-10.md — known open risk: GitLab CI wipes env files on every deploy without an auto-retriggered secrets-sync
  • docs/playbooks/pinbox24-w3-operations.md — general W3 ops reference (note: its build-dir path is stale as of 2026-07-06; trust docker inspect over any hardcoded path in any doc)
  • infra-src/pinbox24/w3/docker-compose.yml — tracked source of truth for the redis: service definition (see #3712 Phase 2 comment)

Graph View