w3-s3-hostname-alias-drift

Playbook: W3 s3-v32-prod hostname alias drift → upload/import hangs (504), 404s, or 403s

Trigger

• Client report: “can’t upload files” / RESO Excel import fails with excel file is not found.
• nginx-proxy logs upstream timed out (110: Connection timed out) on POST /api/files/{officeId}/upload to v32-prod, ending in HTTP 504 — or, once DNS is fixed but the sidecar is still unpatched, a fast clean HTTP 404 (X-Powered-By: Express, plain <pre>Not Found</pre> body) — or, once patches are mounted but credentials are stale, a fast HTTP 403 (X-Powered-By: Express, plain <pre>Forbidden</pre> body — Wasabi’s own auth rejection propagated through multer-s3).
• No new documents appear in w3_db.files despite the client submitting.

Three independent faults can stack on the same symptom family. #4659 (2026-07-30) hit all three in sequence on the same underlying container: (1) the s3-v32-prod hostname didn’t resolve at all → 504 hang; (2) once that resolved, the container behind it was running unpatched → fast 404; (3) once patched, it turned out to be running stale Wasabi + MongoDB credentials (matching an already-revoked key from a 2026-07-01 incident) → fast 403 from Wasabi. Fix all three — a green status code at any single step does not mean the upload actually works. Verify with a real multipart upload, not just a connectivity probe, before declaring the incident closed.

Root cause class

v32-prod (and any other v3.2 backend variant — v32-prod-socket, v32-prod-reso) proxies file uploads to a hostname it expects to be s3-v32-prod on the prod-v-3-net Docker network. Over time, the actual file-microservice container on bms-1 has been renamed (s3-v32-prod-renamed — see the artnet-decommission tracker in docs/priorities.md). A container rename does not carry over the old Docker-network DNS name unless an explicit --alias is set. The app has known history (crudService.js Fix B2) of hanging on a broken async file lookup instead of failing fast, so the symptom is a slow 504 (nginx’s own proxy_read_timeout killing it), not an immediate connection error.

A container recreate/restart of v32-prod (e.g. an unrelated MR deploy) can flip this from “silently degraded” to “fully broken” if it resets any in-app DNS/connection caching that had been masking the stale reference.

Confirm

# PLAYBOOK: w3-s3-hostname-alias-drift.md
ssh root@94.23.26.113 "docker exec v32-prod getent hosts s3-v32-prod"
# Empty output = broken. An IP = healthy.
 
ssh root@94.23.26.113 "docker logs nginx-proxy --since <window> 2>&1 | grep 'files.*upload' | grep -i '504\|timed out'"

Fix (safe, reversible, no code/secret change)

Add the network alias back so the old expected hostname resolves to whichever container is actually serving the role:

# PLAYBOOK: w3-s3-hostname-alias-drift.md
ssh root@94.23.26.113 "docker network disconnect prod-v-3-net s3-v32-prod-renamed && \
  docker network connect --alias s3-v32-prod prod-v-3-net s3-v32-prod-renamed"

Brief (~sub-second) network detach on that one container; no restart, no port exposure change, no secret/code change. Does not require a GitLab MR or human merge.

Verify

# PLAYBOOK: w3-s3-hostname-alias-drift.md
ssh root@94.23.26.113 "docker exec v32-prod getent hosts s3-v32-prod"
ssh root@94.23.26.113 "docker exec v32-prod curl -s -o /dev/null -w '%{http_code}' --max-time 15 http://s3-v32-prod:3000/"
# Expect: resolves + 200

Then confirm end-to-end via docs/playbooks/reso-import-post-incident-verify.md (HTTP 200 alone is not proof — need a real new w3_db.tasklogs doc with activityName:"Import poprawny", status:"completed" after the fix timestamp).

If the DNS check above passes (200) but a real multipart upload still 404s (test with a real file, not a bare POST — a bare POST correctly 500s, that’s a different, expected code path):

# PLAYBOOK: w3-s3-hostname-alias-drift.md
docker exec v32-prod sh -c 'printf "%%PDF test" > /tmp/t.pdf'
docker exec v32-prod curl -s -i --max-time 20 -X POST \
  "http://localhost:3000/api/files/<officeId>/upload" -F "uplFile=@/tmp/t.pdf;type=application/pdf" \
  -H "authorization: test"
# A clean 404 here (not 403/422) with a real file attached means the SECOND fault below is present.

This means the container behind the alias (commonly s3-v32-prod-renamed) is running without its persistent-patch mounts — check docker inspect s3-v32-prod-renamed --format='{{json .Mounts}}'; [] confirms it. Per docs/pinbox24-w3-w4-architecture-spec.md §2.2 it needs:

Host source	Container path	Purpose
persistent-patches/local.js	/app/config/local.js	MongoDB/Wasabi config override
persistent-patches/controller.js	/app/build/src/api/storage/controller.js	The real multer-s3 + AWS.S3 upload handler — without it the container serves stock/unpatched code that 404s on this route

Source files live in the GitLab CI build dir, e.g. /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/persistent-patches/{local.js,controller.js} — check they’re real files (not empty directories left by a git clean, see docs/playbooks/w3-w4-redeploy-idempotency.md) before using them as mount sources.

Fix: the container needs a recreate to add bind mounts (Docker cannot hot-add volumes to a running container). Stop + rename (never remove) the old container as a rollback path, then docker run with the same image/env/network + --network-alias s3-v32-prod + the two -v ...:ro mounts above. Gotcha hit live: building the --env list via {{range .Config.Env}}{{.}}{{"\n"}}{{end}} + mapfile leaves a trailing empty line, producing an invalid empty --env "" arg that fails the docker run — filter blank lines (grep -v '^$') before building the env array, or the recreate fails after the old container is already stopped, leaving a real (if brief) outage window.

Verify with the same real-file multipart test above — expect a genuine auth response (403/422 for a bad/missing token) instead of 404, proving the request now reaches the patched handler.

If the real-file test above now returns 403 Forbidden even with a known-good/real client token (not just a deliberately-bad test token — check nginx-proxy’s own access log for the client’s actual retry to confirm it’s hitting the same 403, not assuming your test token explains it):

The container is reachable and patched but running stale credentials — a long-dormant/legacy container (commonly one that’s been renamed, like s3-v32-prod-renamed) can sit for weeks with a Wasabi key or MongoDB URI that was rotated/revoked elsewhere and never propagated to it, because secrets-sync targets known container names, not orphaned renamed ones.

# PLAYBOOK: w3-s3-hostname-alias-drift.md — compare live vs current, never print either value
docker exec s3-v32-prod-renamed printenv s3Bucket_api_accessKeyId > /tmp/live_key
grep '^V32_s3Bucket_api_accessKeyId=' /opt/p24-infra/bms-1/pinbox24-w3.env | cut -d= -f2- > /tmp/sops_key
diff -q /tmp/live_key /tmp/sops_key && echo MATCH || echo MISMATCH
rm -f /tmp/live_key /tmp/sops_key

Also check s3Bucket_api_secretAccessKey and DB_URI the same way. This step touches real credential values — building a docker run --env KEY=value command with secret material is expected to trip the auto-mode safety classifier; get explicit user confirmation before retrying, same as any credential-write action. Fix: recreate the container once more (same pattern as above — stop, rename old as backup, docker run with the same image/network/patch-mounts) with just the stale env vars replaced by the current values from the SOPS-deployed reference file (/opt/p24-infra/bms-1/pinbox24-w3.env for W3) — this is applying an already-current, already-decrypted value to fix container drift, not minting or rotating a new secret, so it stays in sys-admin scope per docs/w3-w4-stack-operations.md §2 rather than needing a secret-manager handoff.

Verify with a real success, not just a status code: the multipart test should return 200 with a JSON body containing a real Wasabi location URL and bucket field — that’s the only proof the full chain (routing → patched handler → Wasabi auth → S3 write) actually works. Delete any test file’s resulting w3_db.files record afterward if you used a real client’s officeId/recId for the test — don’t leave synthetic test data mixed into their case history.

Prevention

• This is a network-level fix — it survives docker-compose restart/up -d on v32-prod (the alias lives on the target container’s network attachment, not on v32-prod), but is lost if s3-v32-prod-renamed itself is force-recreated (a recreate drops manually-added aliases; only the compose-defined name/aliases survive). If a future redeploy of the s3 sidecar re-breaks this, re-apply the fix above, then also add s3-v32-prod under networks.prod-v-3-net.aliases in the committed docker-compose.yml on the GitLab side so it survives a real CI redeploy — that requires a human-merged MR per docs/w3-w4-stack-operations.md §2.
• Any future container rename on bms-1 must add a network alias for the old name (or update every consumer’s config) in the same change — do not rename in place and assume DNS “just follows”.

Related

• Incident: #4659
• docs/pinbox24-w3-w4-outage-diagnosis.md Fix D — same failure class (network split after recreate), different symptom (502 not 504)
• docs/playbooks/reso-import-post-incident-verify.md — post-fix confirmation
• docs/w3-w4-stack-operations.md — ownership/escalation for any code-level (docker-compose.yml) follow-up