p24-infra Docs

pinbox24-nginx-roundrobin-s3

4 min read

Playbook: Pinbox24 nginx-proxy S3 Round-Robin 404 Bug

Created: 2026-07-07
Incident: api.w4.pinbox24.com alternating 200/404/CORS — every other request fails
Root cause: s3-v42-prod had VIRTUAL_HOST=api.w4.pinbox24.com identical to v42-prod, causing nginx-proxy round-robin to both containers

Symptom pattern

  • Browser network panel shows alternating 200/404 for the same endpoints (/api/i18n/langs, /api/profile/workspaces, etc.)
  • CORS errors on ~50% of requests (S3 service lacks CORS headers for w4.pinbox24.com origin)
  • Specific pattern: every 2nd request fails — classic round-robin signature
  • PM2 shows stable (no restarts, single process online) — NOT a crash loop

Root cause

jwilder/nginx-proxy groups all containers with the same VIRTUAL_HOST into one upstream and round-robins ALL requests between them. When s3-v42-prod (the OLD S3 microservice, v4-s3 image) shares VIRTUAL_HOST=api.w4.pinbox24.com with the Node.js backend (v42-prod):

upstream api.w4.pinbox24.com {
    # v42-prod — handles /api/* correctly → 200
    server 172.18.0.6:3000;
    # s3-v42-prod — handles /upload/* only, returns 404 for all other /api/*
    server 172.18.0.4:3000;
}

nginx round-robins ALL requests → 50% hit the S3 service → 404 or CORS error.

Confirm diagnosis

ssh root@94.23.26.113 'docker exec nginx-proxy grep -A20 "^upstream api.w4.pinbox24.com {" /etc/nginx/conf.d/default.conf | head -22'
# BUG: shows both v42-prod AND s3-v42-prod
# FIXED: shows only v42-prod

Also confirm the S3 container’s VIRTUAL_HOST:

ssh root@94.23.26.113 'docker exec s3-v42-prod printenv VIRTUAL_HOST 2>/dev/null'
# BUG: returns "api.w4.pinbox24.com"
# FIXED: no output (VIRTUAL_HOST not set)

Fix

Remove VIRTUAL_HOST and LETSENCRYPT_HOST from s3-environment.env, then recreate the container.

Step 1 — Remove VIRTUAL_HOST from s3-environment.env

# Use Python — sed has quoting issues when called from PowerShell here-strings
$script = @"
import re
path = '/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/s3-environment.env'
with open(path) as f:
    lines = f.readlines()
filtered = [l for l in lines if not re.match(r'^(VIRTUAL_HOST|LETSENCRYPT_HOST)=', l)]
with open(path, 'w') as f:
    f.writelines(filtered)
print('removed', len(lines) - len(filtered), 'lines')
"@
$script | ssh root@94.23.26.113 'python3 /dev/stdin'
# Expected: "removed 2 lines"

Step 2 — Recreate s3-v42-prod

ssh root@94.23.26.113 'cd /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts && CONTAINER_NAME=v42-prod docker-compose up -d --force-recreate --no-build s3 2>&1'
# Expected: "Recreating s3-v42-prod ... done"

Step 3 — Restart nginx-proxy

ssh root@94.23.26.113 'docker restart nginx-proxy 2>&1'
# Expected: "nginx-proxy"

Step 4 — Verify upstream

ssh root@94.23.26.113 'sleep 3 && docker exec nginx-proxy grep -A12 "upstream api.w4.pinbox24.com {" /etc/nginx/conf.d/default.conf | head -14'
# Expected: ONLY "# v42-prod" in upstream, no "# s3-v42-prod"

Step 5 — Test API

1..5 | ForEach-Object {
    $r = Invoke-WebRequest "https://api.w4.pinbox24.com/api/i18n/langs" -TimeoutSec 10 -UseBasicParsing -ErrorAction SilentlyContinue
    Write-Host "Request $_ : $($r.StatusCode)"
}
# Expected: 200 200 200 200 200

What does s3-v42-prod do after this fix?

s3-v42-prod (OLD v4-s3 image) remains running and accessible within prod-v-4-net. It is still reachable from v42-prod at http://s3-v42-prod:3000 for internal Docker network calls. Only the external nginx routing is removed. This preserves:

  • Internal S3 proxy calls from v42-prod (if any)
  • File upload handling for paths the backend delegates internally
  • The bms1-post-restart-recovery.md step 3 still starts this container — that’s fine

After bms-1 restart

The bms1-post-restart-recovery.md playbook starts s3-v42-prod via docker start s3-v42-prod. Since the container now has no VIRTUAL_HOST, nginx-proxy will NOT add it to any upstream on startup. This is the correct behavior.

Prevention

  • The infra-src/pinbox24/w4/docker-compose.yml in p24-infra should document that s3-environment.env must NOT set VIRTUAL_HOST=api.w4.pinbox24.com. This env var was set when the OLD S3 service was intended to receive file upload requests directly from nginx. With the uploadAwsS3.helper.js persistent patch in v42-prod handling uploads directly, the S3 service no longer needs to be reachable externally.
  • TODO (issue #3161 follow-up): migrate W4 to use s3-v2-api.w4.pinbox24.com for all S3 operations and remove s3-v42-prod entirely (tracked in CI/CD redesign).

References

  • Issue #3161 — v42-prod crash + CORS root cause investigation
  • PR #3163 — PM2 fork mode fix (partial fix for the same incident)
  • pinbox24-w3-w4-outage-diagnosis.md — general W4 outage diagnosis
  • bms1-post-restart-recovery.md — container start sequence after bms-1 restart

Audit Log

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 's3-v42-prod', 'success', 'Removed VIRTUAL_HOST=api.w4.pinbox24.com from s3-environment.env — nginx round-robin 404 fix', 'bms-1', 3163)
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View