p24-infra Docs

s3-v2-container-empty-env-vars

4 min read

Playbook: s3-v2-v42-prod — Empty env vars / 403 Wasabi / CORS after deploy

Trigger

Any of:

  • File download returns {"code":"Forbidden","statusCode":403} in W4
  • CORS errors for /api/i18n/langs, /api/record-tag, /api/reg/* in W4 browser console
  • EAI_AGAIN s3-v2-v42-prod in v42-prod logs
  • PM2 crash loop with Error: bucket is required

Root cause pattern

docker restart / --restart unless-stopped does not re-read --env-file. Env vars are baked at docker run time. The container may restart (crash loop, manual restart, OOM) with stale env vars even though the source file on disk was updated.

Symptom chain

  1. pinbox24PublicBucketName / pinbox24PublicOfficeAccessKeyId empty in container → PM2 multer-s3 throws Error: bucket is required
  2. Correct keys in /opt/p24-infra/bms-1/pinbox24-w4.env (deployed by secrets-sync) but container was started before the file was populated → stale env
  3. frontendUrl missing from container (only in s3-v2-environment.env, not in pinbox24-w4.env) → CORS failure on every API call that checks origin

Confirm it

# On bms-1 — no values printed, only counts
docker exec s3-v2-v42-prod printenv pinbox24PublicBucketName | wc -c   # should be 29, not 1
docker exec s3-v2-v42-prod printenv frontendUrl | wc -c                # should be 16, not 0
docker exec s3-v2-v42-prod printenv pinbox24PublicOfficeAccessKeyId | wc -c  # should be 21
docker exec s3-v2-v42-prod env | grep -c pinbox24                      # should be ~30

If any value is 0 or 1 (empty), the container needs to be recreated.

Fix

Step 1 — verify source files have correct content

# Count length of key value in deployed file (safe — no value shown)
grep 'pinbox24PublicBucketName' /opt/p24-infra/bms-1/pinbox24-w4.env | cut -d= -f2- | wc -c
# Must be >5. If 1 (empty) → run secrets-sync first (see §Secrets-sync not run yet)

Step 2 — fix env file permissions (if CI deploy will follow)

chown root:gitlab-runner /root/s3v2-prod/s3-v2-environment.env
chmod 640 /root/s3v2-prod/s3-v2-environment.env

Step 3 — recreate container with both env files

docker stop s3-v2-v42-prod
docker rm s3-v2-v42-prod
docker run -d \
  --name s3-v2-v42-prod \
  --network test-net \
  --restart unless-stopped \
  --env-file /root/s3v2-prod/s3-v2-environment.env \
  --env-file /opt/p24-infra/bms-1/pinbox24-w4.env \
  -e CONTAINER_NAME=s3-v2-v42-prod \
  -e NODE_ENV=production \
  -e TZ=Europe/Warsaw \
  -e PORT=3000 \
  -e MAILGUN_USER_NAME=api \
  -e VIRTUAL_HOST=s3-v2-api.w4.pinbox24.com \
  -e LETSENCRYPT_HOST=s3-v2-api.w4.pinbox24.com \
  -v /root/s3v2-prod/patches/mailgunFileHandler.helper.js:/app/dist/apps/storage/mailgunFileHandler.helper.js:ro \
  -v /var/log:/var/log \
  -v /root/s3v2-prod/persistent-patches/storage.controller.js:/app/dist/apps/storage/storage.controller.js:ro \
  -v /root/s3v2-prod/persistent-patches/storage.config.js:/app/dist/config/storage.config.js:ro \
  -v /root/s3v2-prod/persistent-patches/app.routing.js:/app/dist/app.routing.js:ro \
  --log-opt max-size=10m \
  --log-opt max-file=5 \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com/s3-v2-v42-prod:latest
 
# Connect to prod-v-4-net so v42-prod can resolve hostname
docker network connect prod-v-4-net s3-v2-v42-prod || true

Step 4 — verify

sleep 5
docker exec s3-v2-v42-prod printenv pinbox24PublicBucketName | wc -c   # >=29
docker exec s3-v2-v42-prod printenv frontendUrl | wc -c                # >=16
docker exec s3-v2-v42-prod pm2 status | grep online | wc -l            # 4
docker network inspect prod-v-4-net | grep s3-v2-v42-prod             # should appear

Secrets-sync not run yet

If pinbox24-w4.env was recently updated in SOPS (new PR merged) but sync hasn’t run:

# From Windows dev machine
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-w4
# Wait for completion, then check bms-1 file size (should grow after adding pinbox24Public* keys)

Why two env files?

  • /root/s3v2-prod/s3-v2-environment.env — contains: frontendUrl, envirmnmentUrl, MONGODB_URI, SESSION_SECRET, port, monitoringToken, mailgunPassword, PM2_*. NOT in SOPS.
  • /opt/p24-infra/bms-1/pinbox24-w4.env — deployed by secrets-sync from pinbox24-w4.env.sops. Contains: all V42_* keys + pinbox24Public* Wasabi credentials.
  • Second file overrides first for any key that exists in both.

Long-term fix: migrate remaining keys from s3-v2-environment.env into pinbox24-w4.env.sops so all secrets are SOPS-managed. Tracked in issue #3171.

Escalation

If container still crashes after recreation:

  1. Check docker logs s3-v2-v42-prod --tail 50 for startup errors
  2. Check PM2 status: docker exec s3-v2-v42-prod pm2 status
  3. Verify storage.config.js patch uses process.env.* not hardcoded keys: docker exec s3-v2-v42-prod grep -c 'process.env' /app/dist/config/storage.config.js → should be ≥3
  4. If wkhtml EAI_AGAIN: docker network connect test-net wkhtml-v42-prod || true See also: pinbox24-wkhtml-network-isolation.md

Prevention

  • GitLab CI .gitlab-ci.yml (master branch) now uses BOTH env files and adds prod-v-4-net connect
  • secrets-sync.yml sync-pinbox24-w4 deploys correct content to pinbox24-w4.env
  • File permissions: 640 root:gitlab-runner on both env files (set by secrets-sync + this playbook)

Graph View