2026-07-09-w3-cascading-outage-postmortem

Postmortem: W3 (Pinbox24) cascading production outage — 2026-07-09

Status: Resolved. Severity: P1 (full production outage, multi-hour). Services affected: v32-prod (W3 backend, bms-1), s3-v32-prod (W3 file storage, bms-1). Related: #3171 (orchestrator), #3562 (hardcoded-credential fix that triggered this), #3603 (redis v5 fix), #3611 (mongoose/PMONGODB_URL bug), #3614 (security follow-up).

TL;DR

A routine merge to fix a hardcoded-credentials security issue (#3562) triggered a GitLab CI rebuild+redeploy of W3. The CI pipeline’s checkout/clean step wiped two untracked env files (backend-environment.env, s3-environment.env) that held the entire runtime configuration — not just secrets. What followed was a chain of seven distinct, independently-diagnosed bugs, each masking the next, that took the service from “completely down” to “fully working” over several hours. The deepest and most expensive bug was a silent drift between two separate SOPS keys (V32_MONGODB_URL vs V32_PMONGODB_URL) that only one of the app’s two database clients (mongoose) actually used — and an automated CI job kept re-introducing the stale value on every deploy.

Timeline (chronological, cause → symptom → fix)

0. Trigger

Merging the JWT/Wasabi hardcoded-credential fix (#3562, GitLab MRs !66/!67) triggered GitLab CI’s prod-back-end-deploy job. Its checkout/git clean step wiped backend-environment.env and s3-environment.env on bms-1 — these files are not tracked in git (by design, they’re SSH-delivered by secrets-sync.yml), so GitLab’s own clean step has no way to know not to delete them. Result: v32-prod restarted with zero environment variables.

1. Bug: update_key() silently no-ops on an empty file

secrets-sync.yml’s Python helper update_key() only replaces an existing KEY=value line — if the file is empty, it finds nothing to replace and does nothing, but the code unconditionally prints "Updated KEY" regardless of whether anything actually changed. Re-triggering secrets-sync.yml looked successful in the logs but left the file empty. Fix: added upsert_key() (appends if not found) earlier this session (#3590); this incident is why it needed to be used everywhere, not just for the newly-added JWT/S3 keys (see Follow-ups).

2. Bug: entire non-secret runtime config was missing

backend-environment.env originally held ~15 non-secret keys never managed by secrets-sync.yml’s narrow secret-rotation logic: CONTAINER_NAME, NODE_ENV, LOGDNA_KEY, REDIS_HOST, REDIS_PORT, REDIS_PASSWORD, VIRTUAL_HOST, LETSENCRYPT_HOST. Once the file was fully wiped, none of these came back automatically. Symptom: PM2 crash-looped every ~1-2s. ecosystem.config.js builds its log file path from CONTAINER_NAME/NODE_ENV (a red herring, non-fatal); the real fatal errors were downstream requires throwing synchronously. Fix: pushed all V32_*-prefixed SOPS keys generically (prefix-stripped) to backend-environment.env in one pass.

3. Bug: VIRTUAL_HOST had an invisible BOM prefix

A PowerShell→SSH stdin pipe introduced a UTF-8 BOM on the first line of one push, silently prefixing VIRTUAL_HOST. nginx-proxy auto-discovers containers by exact env var name — with the BOM, it never saw this container, so api.w3.pinbox24.com returned 503 even once the app itself was healthy. Fix: sed 's/^\xEF\xBB\xBFVIRTUAL_HOST=/VIRTUAL_HOST=/', then force-recreate.

4. Bug: node-redis v5 API break in crudService.js

The rebuilt image picked up redis@5.12.1 (via an unpinned "redis": "^5.10.0" dependency). persistent-patches/crudService.js used the v2/v3-era API: redis.createClient(port, host) (positional args) and .auth(password) (bare string). Both are invalid in v5: createClient takes a single options object, and .auth() needs {password} — calling it with a raw string throws Error: username is required. Fix (permanent, merged 3607 by a parallel session + this session): rewrote to redis.createClient({socket:{host,port}, password}) + explicit .connect().

5. Bug: Twilio/OneSignal SDK clients constructed unconditionally at require-time

Twilio and OneSignal credentials were correctly removed from SOPS as decommissioned (#3581 — confirmed unused: “one signal is push notification service - we not use it on w3 too”). But services/twilioService.js (require('twilio')(accountSid, authToken)) and helperFunction/pushNotification.helper.js (new OneSignal.Client({userAuthKey, ...})) both construct their SDK client synchronously at module load, with no guard. With the credentials gone, both threw (username is required / userAuthKey must be a string) on every single boot — crash-looping PM2 in a tight ~1-2s cycle. Fix: guard both client constructions behind a truthiness check on the credential, falling back to a stub object that rejects if actually invoked (services are confirmed unused, so this is safe).

6. Bug: w3_app MongoDB password drifted between live and SOPS

During recovery, the w3_app password was rotated live (atomic generate→verify→persist), but the SOPS commit landed on a stray local branch and was never merged to main — meaning main’s SOPS state still had the pre-rotation password. Any future secrets-sync run would have pushed the stale value back and re-broken the service. Fix: cherry-picked the rotation commit onto a clean branch off main, merged (PR #3609).

7. Bug (root cause, most expensive): V32_PMONGODB_URL is a separate SOPS key from V32_MONGODB_URL, and they drifted independently

This is the one that took the longest to find. The app has two independent MongoDB clients:

• config/mongo_db.js (mongojs) — connects via props.MONGODB_URL
• config/mongoose.js (mongoose) — connects via props.PMONGODB_URL

With NODE_ENV=production set (needed for correct PM2 log paths), env.config.js’s production block reads process.env.MONGODB_URL / process.env.PMONGODB_URL directly — two entirely separate raw env vars, sourced from two entirely separate SOPS keys (V32_MONGODB_URL and V32_PMONGODB_URL). Multiple rotation passes during the incident (by this session and a parallel one) kept updating V32_MONGODB_URL but not V32_PMONGODB_URL in lockstep. Symptom: mongojs-backed routes (login, offices, profile) worked perfectly; every mongoose-model-backed route (forms, /api/i18n/langs, dashboard widgets, “datatable” listings) hung for mongoose’s 10s buffering timeout and returned 422 with MongooseError: Operation \X.findOne()` buffering timed out after 10000ms` — this exact symptom is the original “langi się nie ładują” report that kicked off the whole session, i.e. it pre-dates today’s other bugs entirely.

Made worse by: secrets-sync.yml re-populates PMONGODB_URL from V32_PMONGODB_URL on every push to main — so even a correct manual live fix got silently reverted by the next unrelated merge (there were several, from parallel sessions working the same incident). Fix: aligned V32_PMONGODB_URL = V32_MONGODB_URL in SOPS itself (PR #3612) — fixing the source, not just the live container, so it survives future syncs. Also added a connect-retry loop to config/mongoose.js (this driver has no built-in reconnect) as defense in depth.

8. Bug: fixes #4 and #5 were initially only live-patched (docker exec), not committed

Every docker-compose up -d --force-recreate (by any session, or any automated deploy step) discards the container’s writable layer — including live docker exec file edits. The Twilio/OneSignal guards and the mongoose retry loop kept vanishing every time anyone recreated the container, including an automated secrets-sync run triggered by this incident’s own PR merges. Fix: moved all three patched files into infra-src/pinbox24/w3/persistent-patches/ (the same volume-mount mechanism already used for crudService.js/filesUpload.js) and wired them into docker-compose.yml (PR #3613) — and committed that docker-compose.yml change itself, since an earlier version of this exact fix was deployed by hand-SCP without a git commit, and the very next automated secrets-sync run reverted it back to the un-patched version from git.

9. Bug: s3Bucket_endpoint / s3Bucket_BUCKET_NAME missing from s3-environment.env

Same root cause as #2 (file-wipe never fully recovered) but for the s3 container specifically — secrets-sync.yml only ever pushed DB_URI + the two S3 key-pair values to s3-environment.env, never the endpoint/bucket name. Wasabi client construction in s3-v32-prod’s config/local.js got undefined for both, causing file upload/read to hang until a 504 Gateway Timeout. Fix: pushed s3Bucket_endpoint/s3Bucket_BUCKET_NAME from the corresponding V32_* SOPS keys; force-recreated s3-v32-prod. Confirmed fixed: file upload works.

Security incident (separate, during recovery)

A diagnostic printenv DB_URI on s3-v32-prod printed the full MongoDB connection string — including the w3_app password — in plaintext to chat output. Filed as #3614. Root cause: DB_URI is a full connection URI, unlike the host/user-only vars this session was otherwise careful to redact before printing. Rotation deferred to the next full deployment per explicit direction (credentials get rotated on every redeploy per existing team practice).

Where the credentials actually live (key names only, never values)

All in secrets/pinbox24-w3.env.sops (SOPS+age), delivered to bms-1 via .github/workflows/secrets-sync.yml:

SOPS key	Consumed as (env var on bms-1)	Used by
V32_MONGODB_URL	MONGODB_URL	config/mongo_db.js (mongojs) — auth, offices, profile, most routes
V32_PMONGODB_URL	PMONGODB_URL	config/mongoose.js (mongoose) — forms, langs, dashboard widgets, datatables. Must always equal V32_MONGODB_URL — they are the same w3_app@w3_db account, split into two keys only because two separate legacy DB clients exist.
V32_DB_URI	DB_URI	s3-v32-prod’s config/local.js (mongoose, file-storage service)
V32_PINBOX_PROD_DB	PINBOX_PROD_DB	env.config.js’s local block fallback for MONGODB_URL/PMONGODB_URL when NODE_ENV is not production
V32_MONGODB_W3_APP_PASSWORD	— (reference only)	Not consumed directly by the app; tracks the same w3_app password for rotation-script bookkeeping
V32_JWT_TOKEN_SECRET	JWT_TOKEN_SECRET	env.config.js local block (post-#3562 fix, no hardcoded fallback)
V32_s3Bucket_api_accessKeyId / V32_s3Bucket_api_secretAccessKey	s3Bucket_api_accessKeyId/secretAccessKey (both containers) + S3_ACCESS_KEY_ID/S3_SECRET_ACCESS_KEY (backend only, #3562 fix)	Wasabi S3 client, both v32-prod and s3-v32-prod
V32_s3Bucket_endpoint / V32_s3Bucket_BUCKET_NAME	same names	Wasabi S3 client config, s3-v32-prod only
V32_REDIS_HOST / V32_REDIS_PORT / V32_REDIS_PASSWORD	same names	persistent-patches/crudService.js
V32_LOGDNA_KEY	LOGDNA_KEY	persistent-patches/crudService.js (Mezmo/LogDNA app logging)
V32_CONTAINER_NAME / V32_NODE_ENV	CONTAINER_NAME / NODE_ENV	ecosystem.config.js (PM2 log paths) + env.config.js (which config block gets selected)
V32_VIRTUAL_HOST / V32_LETSENCRYPT_HOST	same names	nginx-proxy auto-discovery / Let’s Encrypt cert issuance

The single most important invariant going forward: V32_MONGODB_URL, V32_PMONGODB_URL, V32_DB_URI, and V32_PINBOX_PROD_DB must all carry the same w3_app@w3_db password at all times. scripts/sops-reset-mongodb.ps1’s production rotation path already groups all four under one user@db key so a single rotation updates all four atomically — the drift this incident hit came from manual/live fixes made outside that script, not from the script itself.

PRs and issues from this incident

• #3609 — sync w3_app password rotation to main (fixed SOPS/live drift)
• #3612 — align V32_PMONGODB_URL to V32_MONGODB_URL (root cause fix)
• #3613 — persist Twilio/OneSignal/mongoose-retry patches via persistent-patches/
• 3607 — node-redis v5 API compatibility (parallel session)
• #3611 — tracking issue for the mongoose/langs bug, resolved and closed
• #3614 — security follow-up (credential exposure in diagnostic output), rotation pending next deploy

Follow-ups / not yet done

1. secrets-sync.yml still has narrow, hand-maintained key lists per file. It should push every V32_*/V42_* key generically (prefix-stripped) rather than an explicit allowlist — this incident repeatedly hit “key X was never in the managed set” gaps (CONTAINER_NAME, NODE_ENV, LOGDNA_KEY, REDIS_*, VIRTUAL_HOST, LETSENCRYPT_HOST, s3Bucket_endpoint, s3Bucket_BUCKET_NAME). Tracked as a todo for this session; not yet merged.
2. GitLab CI’s checkout/clean still wipes backend-environment.env/s3-environment.env on every deploy. The permanent fix (proposed mid-incident, not yet implemented): add a step at the end of GitLab’s prod-back-end-deploy job that re-triggers secrets-sync.yml automatically, so every CI-driven deploy is immediately followed by a full env resync rather than relying on someone noticing the gap.
3. w3_app MongoDB password rotation (from the #3614 exposure) — deferred to the next full deployment per explicit direction.
4. bms-servers.env.sops’s stale mongodb_w3_app_password — superseded by the per-system V32_MONGODB_W3_APP_PASSWORD, cleanup not yet done.
5. Consider whether mongojs (used in ~60 files) and mongoose (used in ~10) should eventually be consolidated onto one client — ruled out of scope for this incident per standing guidance (“fixujemy bez refaktoringu”), but the two-separate-connections structure is what made the V32_PMONGODB_URL drift possible in the first place.