p24-infra Docs

incident-2026-07-08-w3-wasabi-mongo-cascading-regression

5 min read

Incident: W3 cascading regression during Wasabi key fix (2026-07-08)

Severity: P1 (W3 Excel import + file upload both broken for a window) Window: ~14:26 UTC – ~14:50 UTC (upload), fully resolved ~14:50 UTC Reported by: user, via W3 Excel import InvalidAccessKeyId error

Timeline

  1. Original report: W3 RESO Excel import failing with InvalidAccessKeyId on /api/reg-process/.../5.
  2. Root cause found: secrets-sync.yml’s sync-pinbox24-w3 job only ever propagated the Wasabi S3 key to s3-environment.env, never to backend-environment.env — a split-brain left over from issue #3226 (which populated SOPS with the correct key but the sync job’s gap meant v32-prod never received it).
  3. Fix: PR #3410 — added the missing update_key(back_env, ...) calls. Merged, then manually dispatched secrets-sync.yml -f target=pinbox24-w3 (scoped to W3 only) to deploy immediately.
  4. Regression #1: That same dispatch re-ran the entire W3 sync job — including unconditionally overwriting MONGODB_URL/PMONGODB_URL in backend-environment.env from SOPS. SOPS held a stale admin MongoDB password (a prior incident, INCIDENT-2026-07-08-002, had patched the live file directly without updating SOPS — see docs/secrets-rotation-log.md). Result: v32-prod AuthenticationFailed against MongoDB, /api/i18n/langs broken (“langs not loading”).
  5. Fix attempt via prepared script: Located scripts/rotate/w3_app_mongodb_password.sh (designed to rotate a dedicated w3_app MongoDB user and migrate v32-prod off the shared admin account — the proper long-term fix for #3060 / INCIDENT-2026-07-08-002).
    • First run failed at find-primary: claude-runner on bms-4 had no IdentityFile mapping for bms-1/bms-2/bms-3 in ~/.ssh/config (only vps-i1 was configured) — fixed by adding the missing host entries (key was already trusted server-side, just not offered locally).
    • Bug found: the script’s mongosh payload starts with use w3_db — a mongosh REPL shell-helper, not valid JavaScript. In mongosh --file (non-interactive) mode this silently aborts the script with no output and exit 0 — every subsequent statement, including changeUserPassword(), never runs. Same bug present in the sibling w4_app_mongodb_password.sh. Fixed via PR #3411 (merged): replaced with db.getSiblingDB('w3_db').changeUserPassword(...).
    • Second run (corrected script) succeeded end-to-end: rotated w3_app’s password, updated both MONGODB_URL/PMONGODB_URL in backend-environment.env, force-recreated v32-prod, verified /api/i18n/langs HTTP 200.
  6. Regression #2: The rotation script only updates backend-environment.env (used by v32-prod). It does not touch s3-environment.env’s DB_URI, which is also a w3_app connection (used by s3-v32-prod). That container had been force-recreated earlier (step 3’s secrets-sync run) at 14:27 UTC with the old w3_app password baked into its Mongoose connection. Once the password was rotated at 14:47, s3-v32-prod’s stale credential caused file uploads to hang — POST /api/files/.../upload → 504 Gateway Timeout.
  7. Fix: copied the new w3_app password from backend-environment.env’s MONGODB_URL into s3-environment.env’s DB_URI directly on bms-1 (user-authorized emergency SSH fix, backed up first), force-recreated s3-v32-prod. Upload confirmed working by user.
  8. Secondary bug found: sops-sync-receiver.yml (the workflow the rotation script dispatches to persist the new password into secrets/bms-servers.env.sops) failed: canary decrypt failed after encrypt — NOT committing: invalid dotenv input line: {. Failed safely (did not corrupt SOPS), but the new password was never persisted to SOPS. Needs investigation — likely a payload-encoding bug in scripts/sops-sync-receiver.py.

Current state (end of incident)

  • W3 Excel import: fixed (Wasabi key, PR #3410, live + SOPS both correct)
  • W3 file upload: fixed (MongoDB w3_app password rotated + propagated to both v32-prod and s3-v32-prod, live)
  • W3 login (/api/i18n/langs): fixed (same rotation)
  • secrets/bms-servers.env.sops W3_APP_MONGODB_PASSWORD: NOT updated — sops-sync-receiver.yml failed (see bug above)
  • secrets/pinbox24-w3.env.sops V32_MONGODB_URL/V32_PMONGODB_URL/V32_DB_URI: NOT updated — still hold pre-incident stale values. This means the next secrets-sync run for pinbox24-w3 (manual or automatic) will silently revert both fixes. User explicitly deferred this SOPS write to a follow-up P0 task (see below) rather than doing it ad hoc.

Root causes (all confirmed, not speculative)

  1. secrets-sync.yml’s W3 job propagates different key sets to different files inconsistently (hand-rolled per-file update_key() calls instead of the generic update_env() the W4 job uses) — partially fixed (PR #3410), but the underlying pattern risk remains for any future key.
  2. SOPS was allowed to drift from live server state on two separate prior incidents (INCIDENT-2026-07-08-002 MongoDB password, and the original Wasabi key gap from #3226) without the drift being corrected in SOPS — meaning any later automated sync silently reverts the manual fix.
  3. mongosh --file silently no-ops on REPL-only syntax (use <db>) with exit 0 and zero output — a genuinely surprising footgun that made two “tested” rotation scripts non-functional when run non-interactively. Nothing in their test plans caught this because dry-run mode exits before reaching that code path.
  4. No single script/process rotates all of W3’s interdependent secrets (Wasabi key × 2 files, MongoDB × 2 files/2 users) atomically — each rotation touches only part of the surface, so partial rotations reliably produce split-brain states like this one.
  5. claude-runner’s SSH config on bms-4 was incomplete (only vps-i1 mapped), silently breaking any script that needs to reach bms-1/2/3 from that host.

Follow-up work

Tracked in a dedicated P0 issue (due 2026-07-08 20:00 UTC): SOPS durability write, E2E rebuild test suite, comprehensive W3 secret rotation script, W3/W4 SOPS file separation (builds on existing issue #2731).

A separate night-task issue queues an audit of all other scripts/rotate/*.sh for the same use <db> pattern and other latent non-interactive-execution bugs.

Graph View