p24-infra Docs

w3-connection-handling-and-rotation

6 min read

Pinbox24 W3 (v32-prod) — MongoDB connection handling & rotation implications

Investigated: 2026-07-09 (read-only SSH source review, sys-admin role — no server modifications) Source of truth: deployed W3 app source in the build dir on bms-1 /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/app-backend/. Code is baked into the container image (563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod), not bind-mounted — the container only mounts /var/log. The build dir reflects the last build/push, not instantaneous live edits.

Path correction (#4985): the original 2026-07-09 review read the root-owned /root/builds/pn3C9eHo/0/pinbox24/p24-v-3.2/ copy and assumed it matched secrets-sync.yml. It does not — that directory is not the prod serving copy (though it is not abandoned either: an on-server audit on 2026-08-01 found it is an active W3-staging mount source, so do not delete it). The authoritative serving dir (per the sync-pinbox24-w3 job in secrets-sync.yml + docs/bms-1/gitlab-ci-w3.yml → BUILDER_DIR) is the GitLab-runner-owned /home/gitlab-runner/builds/eZQeLfuJe/... path above. Confirm live with docker inspect v32-prod --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'.

Origin: GitHub issue #3564 (follow-up to #3449). This is a documentation artifact of a read-only investigation — no files, containers, or credentials were modified on bms-1. The related credential-exposure incident is tracked separately on #3562 (do not duplicate rotation work here).

Companion doc: high-level container/env inventory lives in ../pinbox24-w3-w4-architecture-spec.md §2 (W3 microservices) and §8 (env keys). This doc is the connection-layer deep dive behind that table.

1. Two independent MongoDB connection paths

W3’s v32-prod (main Node app, app-backend/) opens two separate, independent MongoDB connections. Both are env-var-driven and both are cached once at process startup with no dynamic reconnect.

Path APath B
Moduleconfig/mongo_db.jsconfig/mongoose.js
Driverraw mongojsMongoose ODM
Consumers~70 controllers/helpers via .get()any model/*.js (require('mongoose'))
Env varMONGODB_URL (see §3 for the fallback)PMONGODB_URL
Establishednon-blocking from app.js (startup does not wait on the connect callback)required once at module load, before route registration
On error.get() still returns the cached state.db handle regardless of whether auth succeeded — silent failure, no crash, no retryhandler logs and calls mongoose.disconnect() — no retry, no process exit; stays disconnected for the life of the process (no reconnect timer)

Key consequence: the two paths fail independently. A credential problem that only affects the Mongoose/PMONGODB_URL path (Path B) leaves Path B routes permanently broken while all ~70 Path A controllers keep working — the container stays “up” and mostly functional. This is the exact shape of the “langs not loading” incident (§4).

The sibling file microservice s3-v32-prod is a third, separate Mongoose connection via DB_URI (file metadata only) — see the architecture spec §2.2.

2. Env-var read timing — rotation requires a container restart

props.js requires config/env.config.js exactly once at require() time and caches the resolved values as static exports. Both connection paths read their URLs from that cache.

Confirmed by code: a rotated MongoDB credential does NOT take effect until the container is restarted. The URL is never re-read per request or on any interval. This validates the assumption already baked into secrets-sync.yml’s force-recreate-container step for W3 — no change needed there.

3. Structural landmine — the config.local fallback

config/env.config.js’s resolver is effectively:

config[env] || config.local

If process.env.NODE_ENV ever fails to exactly match "production" (e.g. CRLF/whitespace corruption — a known recurring failure mode in this ecosystem), the app silently falls through to a hardcoded local credential block instead of erroring.

  • NODE_ENV=production was confirmed clean in the deployed env file at investigation time, so this path is not currently active.
  • The local block is exactly the hardcoded-and-exposed credential material flagged on #3562.
  • This is documented for awareness only; the app source is outside p24-infra’s control (we cannot patch it). See §6 recommendation 3.

4. “Langs not loading” — confirmed root cause

GET /api/i18n/langs runs through Path B (Mongoose / PMONGODB_URL), not Path A. Because Mongoose connects once at boot with no retry (§1), a bad/stale PMONGODB_URL leaves /api/i18n/langs — and any other Mongoose-model route — permanently broken for the life of the container, while the ~70 mongojs/MONGODB_URL controllers keep serving. That precisely matches the observed symptom “container up, langs specifically broken” — it is structural, not coincidental.

Independent confirmation (non-secret, git-tracked): .github/workflows/secrets-sync.yml carries a comment dated to the incident window (2026-07-09):

“PINBOX_PROD_DB: env.config.js’s hardcoded ‘production’ block falls back to process.env.PINBOX_PROD_DB (NOT MONGODB_URL) for the mongojs-based connection used by /api/i18n/langs — see incident 2026-07-09 W3 langs-hang. Must stay in sync with MONGODB_URL until env.config.js itself is fixed to drop the fallback.”

This surfaces a third Mongo-URL SOPS variable, V32_PINBOX_PROD_DB (distinct from V32_MONGODB_URL / V32_PMONGODB_URL / V32_DB_URI), confirmed present in secrets/pinbox24-w3.env.sops. Per the workflow comment, V32_PINBOX_PROD_DB and V32_MONGODB_URL share the same underlying MongoDB user (w3_app@w3_db) and must stay in sync.

5. Rotation-script gap — the actionable finding

Two independent facts combine into a latent regression:

  1. Commit 94a787ac (“sync stale V32_MONGODB_URL/PMONGODB_URL/DB_URI in pinbox24-w3.env.sops to live bms-1 values”) explicitly states “No other keys were added, removed, or changed” — so V32_PINBOX_PROD_DB was not touched by that fix.
  2. scripts/sops-reset-mongodb.ps1’s W3 key group ($uriKeyDefs, ~lines 262–266) lists only V32_MONGODB_URL, V32_PMONGODB_URL, V32_DB_URI — V32_PINBOX_PROD_DB is absent from the script entirely.

Consequence: every future run of sops-reset-mongodb.ps1 for W3 rotates the three listed keys (which share the one w3_app@w3_db credential) while leaving V32_PINBOX_PROD_DB on the old password — reproducing the “langs not loading” incident on every subsequent rotation, even after a manual fix.

6. Recommendations (not implemented here — read-only investigation)

  1. Add V32_PINBOX_PROD_DB to scripts/sops-reset-mongodb.ps1’s W3 $uriKeyDefs list, grouped under the same w3_app@w3_db credential so it rotates in the same pass as V32_MONGODB_URL / V32_PMONGODB_URL / V32_DB_URI. (Rotation-script change — secret-manager scope; file a secret-manager/dev issue to action.)
  2. Strengthen the W3 post-rotation health check. secrets-sync.yml’s current check (~lines 1733–1739) only greps the last 50 PM2 stderr lines for the substring "authentication fail" — a weak, indirect signal. Add a direct GET /api/i18n/langs (via api.w3.pinbox24.com/api/i18n/langs) HTTP 200 check as a positive functional signal, since that endpoint exercises the Mongoose/Path B connection that actually failed in the incident.
  3. Flag the config[env] || config.local fallback (§3) to the app owner as worth removing — outside p24-infra’s control since we cannot patch the app source, but worth a documented ask if/when source access is restored.
  • #3449 — Twilio / OneSignal / Jabber / Mailgun usage investigation (same source review). Short version: Mailgun live (core auth flow); Jabber dead (calls commented out); Twilio + OneSignal wired into a MongoDB-process-definition-driven task dispatcher (reachable in principle, not dead code) — real usage frequency needs a narrow follow-up MongoDB query. Full write-up on #3449.
  • #3562 — W3 hardcoded-credential exposure incident (the local block from §3). Rotation is blocked pending explicit authorization for production credential rotation + container restart — tracked there, not here.
  • docs/pinbox24-w3-w4-architecture-spec.md — container/env inventory.
  • docs/evaluation/04-pinbox24-map-dr-audit.md — DR audit (notes the GitLab source-repo location as an undocumented gap; a .git checkout exists in the W3 build dir if source history is ever needed).

Graph View

Backlinks