p24-infra Docs

v32-prod-mongoose-authfail-storm

6 min read

Playbook: v32-prod Mongoose auth-fail / reconnect storm after a W3 rotation

Trigger: Alert MongooseDisconnect on container v32-prod (e.g. “75 in 5m”), usually alongside EndpointDown / EndpointSlow on https://api.w3.pinbox24.com/*. First seen: #3908 (2026-07-11), same day as the W3 MongoDB rotation + local-redis-v32 cutover (#3712).

One-line cause: the running v32-prod backend on bms-1 is deployed with stale W3 credentials (MongoDB w3_app password and/or Redis host+password) that predate the latest rotation/cutover. SOPS is correct; bms-1 is the stale side — it was never recreated with the new env.

Agents cannot SSH bms-1 (root key is human-workstation only, #1270). Diagnose from Mezmo + rs0, fix via CI.

Diagnose (from bms-4 / vps-i1 — no bms-1 SSH needed)

  1. Confirm the log signatures (Mezmo, last 1h):

    • app:pm2_v32-prod_production_err.log → many Mongoose: MongoNetworkError … [MongoError: Authentication failed] to 145.239.133.104:27017 (rs0 PRIMARY).
    • Often also Redis config error … WRONGPASS and/or ECONNREFUSED <old-external-redis>:<port> (the decommissioned OVH-managed Redis — means the backend still points at the pre-cutover host).
    • Query: "Mongoose disconnected" host:bms-1 — a sustained burst (≫ background of a few/hour) = a real storm.

  2. Rule out rs0 (live from bms-4 arbiter — password from secrets/bms-servers.env.sops key mongodb_rs0_admin_password, never printed):

    RS_PW=$(sops -d --input-type dotenv --output-type dotenv secrets/bms-servers.env.sops | grep '^mongodb_rs0_admin_password=' | cut -d= -f2- | tr -d '"')
mongosh "mongodb://admin:${RS_PW}@127.0.0.1:27017/admin?authSource=admin" --quiet --eval 'rs.status().members.forEach(m=>print(m.name,m.stateStr,m.health))'; unset RS_PW

    Healthy PRIMARY/SECONDARY/ARBITER with no recent election → the storm is app/credential-side, not rs0.

  3. Confirm SOPS is the correct side — the w3_app/w3_db credential in secrets/pinbox24-w3.env.sops (V32_MONGODB_URL) should authenticate against rs0 primary (145.239.133.104, authSource=w3_db). If it does, bms-1 is stale → go to Fix. If SOPS also fails, the rs0 user password itself was changed and not propagated → this is a secret-manager task (rotate/repropagate), not a redeploy.

Fix (operator — one CI command; recreates the stack from current SOPS)

gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-w3

sync-pinbox24-w3 installs /opt/p24-infra/bms-1/pinbox24-w3.env, rewrites backend-environment.env (V32_* → stripped) + redis-v32-environment.env, then docker-compose up -d --force-recreate backend s3 reso socket + recreates redis-v32, and self-validates authentication fail == 0 (the job fails if auth is still broken). Safe to re-run.

⚠️ It is a production recreate (~30–60s W3 blip). If a rotation/cutover session may be in flight on W3, confirm first — concurrent deploys on this stack collide (see #3908 context).

Variant — rotation done on rs0 + SOPS, but the v32-prod backend container was skipped (#3958)

Seen on 2026-07-12 (#3958), the day after 3911. The w3_app password rotation completed on rs0 PRIMARY and a fix branch, yet MongooseDisconnect re-fires on container=v32-prod with GET /api/i18n/langs → 422. Distinguishing signature vs the main case:

  • rs0 healthy; the new w3_app password authenticates from the rotation PR branch SOPS, but the main SOPS value still fails auth against rs0 (the PR is unmerged / CONFLICTING).
  • s3/reso/socket were recreated with the new cred, but the backend (v32-prod) container was not — it lives in the separate /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2 GitLab-CI compose project (#3892), so a secrets-sync recreate hits a container-name conflict and skips it, leaving it on the old w3_app password. The pm2_v32-prod_* log basename is what the mezmo-exporter labels container=v32-prod, so those disconnects come from the backend itself.

Do NOT run the one-CI-command Fix above in this state: main SOPS holds the OLD password until the rotation PR merges — a sync would deploy the stale cred. And a branch-dispatch sync still skips the backend (same name conflict). The last mile is a human / secret-manager action:

  1. Rebase + resolve conflicts on the rotation PR, merge to main (lands the new w3_app in SOPS).
  2. Recreate the v32-prod backend on bms-1 in the /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2 project — docker-compose up -d --force-recreate --no-deps backend, preserving prod-v-3-net + test-net (per the #3892 pattern; not /root/recreate_v32_3714.sh). Needs bms-1 SSH (agents lack it, #1270).

Quick triage query (from bms-4, no bms-1 SSH): auth-test the V32_MONGODB_URL w3_app password from both main and the rotation PR branch against rs0 — if main fails but the PR branch passes, you are in this variant (fix = merge PR + recreate backend), not the main case (fix = one CI command).

Variant 2 — SOPS is the stale side: NO recorded w3_app password authenticates against rs0 (#3996)

Seen 2026-07-12 (#3996), ~4.5h after the #3911 rotation + #3958 both closed. MongooseDisconnect re-fired on container=v32-prod (110/5m) but the live W3 API still returns 200 fast — so it is reconnection churn, not a full outage. Distinguishing signature vs the main case and vs #3958:

  • rs0 healthy; every recorded w3_app password fails auth against rs0 PRIMARY (w3_db): main SOPS V32_MONGODB_URL, main SOPS V32_MONGODB_W3_APP_PASSWORD (which itself differs from the URI-embedded value — internal SOPS drift), and all rotation branches (fix/3911-w3app-mongodb-rotation, fix/3911-w3app-rotation-v2, security/3614-w3-app-final-rotation, fix/3171-…-live-sync). Confirm the test harness with the rs0 admin cred via the same -u/-p path — if admin passes and every w3_app value fails, the failures are real.
  • Because the API is 200, the running backend on bms-1 holds a valid w3_app password that exists only in bms-1’s live env and was never captured back into SOPS.

➡️ This is the inverse of the main case: SOPS (both keys) is the stale/desynced side, not bms-1.

Do NOT run the one-CI-command Fix (secrets-sync -f target=pinbox24-w3) in this state — it would push the wrong main SOPS password to bms-1 and break the currently-working backend, turning a warning-level churn into a full W3 outage. Reconcile SOPS to the live value first. This is a human-supervised secret-manager action (bms-1 SSH is human-only, #1270):

  1. Capture the live working w3_app password from bms-1 (/opt/p24-infra/bms-1/*.env / the /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2 backend env) — the only source of the valid value.
  2. Verify it authenticates against rs0 (w3_db), then write it into secrets/pinbox24-w3.env.sops reconciling all carriers: V32_MONGODB_URL (URL-encoded), V32_MONGODB_W3_APP_PASSWORD, and V32_DB_URI/V32_PMONGODB_URL if they embed it.
  3. Or, if the live value is unrecoverable: rotate fresh — db.changeUserPassword('w3_app', …) on rs0 PRIMARY simultaneously with recreating the v32-prod backend on bms-1 (in /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2, --force-recreate --no-deps backend, preserving prod-v-3-net+test-net) so W3 is never left on a password rs0 rejects.
  4. Only after reconciliation is secrets-sync -f target=pinbox24-w3 safe again.

Quick triage (from bms-4, no bms-1 SSH): auth-test the w3_app value from main and every open rotation branch against rs0. If main fails AND all branches fail AND the API is 200, you are in this variant (SOPS desynced from a live-only cred) — not the main case, not #3958.

Verify recovery

  • Mezmo "Mongoose disconnected" host:bms-1 drops back to baseline (a few/hour, not tens/min).
  • curl -s -o /dev/null -w '%{http_code} %{time_total}s' https://api.w3.pinbox24.com/api/i18n/langs → 200, fast.
  • Close the MongooseDisconnect alert issue and any paired EndpointDown/EndpointSlow on api.w3.
  • mongotimeout-false-positive.md — genuine vs self-ref MongoTimeout issues (agents can’t SSH bms-1).
  • pinbox24-bms1-redis-wrongpass.md — the Redis WRONGPASS side.
  • w3-mongodb-credential-rotation.md, w3-w4-rotation-orchestration.md — the rotation flow that, if it doesn’t recreate the running backend, leaves this stale state.

Graph View

Backlinks