Playbook: v42-prod GUS WSDL Crash-Loop — Unhandled Rejection, NOT Memory

Trigger: PM2 Plus dashboard shows v42-prod_backend restart counter climbing rapidly (uptime resets every 5-30s) while CPU/memory stay within normal bounds — this is the tell that distinguishes it from v42-prod-memory-leak.md, where memory visibly grows toward the max_memory_restart ceiling before each restart. Host of record: bms-1 (94.23.26.113, OVH ns367522) — Pinbox24 v42-prod (w4.pinbox24.com). First documented: 2026-07-08, issue #3407. Time budget: P1 — mitigate within 15 min.

1. Symptom pattern (how to tell this apart from the memory-leak playbook)

Signal	Memory leak (`v42-prod-memory-leak.md`)	GUS WSDL crash-loop (this doc)
Memory before restart	Grows to 90%+ heap / near `max_memory_restart`	Stays well under threshold (e.g. 260-360 MB vs 512M cap)
Trigger correlation	Time-based (grows over minutes)	Volume-based (spikes with inbound Mailgun batch size)
`pm2 logs --err`	Heap warnings, GC pressure	`UnhandledPromiseRejectionWarning: Error: ENOENT ... gus-api-regon ... .wsdl`
Fix	Raise `max_memory_restart`, find leak source	Stop Node crashing on the specific rejection

If raising max_memory_restart does not slow the restart rate, stop assuming it’s memory — check for this failure mode instead.

2. Confirm

ssh root@94.23.26.113 'docker exec v42-prod pm2 list'
# Restart count (↺ column) climbing across repeated checks a few seconds apart, uptime resetting
 
ssh root@94.23.26.113 'docker exec v42-prod pm2 logs v42-prod_backend --lines 200 --nostream 2>&1 | grep -v "mongodb://" | grep -iE "UnhandledPromiseRejectionWarning|gus-api-regon|ENOENT"'
# Look for: Error: ENOENT ... @pobidowski/gus-api-regon/dist/wsdl/...wsdl
 
ssh root@94.23.26.113 'docker exec v42-prod node --version'
# If >= v15: unhandled promise rejections are FATAL by default (this is why it crashes, not just warns)
 
ssh root@94.23.26.113 'docker exec v42-prod ls -la /app/node_modules/@pobidowski/gus-api-regon/dist/wsdl/'
# Expected on a broken image: "No such file or directory" — the whole wsdl/ dir is missing

CAUTION — same mongodb password leak risk as the memory-leak playbook (issue #2970): always pipe pm2 logs --err through grep -v 'mongodb://'.

3. Root cause

Correction (2026-07-08, after deeper investigation): initial analysis attributed the trigger to the “doc process with ai” workflow step doing a per-document GUS lookup. That was wrong. The actual call site is dead debug/test code at the top level of app.js, outside any workflow, that runs unconditionally on every process startup:

const service = new gus_api_regon_1.GusApiRegon("abcde12345abcde12345", true); // fake/placeholder key
const main = () => __awaiter(void 0, void 0, void 0, function* () {
    const results = yield service.search({ Nip: "2040000177" }); // hardcoded test NIP
});
main();

It was never wired to any real feature or Mailgun-triggered code path. The @pobidowski/gus-api-regon npm package is missing its dist/wsdl/*.wsdl assets in the deployed v42-prod image (likely stripped by a .dockerignore or npm prune/build step that only expects JS files), so this startup call always throws ENOENT. Because the container runs Node 15+, an unhandled promise rejection is fatal (crashes the process instead of just warning, which was the pre-v15 behavior) — PM2 immediately restarts, the fresh process hits the exact same startup code path, and it crashes again. This is a startup-time loop, not a per-document one — high Mailgun volume didn’t directly drive the crash rate, though it made the backlog/impact worse each time the process was down.

There was also a stale bind-mount already in docker-compose.yml (persistent-patches/gus-api-regon-wsdl → /app/dist/wsdl) from an earlier, incomplete attempt to fix this by supplying the WSDL — but the actual require() resolves from /app/node_modules/@pobidowski/gus-api-regon/dist/wsdl/, so that mount never took effect.

This is not a memory leak — confirmed 2026-07-08 by raising max_memory_restart to 512M with zero effect on restart rate.

4. Immediate mitigation (stops the crash loop, zero-downtime-ish — one brief restart)

# Step 1: add --unhandled-rejections=warn to node_args in the host-mounted ecosystem config
# (avoid nested double-quotes over SSH — match the unique substring and append via & backreference)
ssh root@94.23.26.113 "sed -i 's/--max-old-space-size=512/&  --unhandled-rejections=warn/' /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ecosystem.config.js"
 
# Verify
ssh root@94.23.26.113 "grep node_args /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ecosystem.config.js"
# Expected: node_args: "--max-old-space-size=512  --unhandled-rejections=warn"
 
# Step 2: restart to pick up new node_args (pm2 reload does NOT change spawn args — use restart)
ssh root@94.23.26.113 "docker exec v42-prod pm2 restart v42-prod_backend --update-env"
 
# Step 3: persist
ssh root@94.23.26.113 "docker exec v42-prod pm2 save"
 
# Step 4: verify restart counter freezes (check pm2 list twice, a minute apart)
ssh root@94.23.26.113 "docker exec v42-prod pm2 list"

Effect: GUS/company-lookup failures now log as warnings instead of crashing the process. Kept in place as a fleet-wide safety net after the real fix below, but by itself it did not stop the loop — see §5, the crash was startup-time, not per-request, so this alone only slowed symptom severity.

5. Permanent fix applied (2026-07-08, live on bms-1)

The real fix: the GUS call was dead debug/test code, never wired to any real feature (see §3). It was removed from the running container using this repo’s standard runtime-patch pattern:

# Step 1: copy the current app.js out of the container to the host patch dir
ssh root@94.23.26.113 "docker cp v42-prod:/app/dist/app.js /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/persistent-patches/app.js"
 
# Step 2: comment out the dead block (lines 278-284 at time of fix — confirm with:
#   docker exec v42-prod grep -n 'main();' /app/dist/app.js
# and grep -n 'gus-api-regon' /app/dist/app.js  — adjust line range if the source has since changed)
ssh root@94.23.26.113 "sed -i '278,284 s/^/\/\/ DISABLED (issue #3407, dead test code): /' /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/persistent-patches/app.js"
 
# Step 3: add a persistent bind mount in docker-compose.yml (same directory as ecosystem.config.js)
ssh root@94.23.26.113 "sed -i '/ecosystem.config.js:\/app\/ecosystem.config.js:ro/a\      - /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/persistent-patches/app.js:/app/dist/app.js:ro' /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/docker-compose.yml"
 
# Step 4: recreate the container to pick up the new bind mount (CONTAINER_NAME/IMAGE_NAME must be
# passed explicitly — the compose file uses $CONTAINER_NAME/$IMAGE_NAME vars with no .env default)
ssh root@94.23.26.113 "cd /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts && CONTAINER_NAME=v42-prod IMAGE_NAME=563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod:latest docker-compose up -d --no-build --force-recreate backend"
 
# Verify: restart counter is 0 on fresh boot, no GUS/WSDL mentions anywhere in logs
ssh root@94.23.26.113 "docker exec v42-prod pm2 list"
ssh root@94.23.26.113 "docker exec v42-prod pm2 logs v42-prod_backend --lines 60 --nostream 2>&1 | grep -v 'mongodb://' | grep -iE 'gus|wsdl'"
# Expected: pm2 list shows ↺ 0; grep finds nothing (exit code 1)

Update 2026-07-09: a later redeploy (by a concurrent worker session) reset docker-compose.yml and moved the crash mitigation from ecosystem.config.js’s node_args into a proper environment: [NODE_OPTIONS=--unhandled-rejections=warn] block — a cleaner placement. That redeploy also dropped the persistent-patches/app.js bind mount from §5 (the dead-code-removal patch), so the running container is back to the image’s original unpatched app.js — but it stays stable because the env-level NODE_OPTIONS still catches the crash. Functionally equivalent, no action required; confirmed via docker exec v42-prod pm2 env 0 | grep NODE_OPTIONS (present) and docker exec v42-prod grep -c 'DISABLED 2026-07-08' /app/dist/app.js (0 — patch not present, as expected). The stale gus-api-regon-wsdl bind mount was removed from docker-compose.yml on 2026-07-09 (harmless either way, now cleaned up — takes effect on next container recreate).

Update 2026-07-12 — the dead-code-removal patch is now committed to this repo (not just SSH-applied). Confirmed the “no action required” state from the note above had regressed: the GUS ENOENT/unhandled rejection was firing again on every startup (found during unrelated W4 cluster-mode testing), because the §5 patch only ever existed as a manual edit on bms-1 — infra-src/pinbox24/w4/persistent-patches/ in this repo never actually carried it, so the next secrets-sync deploy from the (unpatched) infra-src source silently brought the dead code back, exactly as this doc’s own “Remaining cleanup” section warned it would. Re-applied properly this time: infra-src/pinbox24/w4/persistent-patches/app.js now ships the same lines-278-284 comment-out committed in git, wired via docker-compose.yml, so it survives every future redeploy instead of only the current container. Also removed the stale (and never functional) gus-api-regon-wsdl bind mount in the same change — confirmed unused now that the calling code is disabled. NODE_OPTIONS=--unhandled-rejections=warn is left in place as a fleet-wide safety net per the original recommendation.

Remaining cleanup (low priority, not urgent):

Land the same removal in the GitLab p4-back-ts source directly, so a future image rebuild (not just a docker-compose recreate) doesn’t need the persistent-patch to stay clean
Re-evaluate whether --unhandled-rejections=warn should remain permanently across Node 15+ services in this fleet (safety net) or be removed now that this specific bug is fixed — a global “never crash on unhandled rejection” setting also hides other bugs that used to be loud

Track via issue #3407.

References

Issue #3407 — this incident (2026-07-08, P1)
docs/playbooks/v42-prod-memory-leak.md — related but distinct restart-loop failure mode (actual memory growth)
docs/playbooks/pinbox24-mailgun-duplicate-check-fix.md — same inbound Mailgun/P24-WF pipeline, different bug (#2048)
docs/pinbox24/incidents/v42-prod-gus-wsdl-crash-2026-07-08.log — captured log evidence (secrets stripped)

p24-infra Docs

Explorer

v42-prod-gus-wsdl-crash-loop