Playbook: v42-prod Node.js Memory Leak — PM2 Restart Loop on bms-1

Trigger: Pinbox24BackendRestartLoop Prometheus alert (>5 PM2 restarts in 10m), or V42ProdPM2MemoryWarning (container working-set > 1024 MiB), or manual report of continuous PM2 cycling.
Host of record: bms-1 (94.23.26.113, OVH ns367522) — Pinbox24 production.
First documented: 2026-07-02, issue #2599 — 98+ PM2 restarts, processes cycling every 3–5 min.
Time budget: P0 — mitigate within 15 min, investigate within 2h.

1. Symptom pattern

pm2 list shows restart counter climbing (≥5 new restarts in 10 min)
Processes grow from ~77 MB to 230+ MB per worker in under 2 minutes post-restart
PM2 max_memory_restart triggers (280M original, now 512M after #2599 fix)
Event Loop Latency p95 spikes to 2000+ ms during high-memory phase
Container itself stays up (0 Docker-level restarts) — only Node.js workers cycle
Heap usage reported at 94–95% of heap size immediately before restart

1a. Triage: is this the leak, or a benign `Pinbox24HeapUsageHigh` ratio alert?

The newer Pinbox24HeapUsageHigh alert (#4014) fires on pm2_heap_usage_ratio ≥ 0.85 for 10 min — i.e. heapUsed / heapTotal. A high ratio alone is NOT a leak. V8 keeps heapTotal sized just above heapUsed whenever the heap is well below its ceiling, so a 90–96% ratio is the normal steady-state for these processes and is stack-wide (v42-prod and v32-prod both sit ~90–96%).

Decide which condition you have before acting:

Signal	Benign ratio alert (no action)	Real leak (mitigate — §3)
PM2 restart counter `↺`	stable (e.g. 1), uptime hours	climbing ≥5 in 10 min, uptime < 5 min
Absolute `heapTotal`	small (~150 MB) — a few % of ceiling	large / approaching `--max-old-space-size`
Container RSS vs limit	far below limit (581 MB, no limit set)	near `max_memory_restart` (512M)
3h heapTotal trend	flat / ~2 MB/hr	steep, MB/min growth

Ceiling check (why the ratio misleads): if --max-old-space-size is not set, V8’s old-space ceiling is ~2 GB. A heapTotal of ~148 MB is only ~7% of that — no OOM path for weeks even at 96%.

ssh root@94.23.26.113 'docker exec v42-prod sh -c "echo \$NODE_OPTIONS"' | grep -oE -- "--max-old-space-size=[0-9]+" || echo "unset → V8 default ~2GB"

Read the real numbers straight from the PM2 exporter (secret-free) instead of eyeballing the ratio:

ssh root@94.23.26.113 'curl -s http://localhost:9256/metrics | grep -E "pm2_heap_(used|total)_bytes|pm2_heap_usage_ratio"'
# and the 3h trend from Prometheus on vps-i1:
ssh root@217.154.82.162 'curl -s -G http://localhost:9090/api/v1/query --data-urlencode "query=pm2_heap_total_bytes{container=\"v42-prod\",pm_id=\"1\"} - (pm2_heap_total_bytes{container=\"v42-prod\",pm_id=\"1\"} offset 3h)"'

If restarts are stable and absolute heap is small and the 3h trend is flat/slow → this is a benign symptom of the slow leak tracked in #2542, not a new incident. No restart is warranted; record findings on the alert issue and move on. First observed via #4080 (2026-07-13): all three v42-prod instances at 90–96% ratio, heapTotal ~148 MB, RSS 581 MB, ~2 MB/hr growth — no action needed.

RESOLVED in code (#4122, 2026-07-13): Pinbox24HeapUsageHigh (and Pinbox24HeapUsageCritical) no longer fire on a bare ratio. Each container arm now carries an absolute-heap floor — pm2_heap_total_bytes > 500 MiB joined per-process via and on(container, app, pm_id) — so the alert pages only when the ratio is high and the heap has actually grown large (the real leak signature). The benign low-absolute-heap steady state described above no longer alerts, so if this playbook is reached today it should be because a genuine leak (heapTotal > 500 MiB) is present. Regression-tested by monitoring/prometheus/rules/tests/pinbox24_heap_floor_test.yml (run in CI by deploy-monitoring-config.yml).

2. Confirm the leak

ssh root@94.23.26.113 'docker exec v42-prod pm2 list 2>&1'
# Look for:  ↺  column > 5 and growing, uptime < 5m per worker
 
ssh root@94.23.26.113 'docker exec v42-prod pm2 show 0 2>&1 | grep -iE "restart|uptime|Used Heap|Heap Usage|Heap Size"'
# If Heap Usage > 90% and restarts > 5, memory leak is active

CAUTION — pm2 logs —err leaks mongodb_w4_app_password (issue #2970, 2026-07-07): Node.js emits [DEP0170] DeprecationWarning: The URL mongodb://w4_app:<password>@... to stderr when the legacy connection string format is used. Running pm2 logs --err will expose the password in plaintext. Always filter: | grep -v 'mongodb://'

3. Immediate mitigation (stops the restart loop, zero downtime)

Run from your workstation (requires ssh root@94.23.26.113 access):

# Step 1: Raise PM2 restart threshold to 512M WITHOUT restarting workers
ssh root@94.23.26.113 'docker exec v42-prod pm2 restart all --max-memory-restart 512M 2>&1'
# NOTE: pm2 restart re-spawns workers. To avoid downtime with cluster mode:
# ssh root@94.23.26.113 'docker exec v42-prod pm2 reload all --max-memory-restart 512M 2>&1'
# (reload does zero-downtime rolling restart for cluster mode)
 
# Step 2: Persist the setting in PM2 dump (survives container restart, not recreate)
ssh root@94.23.26.113 'docker exec v42-prod pm2 save 2>&1'
 
# Step 3: Verify both workers are stable
ssh root@94.23.26.113 'docker exec v42-prod pm2 list 2>&1'
# Expected: status=online, restart count no longer growing, memory 190-250MB

Why this works: PM2’s dump.pm2 stores the overridden max_memory_restart value. The running processes continue without disruption (restart count resets; new threshold applies).

Why this is NOT permanent: On docker compose up --force-recreate, PM2 reads ecosystem.config.js from the image (or volume mount) and ignores dump.pm2. See §5 for the permanent fix.

4. Investigate the leak — heap snapshot

# Step 1: Trigger heap snapshot on process 0 (PM2+ IPC command)
ssh root@94.23.26.113 'docker exec v42-prod pm2 trigger v42-prod_backend km:heapdump 2>&1'
# Expected: "2 processes have received command km:heapdump"
#           "[v42-prod_backend:0:default]={"success":true}"
 
# Step 2: Find the snapshot file (may be in /app, /tmp, /root/.pm2, or /home)
ssh root@94.23.26.113 'docker exec v42-prod find / -name "*.heapsnapshot" -newer /proc/1 2>/dev/null'

If no snapshot file appears (known PM2+ Cloud behaviour): PM2+ may stream the heap snapshot to the PM2 Cloud dashboard (app.pm2.io) rather than writing to disk. In that case:

Log into https://app.pm2.io — look for a snapshot in the v42-prod app
Download and analyse with Chrome DevTools > Memory > Load profile
Sort by “Retained Size” to find the accumulating object type

Alternative: V8 Inspector (requires brief downtime for attaching)

# Stop one worker, restart it in inspect mode
ssh root@94.23.26.113 'docker exec v42-prod pm2 stop 1 2>&1'
ssh root@94.23.26.113 'docker exec v42-prod node --inspect=0.0.0.0:9229 --max-old-space-size=512 /app/dist/server.js 2>&1 &'
# Forward port locally and open chrome://inspect

Known suspects from 2026-07-02 analysis

Candidate	Evidence	Fix location
OfficeCron timer accumulation	`officeCron: skipping null cronTime <ObjectId>` on every startup	v42 src — filter null cronTime before scheduling
Socket.io handle leak	24 active handles after stabilisation	v42 src — ensure socket.io server closes all refs on SIGINT
RabbitMQ channel accumulation	`Connected to RabbitMQ` on every PM2 restart	v42 src — verify channel.close() called on worker exit
Mongoose cursor leak	Duplicate-check queries (issue body)	v42 src — ensure cursors are closed after iteration

5. Permanent fix — ecosystem.config.js bind-mount (no image rebuild needed)

Discovery: v42-prod’s docker-compose.yml already bind-mounts the ecosystem.config.js from the host:

/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ecosystem.config.js  →  /app/ecosystem.config.js  :ro

Editing the host file immediately propagates into the container AND persists across --force-recreate.

Apply the 512M threshold permanently on bms-1:

ssh root@94.23.26.113 'grep max_memory_restart /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ecosystem.config.js'
# Confirm current value: max_memory_restart: "280M" (or "512M" if already applied)
 
ssh root@94.23.26.113 'sed -i "s/max_memory_restart: \"280M\"/max_memory_restart: \"512M\"/" \
  /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ecosystem.config.js'
 
# Verify
ssh root@94.23.26.113 'grep max_memory_restart /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ecosystem.config.js'
# Expected: max_memory_restart: "512M"

The reference copy in p24-infra at bms-1/v42-prod-ecosystem.config.js documents the intended state. On next image rebuild (#2418), this threshold should be baked into the Docker image.

6. Verify stabilisation

# 15 min after mitigation: check restart counter has stopped growing
ssh root@94.23.26.113 'docker exec v42-prod pm2 list 2>&1'
# Expected: ↺ column frozen (e.g. 98 or 99), status=online, mem 190-250MB
 
# 1h after mitigation: memory trending
ssh root@94.23.26.113 'docker exec v42-prod pm2 show 0 2>&1 | grep -iE "mem|heap|restart|uptime"'
# Expected: mem stable or slowly growing but well below 512M, no new restarts

Check Grafana (https://grafana.vps-i1.infra.zintegrowana.online) → Pinbox24 dashboard:

container_memory_working_set_bytes{name="v42-prod"} flat or bounded
pinbox24_pm2_restarts_total counter frozen
V42ProdPM2MemoryWarning alert resolved

7. After incident — hand-off to v42 dev team

Create or update a ticket in the Pinbox24 / p24-back-ts project with:

Symptom: restart count, memory growth rate, timestamp
Suspected leak locations: see §4 known suspects table
Heap snapshot (if retrieved from PM2 Cloud dashboard): attach the .heapsnapshot file
Request: memory profiling in staging with the same workload; fix the top retained-size class

Reference: issue #2418 (v42 image rebuild) should include the 512M ecosystem.config.js threshold as a permanent change in the Dockerfile/build context.

8. Prevention

V42ProdPM2MemoryWarning alert (monitoring/prometheus/rules/pinbox24.yml) fires at 1024 MiB (1 GiB) container working-set — recalibrated from 400 MB in issue #3971 for the 3-worker era (~210 MB/worker healthy baseline; old 400 MB fired continuously at healthy state). Gives warning before PM2 mass-restarts (which need workers near the 512M per-worker kill line). Check Grafana immediately on this alert.
Pinbox24BackendRestartLoop alert fires on >5 restarts in 10 min — this is the “too late but still actionable” backstop.
After any --force-recreate on bms-1: immediately run pm2 list and verify restart counter is 0 and memory is bounded — the host-side ecosystem.config.js fix should prevent recurrence, but verify it survived the recreate.
Long-term: deploy pm2-exporter sidecar on bms-1 for per-process heap metrics (issue #2542).

9. Container inspection — safe commands

WARNING (issue #2970): Never run docker inspect v42-prod without --format. The bare command dumps the full Env section into the terminal, exposing all credentials in plaintext. Always scope with --format to a specific field.

# SAFE: --format scoped to avoid Env section exposure (issue #2970)
 
# Check volume mounts (no env values exposed)
ssh root@94.23.26.113 "docker inspect --format '{{json .Mounts}}' v42-prod | python3 -m json.tool"
 
# Check restart policy
ssh root@94.23.26.113 "docker inspect --format '{{.HostConfig.RestartPolicy.Name}}' v42-prod"
 
# Check bind mounts (confirm ecosystem.config.js is mounted from host)
ssh root@94.23.26.113 "docker inspect --format '{{json .HostConfig.Binds}}' v42-prod"

These are the only docker inspect calls ever needed for memory-leak diagnosis on this container. For any other field, add a scoped --format path — never run bare docker inspect v42-prod or cat / jq the full inspect JSON.

References

Issue #2599 — this incident (2026-07-02, 98 PM2 restarts, P0)
Issue #2418 — v42 image rebuild (permanent fix target)
Issue #2542 — pm2-exporter sidecar (per-process heap metrics)
bms-1/v42-prod-ecosystem.config.js — reference copy with 512M threshold
monitoring/prometheus/rules/pinbox24.yml — V42ProdPM2MemoryWarning + Pinbox24BackendRestartLoop
docs/playbooks/pinbox24-no-logs-bms1.md — related bms-1 log-forwarding incident class

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="v42-prod",
    result="success",  # "success" | "failed" | "skipped"
    detail="v42-prod memory leak resolved — container restarted with memory limits applied",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'v42-prod', 'success', 'v42-prod memory leak resolved — container restarted with memory limits applied', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

v42-prod-memory-leak

Playbook: v42-prod Node.js Memory Leak — PM2 Restart Loop on bms-1

1. Symptom pattern

1a. Triage: is this the leak, or a benign `Pinbox24HeapUsageHigh` ratio alert?

2. Confirm the leak

3. Immediate mitigation (stops the restart loop, zero downtime)

4. Investigate the leak — heap snapshot

Known suspects from 2026-07-02 analysis

5. Permanent fix — ecosystem.config.js bind-mount (no image rebuild needed)

6. Verify stabilisation

7. After incident — hand-off to v42 dev team

8. Prevention

9. Container inspection — safe commands

References

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

v42-prod-memory-leak

Playbook: v42-prod Node.js Memory Leak — PM2 Restart Loop on bms-1

1. Symptom pattern

1a. Triage: is this the leak, or a benign Pinbox24HeapUsageHigh ratio alert?

2. Confirm the leak

3. Immediate mitigation (stops the restart loop, zero downtime)

4. Investigate the leak — heap snapshot

Known suspects from 2026-07-02 analysis

5. Permanent fix — ecosystem.config.js bind-mount (no image rebuild needed)

6. Verify stabilisation

7. After incident — hand-off to v42 dev team

8. Prevention

9. Container inspection — safe commands

References

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

1a. Triage: is this the leak, or a benign `Pinbox24HeapUsageHigh` ratio alert?