p24-infra Docs

n8n-bms4-worker-crash-storm

6 min read

Playbook — bms-4 n8n worker crash-storms (N8nHighFailureRatio)

Issue: #2258 · Server: bms-4 (54.36.123.110) · Service: n8n 2.26.3 queue-mode (main + 3 workers, Postgres + Redis/Bull) · Alert: N8nHighFailureRatio

Symptom

N8nHighFailureRatio fires when a burst of executions across many unrelated workflows is marked status=crashed at once (worker-process death, not per-workflow logic errors). Prior instances: #975, #1620, #1799, #2179, #2256 — 5× in 8 days (2026-06-22 → 2026-06-30). Aftermath orphans execution_entity rows (see the companion playbook n8n-bms4-stuck-execution-cleanup.md).

The crashed marker is written on the next startup: when a worker process dies, the fresh process logs Last session crashed / Marked executions as crashed and flags every execution that was in-flight — hence one process death shows up as many “unrelated” workflow failures.

Ruled-out: the N8N_RUNNERS_ENABLED=false hypothesis (#2258, 2026-07-01)

Earlier investigations suspected that N8N_RUNNERS_ENABLED=false (set on all 4 services 2026-06-17, the storm-onset date) forced in-process Code-node execution and caused the crashes. This is not the case — confirmed from live container logs on 2026-07-01:

  • n8n 2.26.3 ignores the flag — it logs the deprecation N8N_RUNNERS_ENABLED -> Remove this environment variable; it is no longer needed.
  • The JS runner runs regardless: n8n Task Broker ready on 127.0.0.1, port 5679 and Registered runner "JS Task Runner" appear on every start, in internal mode. No external runner image is required for internal mode.
  • Therefore toggling N8N_RUNNERS_ENABLED changes nothing at runtime. The flag and its “external runner image non-existent; disabled hotfix” comment were removed from bms-4/docker-compose.yml in #2258 to stop misdirecting future investigations.

Separate latent issue found: the image has no Python 3, so n8n logs Failed to start Python task runner in internal mode ... Python 3 is missing from this system on every start. Harmless unless a workflow uses a Python Code node (those executions will fail). Not the cause of the whole-worker crash-storms. Track separately if Python Code nodes are needed.

⚠️ FIRST STEP on every occurrence — capture the evidence BEFORE recreating

The standard remediation (recreate the worker containers) destroys the crash logs — docker compose up -d --force-recreate removes the old containers and their logs, which is why the true process-death cause has never been captured. Always dump logs first:

TS=$(date -u +%Y%m%dT%H%M%SZ); OUT=/var/log/n8n-crash-storm-$TS; mkdir -p "$OUT"
for c in bms-4-n8n-1 bms-4-n8n-worker-1-1 bms-4-n8n-worker-2-1 bms-4-n8n-worker-3-1; do
  docker logs "$c" > "$OUT/$c.log" 2>&1
  docker inspect "$c" --format '{{.Name}} restarts={{.RestartCount}} oom={{.State.OOMKilled}} exit={{.State.ExitCode}} status={{.State.Status}}' >> "$OUT/inspect.txt"
done
echo "Logs captured to $OUT — inspect BEFORE recreating."
# Look for the real process-death cause (not secrets): heap/OOM, uncaughtException, broker/runner disconnect.
grep -iE 'uncaught|unhandled|heap|out of memory|ENOMEM|broker|runner.*(lost|died|disconnect)|SIGKILL|exited with' "$OUT"/*.log \
  | grep -viE 'token|key|pass|secret|auth' | tail -50

Keep $OUT and attach the relevant excerpt to the N8nHighFailureRatio issue. This is the missing piece needed to close #2258 — a durable crash fix cannot be chosen until one storm is captured.

Remediation — recreate the worker containers (after capturing logs)

cd /opt/p24-infra/bms-4
docker compose up -d --force-recreate n8n-worker-1 n8n-worker-2 n8n-worker-3
# Then clear the orphaned 'new' rows the storm left behind:
/opt/p24-infra/scripts/n8n-bms4-execution-cleanup.sh

Verify workers are stable afterward (RestartCount=0, OOMKilled=false, no new crashed):

for c in bms-4-n8n-worker-1-1 bms-4-n8n-worker-2-1 bms-4-n8n-worker-3-1; do
  docker inspect "$c" --format '{{.Name}} restarts={{.RestartCount}} oom={{.State.OOMKilled}}'
done

Known-good baseline (verified 2026-07-01)

Host RAM ~25 GiB free, RestartCount=0, OOMKilled=false, Redis healthy, Bull queue empty during quiet periods — host resources are not the bottleneck (consistent with the 2026-06-17 incident). The trigger is a transient worker-process death whose cause is still uncaptured.

Open follow-ups

  • Root cause still open — needs one captured storm (logs via the FIRST STEP above) to identify the process-death cause. Until then, recurrence is expected; recreate + cleanup is the interim fix.
  • Monitoring gap — Prometheus rules watch only the Redis Bull queue; orphaned DB rows are invisible. A n8n_bms4_stuck_new_executions gauge on the bms4 exporter would close it (#2258).

Variant — recovery-flood blocks the metrics-server bind → N8nWorkerDown (#6050, 2026-08-11)

A recreate of the whole stack (docker compose up -d on all services) while executions are in-flight is itself a mini crash-storm: every in-flight execution is killed and marked crashed, and on the next worker boot n8n’s execution-recovery ([Recovery] Logs available, amended execution) iterates all of them. When that flood is large (~1800 in #6050), the affected worker never finishes startup, so its internal :5678 metrics/health HTTP server never binds — the docker port maps and shows LISTEN, but connections are refused (curl localhost:<port>/metrics → HTTP 000). Prometheus scrapes 54.36.123.110:<5679|5680|5681>, gets no 200 → up=0 → N8nWorkerDown. This is app-level (metrics server down), not a container/network outage.

Distinguish from a real outage: the container is Up, RestartCount=0, the port is LISTEN (ss -ltnp | grep 568), but curl -s -o /dev/null -w '%{http_code}' localhost:<port>/metrics returns 000 and docker logs <worker> --tail 8 is a wall of [Recovery] ... amended execution. A healthy worker returns 200 and shows Worker started/finished execution lines. Workers with a light recovery load bind early and stay green — so you often see one down, one up, on identical config.

Remediation (this variant):

# 1. Probe each worker's metrics endpoint (000 = stuck, 200 = healthy).
for p in 5679 5680 5681; do echo -n "$p "; curl -s -m4 -o /dev/null -w '%{http_code}\n' http://localhost:$p/metrics; done
# 2. Give a stuck worker a few minutes — the flood may drain and it self-binds (worker-3 did in #6050).
# 3. If recovery has plateaued (line count stops growing) but /metrics is still 000, force-recreate
#    JUST that worker — leave healthy ones untouched. The killed in-flight execs are already amended,
#    so the fresh boot has ~0 recovery work and binds in seconds:
cd /opt/p24-infra/bms-4 && docker compose up -d --force-recreate n8n-worker-1   # substitute the stuck worker
# 4. Confirm 200 on all three, workers finishing executions, and 0 fresh `crashed` in the last 2 min.

Note the old crashed backlog is not the driver — recovery only re-processes executions killed in the last session, not all historical crashed rows (5789 pre-existing rows in #6050 were irrelevant; they prune at 168h). Deleting them does not prevent recurrence. Prevention: don’t recreate the stack while executions are in-flight — drain workers (or reduce in-flight concurrency) first.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="n8n-bms4-worker",
    result="success",  # "success" | "failed" | "skipped"
    detail="n8n worker crash storm resolved — root cause fixed, workers restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'n8n-bms4-worker', 'success', 'n8n worker crash storm resolved — root cause fixed, workers restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks