p24-infra Docs

n8n-bms4-stuck-execution-cleanup

4 min read

Playbook — bms-4 n8n stuck / orphaned executions

Issue: #1986 · Server: bms-4 (54.36.123.110) · Service: n8n queue-mode (Postgres + Redis/Bull)

Symptom

Tens of thousands of execution_entity rows stuck in status='new' on the bms-4 n8n Postgres DB, never picked up by workers. Daily/scheduled workflows appear to stop producing data. The Prometheus n8n alerts stay green because they watch the live Redis Bull queue (n8n_scaling_queue_jobs_waiting), not orphaned DB rows.

Root cause

In n8n queue mode an execution_entity row is written with status='new' before the Bull job is enqueued in Redis. If the main process crashes, Redis is flushed, or workers are force-stopped mid-drain, the Redis job is lost but the DB row remains new.

n8n’s execution pruning (EXECUTIONS_DATA_PRUNE / EXECUTIONS_DATA_MAX_AGE=168h) only prunes terminal executions (success/error/crashed). A new row is non-terminal, so it is never pruned — it lives forever. These zombie rows accumulate and bloat the DB.

The 2026-06-17 incident

  • 2026-06-17 08:35–11:24 UTC: a worker crash storm enqueued ~60k executions that the workers could not drain; the forced worker restart (18:19 UTC) orphaned them in new.
  • Result: 59,972 immortal new rows (top workflow: atrax, kravag-scheduled-fleet-updates, 59,713), undetected for 12 days.
  • The crash loop itself was resolved by recreating the worker containers on 2026-06-28 18:14 UTC (restart=0, no OOM, 0 crashes since). Host resources were never the bottleneck (26 Gi RAM free, 3 % disk). Workers had N8N_RUNNERS_ENABLED=false (hotfix dated 2026-06-17, the storm’s start date — see bms-4/docker-compose.yml).

Diagnosis commands (run on bms-4)

# Status breakdown
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -c \
  "SELECT status, COUNT(*) FROM execution_entity GROUP BY status ORDER BY 2 DESC;"
 
# Stuck 'new' by workflow + age window
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -c \
  "SELECT \"workflowId\", COUNT(*) FROM execution_entity WHERE status='new' GROUP BY 1 ORDER BY 2 DESC;"
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -c \
  "SELECT MIN(\"createdAt\"), MAX(\"createdAt\") FROM execution_entity WHERE status='new';"
 
# Confirm the Bull queue is empty (proves the 'new' rows are orphaned, not pending)
REDIS_PW=$(grep -m1 '^REDIS_PASSWORD=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"')
for q in wait active delayed failed; do
  echo -n "bull:jobs:$q "; docker exec bms-4-redis-1 redis-cli -a "$REDIS_PW" --no-auth-warning LLEN "bull:jobs:$q"
done; unset REDIS_PW
 
# Confirm workers are stable now (restart count + OOM)
for c in bms-4-n8n-worker-1-1 bms-4-n8n-worker-2-1 bms-4-n8n-worker-3-1; do
  docker inspect "$c" --format '{{.Name}} restarts={{.RestartCount}} oom={{.State.OOMKilled}}'
done

A new row older than a few minutes while the Bull queue is empty = orphaned. Safe to delete.

Remediation — clear orphaned new executions

Workflow definitions live in git (n8n-workflows/, exported daily by n8n-backup.yml). Execution history is ephemeral (7-day prune) and not backup-critical, so deleting orphaned new rows is non-destructive. Child rows cascade (execution_data/metadata/annotations).

# Guarded delete — only frozen 'new' rows older than the safety window
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -v ON_ERROR_STOP=1 -c \
  "DELETE FROM execution_entity WHERE status='new' AND \"createdAt\" < NOW() - INTERVAL '1 hour';"

NOTE: docker exec needs -i for heredoc stdin; prefer -c "<sql>" (argument form) for one-shots.

On 2026-06-29 this removed all 59,972 rows; post-cleanup new=0, zero orphaned execution_data.

Prevention

scripts/n8n-bms4-execution-cleanup.sh runs every 6 h via cron on bms-4 (installed by ansible/playbooks/bms-4.yml, cron file /etc/cron.d/n8n-bms4-execution-cleanup, log /var/log/n8n-bms4-execution-cleanup.log). It deletes new executions older than 2 h and posts a WARNING to Discord when the backlog exceeds BACKLOG_ALERT_THRESHOLD (500) — an early signal the workers are crash-looping again. Run it on demand with:

/opt/p24-infra/scripts/n8n-bms4-execution-cleanup.sh          # uses defaults (MAX_AGE_HOURS=2)
MAX_AGE_HOURS=1 /opt/p24-infra/scripts/n8n-bms4-execution-cleanup.sh

Monitoring gap (follow-up): the Prometheus rules in monitoring/prometheus/rules/n8n.yml only observe the Redis Bull queue, so orphaned DB rows are invisible to alerting. The cleanup workflow’s backlog alert is the current backstop; a dedicated n8n_bms4_stuck_new_executions gauge on the bms4 exporter would close the gap.

  • n8n-bms4-worker-crash-storm.md — the upstream N8nHighFailureRatio crash-storm that produces these orphaned rows (capture logs BEFORE recreating): #2258
  • Daily atrax workflows error on every run after the crash storm (separate app/credential bug): #1990
  • docs/n8n-postgresql-operations.md, docs/n8n-operations.md

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="other",
    resource="n8n-bms4-executions",
    result="success",  # "success" | "failed" | "skipped"
    detail="Stuck n8n execution cleanup — zombie executions cleared from DB",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'other', 'n8n-bms4-executions', 'success', 'Stuck n8n execution cleanup — zombie executions cleared from DB', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks