p24-infra Docs

n8n-workflow-db-query-hang

4 min read

Playbook — n8n workflow hangs on a remote-Postgres query (crashed accumulation)

Symptom / alert: N8nWorkflowPersistentlyFailing — an n8n workflow records many crashed executions per hour, continuously, even though the n8n containers look healthy.

First seen: 2026-06-30, workflow p24-content-orchestrator (id XXEybVHvKmw0iF92) on bms-4 — ~12 crashed/hour all day. Issue #2254.

Signature (how to recognise this class)

  1. The failing executions have status crashed (not error). crashed = the execution was still running when an n8n worker process restarted and the recovery sweep marked it crashed.

  2. All crashed executions stall at the same node, and that node is a Postgres / HTTP / other network node with no timeout. Confirm via the n8n DB:

    PSQL() { docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -t -A "$@"; }
# last node reached by a crashed execution (look for "lastNodeExecuted")
PSQL -c "SELECT left(data,400) FROM execution_data WHERE \"executionId\"=<ID>;"

  3. The crashed executions ran for exact multiples of the schedule interval (e.g. 301 / 601 / 901 s for a 5-min cron). That means they hung indefinitely and were only stopped by the next worker restart — not by any per-run limit.

    PSQL -c "SELECT id,status,round(EXTRACT(EPOCH FROM (\"stoppedAt\"-\"startedAt\"))::numeric,1) secs
         FROM execution_entity WHERE \"workflowId\"='<WF_ID>' ORDER BY \"startedAt\" DESC LIMIT 12;"

  4. Container health is fine — docker inspect <c> --format '{{.RestartCount}} {{.State.OOMKilled}}' shows 0 false, memory low. The crash is at the workflow-execution layer, not the OS.

Root cause

The node’s query/connection has no timeout, and the workflow settings has no executionTimeout. A single stale / half-open connection to the remote DB (Supabase pooler dropping an idle connection, an IPv4→IPv6 socat tunnel hiccup, a network blip) leaves the query waiting on a socket that never answers. The execution hangs forever, ties up an n8n worker slot, and is swept to crashed on the next worker restart. Because the trigger keeps firing on its schedule, a new hung execution accumulates every interval → the persistent-failure alert.

Immediate mitigation (stops the bleeding)

Recreate / restart the n8n stack — fresh connections clear the stuck sockets:

cd /opt/p24-infra/bms-4 && docker compose restart n8n n8n-worker-1 n8n-worker-2 n8n-worker-3

Verify executions go back to success (sub-second):

PSQL -c "SELECT id,status,\"startedAt\" FROM execution_entity WHERE \"workflowId\"='<WF_ID>'
         ORDER BY \"startedAt\" DESC LIMIT 5;"

This is a symptom fix only — the hang recurs at the next idle-connection drop unless you apply the durable guard below.

Durable fix — bound the execution

Add executionTimeout (seconds) to the workflow settings so a hung run is cancelled as a recoverable error instead of hanging until a restart. Pick a value far above the healthy runtime (healthy content-orchestrator runs take <0.5 s → 120 gives ~240× headroom) and below the schedule interval.

1. Repo source of truth (n8n-workflows/branding/<workflow>.json):

"settings": {
  "executionOrder": "v1",
  "executionTimeout": 120
}

2. Live n8n DB — apply to the running workflow (jsonb merge, scoped to one row):

docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -c \
  "UPDATE workflow_entity SET settings = (settings::jsonb || '{\"executionTimeout\": 120}'::jsonb)::json,
   \"updatedAt\" = NOW() WHERE id='<WF_ID>';"

3. Make it effective immediately. The n8n main process holds the workflow in memory, so the new setting only takes effect after a workflow reload. Either:

  • toggle the workflow Inactive → Active in the n8n UI (scoped, no downtime to other workflows), or
  • docker compose restart n8n (reloads all active workflows; brief interruption — only if a UI toggle is not available).

A DB-only edit persists correctly and applies on the next natural reload/restart, so step 3 is not urgent once the symptom is already resolved.

Notes

  • This is a workflow-class fix. Any n8n workflow that runs a remote-DB/HTTP node on a schedule with no timeout is vulnerable — consider adding executionTimeout to the other branding workflows (publish-linkedin, oauth-token-refresh, brand-credential-health-check) and similar.
  • n8n workflow definitions are not auto-deployed from this repo (see #2141); the repo JSON is a source-of-truth snapshot. Always apply both the repo edit and the live DB/UI change.
  • executionTimeout is in seconds in n8n workflow settings.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="n8n-bms4-db",
    result="success",  # "success" | "failed" | "skipped"
    detail="n8n DB query hang resolved — stale connection killed, n8n restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'n8n-bms4-db', 'success', 'n8n DB query hang resolved — stale connection killed, n8n restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks