p24-infra Docs

n8n-alert-router-empty-webhook

12 min read

n8n alert-router (mezmo-alert-router) persistent failure — two variants

Discovered: 2026-06-27 via alert issue #1788 (N8nWorkflowFailureSpike on mezmo-alert-router / Sc2AKCdslinnSvWs) Affected element: any n8n workflow whose Discord HTTP node resolves its URL from a SOPS-managed secret (e.g. mezmo-alert-router, alertmanager-to-incidents-v3).

This workflow has failed persistently for two distinct root causes. Confirm which variant you are looking at before applying a fix — the failing node tells you which one:

VariantFailing nodeErrorRoot cause
A (§below)Send Discord (HTTP)URL parameter cannot be emptyDiscord webhook secret blank in SOPS
B (§Variant B)Dedup GH Issue (Code)Request failed with status code 403GitHub write token (GITHUB_PAT_ALL_WRITES) empty/invalid and the create/comment POST is unguarded
C (§Variant C)any Code node (e.g. Nologs Suppression Guard)Task request timed out after 60 seconds / Your Code node task was not matched to a runnerThe JS task runner crash-loops on startup because a module in NODE_FUNCTION_ALLOW_EXTERNAL throws at require(). The runner eagerly requires every allowlisted external module; pdf-parse@2.4.5 throws DOMMatrix is not defined → no runner ever registers → every Code node instance-wide times out at 60s. Not specific to this workflow — it just runs a Code node on every webhook so it trips the spike alert first.

2026-07-13 (issue #4084): the recurrence that day was Variant B, not Variant A — the Discord secret was populated and Send Discord (already hardened with onError: continueRegularOutput) succeeded; the crash came from the Dedup GH Issue code node’s GitHub write returning 403. Always pull the actual lastNodeExecuted from the n8n API (Confirmation §4) before assuming the blank-webhook cause.

Variant A — blank Discord webhook secret

Symptom / trigger

  • Prometheus fires N8nWorkflowFailureSpike (and possibly N8nWorkflowPersistentlyFailing) for a workflow that posts to Discord.

  • Every execution of the workflow fails almost instantly (50–300 ms).

  • The failing node is an HTTP Request node (“Send Discord”) and the error is:

    NodeOperationError: URL parameter cannot be empty

This means the expression that supplies the Discord URL ({{ $env.DISCORD_WEBHOOK_URL }} in the live workflow, or the {{ P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL }} placeholder injected at deploy) resolved to an empty string.

Root cause

The Discord webhook secret is blank in its source of truth. The most common cause is a webhook rotation that wrote empty values into SOPS (see the 2026-06-27 rotation row in docs/secrets-rotation-log.md, issue #1506). Chain of events:

  1. secrets/n8n-bms4.env.sops (and monitoring.env.sops, vps-h1.env.sops) end up with DISCORD_WEBHOOK_URL= / P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL= (key present, value empty).
  2. secrets-sync.yml deploys the decrypted file to /opt/p24-infra/bms-4/.env → empty value there too.
  3. n8n containers recreated after the deploy read the empty env var (N8N_BLOCK_ENV_ACCESS_IN_NODE=false, so $env access works — the value is simply blank).
  4. The “Send Discord” node builds an empty URL and the workflow errors on every Mezmo webhook.

Note: older containers started before the blanking (e.g. redis-exporter, audit-engine) may still hold a non-empty value in their environment. Do not harvest that value and restore it blindly — if the rotation deleted the old webhooks (it did in #1506), those stale values point at a 404 webhook.

Confirmation commands (no secret values printed)

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"   # worker: /home/claude-runner/.age/...
 
# 1. Source-of-truth value length (0 == blank == broken)
v=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
      | grep '^DISCORD_WEBHOOK_URL=' | cut -d= -f2-); echo "len=$(printf '%s' "$v" | wc -c)"; unset v
 
# 2. Deployed value on bms-4 (run on host)
grep -q '^DISCORD_WEBHOOK_URL=.' /opt/p24-infra/bms-4/.env && echo "set" || echo "BLANK"
 
# 3. Live container env length (0 == container has blank value)
docker exec bms-4-n8n-1 printenv DISCORD_WEBHOOK_URL | tr -d '\n' | wc -c
 
# 4. Confirm the failing node via the n8n API (key passed as header, never echoed)
N8N_KEY=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
            | grep '^BMS4_N8N_API_KEY=' | cut -d= -f2-)
curl -s -H "X-N8N-API-KEY: $N8N_KEY" \
  "https://n8n.bms-4.infra.zintegrowana.online/api/v1/executions?workflowId=Sc2AKCdslinnSvWs&status=error&limit=1&includeData=true" \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['data'][0]; e=d['data']['resultData']['error']; print(e['node']['name'], '->', e['message'])"
unset N8N_KEY

Fix (requires the human — the secret value is not recoverable by an agent)

The canonical fix is to re-populate the Discord webhook values in SOPS. An agent cannot do this: GH Secret values are write-only, .env.local lives on the dev workstation, and old webhooks may be deleted. Hand to radieu:

  1. From the dev workstation, get the current webhook URLs (created during the last rotation) from .env.local or the GitHub Secret DISCORD_WEBHOOK_URL.
  2. Restore them into each affected SOPS file using the safe SOPS edit pattern in CLAUDE.md (decrypt → edit temp → re-encrypt → canary decrypt → commit). Keys to fix:
    • secrets/n8n-bms4.env.sops: DISCORD_WEBHOOK_URL, P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
    • secrets/monitoring.env.sops: P24_DISCORD_INFRA_WEEKLY_WEBHOOK_URL, P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL, DISCORD_P24_ISSUES_WEBHOOK_URL
    • secrets/vps-h1.env.sops: DISCORD_WEBHOOK_URL
  3. Commit + PR to main. Merge triggers secrets-sync.yml, redeploying /opt/p24-infra/bms-4/.env.
  4. Recreate the n8n containers so they pick up the new value: 
    ssh root@54.36.123.110 "cd /opt/p24-infra/bms-4 && ./start.sh"   # start.sh sources SOPS env first
  5. Verify: re-run a Mezmo alert (or wait for the next one) and confirm executions succeed.

Hardening (optional — stops the failure spike even when the webhook is blank)

A monitoring/alerting workflow should not crash-loop just because one notification channel is unset. On the live workflow, set the Send Discord node’s error handling to Continue (using error output) — onError: continueRegularOutput in the node JSON. The parallel “IF GH Issue” → “Dedup GH Issue” path still creates GitHub issues for actionable alerts, so an empty/dead Discord webhook degrades gracefully instead of producing an N8nWorkflowFailureSpike.

n8n is the source of truth for workflows; n8n-workflows/*.json in the repo are one-way sanitized backups (see docs/n8n-operations.md). Apply the onError change in the n8n UI/API, then re-export the backup with scripts/export-n8n-workflows.sh.

Variant B — GitHub API 403 on issue create/comment (Dedup GH Issue)

Discovered: 2026-07-13 via alert issue #4084 (N8nWorkflowPersistentlyFailing, same workflow).

Symptom / trigger

  • N8nWorkflowPersistentlyFailing (or N8nWorkflowFailureSpike) fires for mezmo-alert-router.

  • Send Discord succeeds (webhook secret is populated); the failure is later in the chain.

  • The last executed node is Dedup GH Issue (a Code node, not an HTTP node) and the error is:

    NodeApiError / AxiosError: Request failed with status code 403

Root cause

The Dedup GH Issue code node calls the GitHub REST API to dedup/create issues for actionable Mezmo alerts. It authenticates with the token built in Prepare Alert:

const GH_TOKEN = 'token {{ GITHUB_PAT_ALL_WRITES }}';   // deploy-time placeholder → SOPS value

Two compounding faults:

  1. GITHUB_PAT_ALL_WRITES is empty/absent in secrets/n8n-bms4.env.sops (verified 2026-07-13, decrypt length 0). The live workflow therefore sends an empty/invalid GitHub credential → GitHub returns 403 on every write (POST /issues, POST /issues/{n}/comments).
  2. The write POSTs are unguarded. The code node wraps only the GET …/issues lookup in try/catch (fail-open); the POST create and comment calls are awaited directly, so a 403 on write throws → the node errors → the whole execution fails → the persistent-failure alert fires on every actionable Mezmo alert. (Non-actionable alerts, create_gh_issue !== 'yes', skip the API and do not fail — which is why only some Mezmo alert types trip it.)

Confirmation commands (no secret values printed)

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"   # worker: /home/claude-runner/.age/...
 
# 1. Token present? length 0 == missing == broken
v=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
      | grep '^GITHUB_PAT_ALL_WRITES=' | cut -d= -f2-); echo "len=$(printf '%s' "$v" | wc -c)"; unset v
 
# 2. Which node actually failed (should print: Dedup GH Issue -> ... 403)
N8N_KEY=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
            | grep '^BMS4_N8N_API_KEY=' | cut -d= -f2-)
curl -s -H "X-N8N-API-KEY: $N8N_KEY" \
  "https://n8n.bms-4.infra.zintegrowana.online/api/v1/executions?workflowId=Sc2AKCdslinnSvWs&status=error&limit=1&includeData=true" \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['data'][0]['data']['resultData']; print(d.get('lastNodeExecuted'), '->', d['error']['message'])"
unset N8N_KEY

Fix

Two independent fixes — the first stops the crash-loop, the second restores GitHub issue creation:

  1. Resilience hardening (stops the alert; sys-admin / infra-task on bms-4). In the live Dedup GH Issue code node, wrap the create/comment POST calls in try/catch fail-open (matching the existing GET lookup) so a bad/missing GH token degrades gracefully — Discord still fires, the parallel path still runs, and the execution no longer errors. Apply live in the n8n UI/API (n8n is the source of truth), then re-export the backup with scripts/export-n8n-workflows.sh. Reference patch shape:

    } else {
  try {
    const created = await this.helpers.httpRequest({ /* POST …/issues */ });
    out.push({ json: { action: 'created', issue: created.number, title: j.gh_title } });
  } catch (e) {
    out.push({ json: { action: 'create_failed', title: j.gh_title, error: String(e).slice(0, 200) } });
  }
}

    (Alternatively / additionally set the code node’s onError: continueRegularOutput.)

  2. Restore the write token (real fix; secret-manager + human). GITHUB_PAT_ALL_WRITES must be a GitHub PAT with issues: write on radieu/p24-infra. An agent cannot mint it — hand to radieu/secret-manager: generate the PAT → add GITHUB_PAT_ALL_WRITES to secrets/n8n-bms4.env.sops (safe SOPS edit + canary) → PR → secrets-sync.yml deploys to /opt/p24-infra/bms-4/.env → recreate the n8n containers so the workflow re-injects the value. Verify: trigger a Mezmo alert and confirm the Dedup GH Issue node returns action: created.

GITHUB_PAT_ALL_WRITES blank in SOPS is likely fallout from the GitHub-token exposure/rotation churn tracked in docs/priorities.md (#4047 / #2824 family). Confirm the intended token identity before restoring.

Variant C — JS task runner crash-loop from a bad NODE_FUNCTION_ALLOW_EXTERNAL module (DOMMatrix)

Discovered: 2026-09-15 via alert issue #7078 (N8nWorkflowFailureSpike, mezmo-alert-router / Sc2AKCdslinnSvWs).

Symptom / trigger

  • N8nWorkflowFailureSpike fires; every execution of a Code-node workflow errors after ~60 s (not the near-instant failure of Variants A/B).

  • The failing node is a Code node (here Nologs Suppression Guard) and the error is:

    Task request timed out after 60 seconds
Your Code node task was not matched to a runner within the timeout period. This indicates
that the task runner is currently down, or not ready, or at capacity...

  • In the n8n main container log (docker logs bms-4-n8n-1): the JS task runner fails to start, repeatedly — [runnner:js] Task runner failed to start with, deeper in the stack, ReferenceError: DOMMatrix is not defined thrown from pdf-parse/.../index.cjs. The n8n Task Broker ready line is present but Registered runner "JS Task Runner" is absent — the broker (server) is up but no runner (client) ever registers.

Root cause

n8n 2.26.3’s JS task runner runs preventPrototypePollution() at startup, which eagerly require()s every module listed in NODE_FUNCTION_ALLOW_EXTERNAL. Any module that throws at load (anything other than MODULE_NOT_FOUND) is re-thrown and kills the runner — which then crash-loops and never registers, so all Code-node tasks time out after 60 s instance-wide.

pdf-parse@2.4.5 (bundled in the n8n image) references the browser DOMMatrix API at module load and tries to polyfill it via @napi-rs/canvas; that native binding fails to load in n8n’s Node runtime, so DOMMatrix stays undefined and the require throws. pdf-parse was added to NODE_FUNCTION_ALLOW_EXTERNAL (alongside the harmless xlsx) in #7009 for a dispo Stage-2 PDF→text Code node; the runner only re-reads the allowlist on container restart, so the failure surfaced on the next recreate, not at merge time. Verify which module is at fault:

# 1. What is allowlisted?
docker exec bms-4-n8n-1 printenv NODE_FUNCTION_ALLOW_EXTERNAL          # e.g. "xlsx,pdf-parse"
# 2. Which allowlisted module throws at require? (run from n8n's module root)
docker exec bms-4-n8n-1 sh -c 'cd /usr/local/lib/node_modules/n8n && node -e "for (const m of process.env.NODE_FUNCTION_ALLOW_EXTERNAL.split(\",\")) { try { require(m); console.log(m+\": OK\") } catch(e){ console.log(m+\": \"+e.message) } }"'

Fix

Immediate mitigation (infra-task on bms-4, no git-tree mutation): recreate the n8n services with a temporary compose overlay that drops the bad module from the allowlist, so the runner registers again within seconds:

cat > /tmp/n8n-allowlist-fix.yml <<'YML'
services:
  n8n:          { environment: { NODE_FUNCTION_ALLOW_EXTERNAL: xlsx } }
  n8n-worker-1: { environment: { NODE_FUNCTION_ALLOW_EXTERNAL: xlsx } }
  n8n-worker-2: { environment: { NODE_FUNCTION_ALLOW_EXTERNAL: xlsx } }
  n8n-worker-3: { environment: { NODE_FUNCTION_ALLOW_EXTERNAL: xlsx } }
YML
cd /opt/p24-infra/bms-4
docker compose -f docker-compose.yml -f /tmp/n8n-allowlist-fix.yml up -d n8n n8n-worker-1 n8n-worker-2 n8n-worker-3
# verify: log shows 'Registered runner "JS Task Runner"' and a Code-node workflow now succeeds in ~2s

start.sh uses an explicit -f docker-compose.yml and does not auto-load docker-compose.override.yml, so a stray override won’t reliably survive a deploy — the overlay above is a stopgap until the tracked compose fix lands.

Permanent fix (repo PR): remove the offending module from NODE_FUNCTION_ALLOW_EXTERNAL in all four n8n services in bms-4/docker-compose.yml. A module that cannot be safely require()d at startup must never sit in the global allowlist — it breaks the runner for every workflow, not just the one that uses it. For the dropped capability (here PDF→text), use a runner-safe path: n8n’s built-in Extract from File node, an external task-runner deployment, or a module that does not need DOMMatrix/native canvas.

Prevention

  • Post-rotation canary: after any Discord webhook rotation, assert every webhook key decrypts to a non-empty value before committing — extend docs/secrets-rotation-log.md rotation steps with: 
    for k in DISCORD_WEBHOOK_URL P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL; do
  v=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops | grep "^$k=" | cut -d= -f2-)
  [ -n "$v" ] || { echo "EMPTY: $k — do NOT commit"; exit 1; }
done
  • Prefer the resilience hardening above so a blank secret never turns into a workflow failure spike.
  • Variant B canary: the same rotation canary must cover GITHUB_PAT_ALL_WRITES in secrets/n8n-bms4.env.sops — a blank write token silently breaks GitHub issue creation for every actionable Mezmo alert.
  • Guard every external write in code nodes: any this.helpers.httpRequest write inside a Code node (not just the lookup) must be wrapped in try/catch — an unguarded POST turns a downstream service outage into an n8n crash-loop and a persistent-failure alert.
  • Variant C canary — vet every NODE_FUNCTION_ALLOW_EXTERNAL addition: before adding a module to the allowlist, confirm it require()s cleanly in the n8n runtime (docker exec bms-4-n8n-1 sh -c 'cd /usr/local/lib/node_modules/n8n && node -e "require(\"<mod>\")"'). The JS task runner eagerly requires the whole allowlist at startup, so a single module that throws at load (e.g. pdf-parse needing DOMMatrix) crash-loops the runner and times out every Code node instance-wide — the blast radius is the entire n8n instance, not just the workflow that uses the module.

Escalation path

If the human is unavailable and alerts are being lost, apply the Hardening step to stop the spike (GitHub-issue alerts keep flowing), and leave issue #1788 open with the human-action label until the webhook secret is restored.

Graph View

Backlinks