p24-infra Docs

n8n-alert-router-empty-webhook

8 min read

n8n alert-router (mezmo-alert-router) persistent failure — two variants

Discovered: 2026-06-27 via alert issue #1788 (N8nWorkflowFailureSpike on mezmo-alert-router / Sc2AKCdslinnSvWs) Affected element: any n8n workflow whose Discord HTTP node resolves its URL from a SOPS-managed secret (e.g. mezmo-alert-router, alertmanager-to-incidents-v3).

This workflow has failed persistently for two distinct root causes. Confirm which variant you are looking at before applying a fix — the failing node tells you which one:

VariantFailing nodeErrorRoot cause
A (§below)Send Discord (HTTP)URL parameter cannot be emptyDiscord webhook secret blank in SOPS
B (§Variant B)Dedup GH Issue (Code)Request failed with status code 403GitHub write token (GITHUB_PAT_ALL_WRITES) empty/invalid and the create/comment POST is unguarded

2026-07-13 (issue #4084): the recurrence that day was Variant B, not Variant A — the Discord secret was populated and Send Discord (already hardened with onError: continueRegularOutput) succeeded; the crash came from the Dedup GH Issue code node’s GitHub write returning 403. Always pull the actual lastNodeExecuted from the n8n API (Confirmation §4) before assuming the blank-webhook cause.

Variant A — blank Discord webhook secret

Symptom / trigger

  • Prometheus fires N8nWorkflowFailureSpike (and possibly N8nWorkflowPersistentlyFailing) for a workflow that posts to Discord.

  • Every execution of the workflow fails almost instantly (50–300 ms).

  • The failing node is an HTTP Request node (“Send Discord”) and the error is:

    NodeOperationError: URL parameter cannot be empty

This means the expression that supplies the Discord URL ({{ $env.DISCORD_WEBHOOK_URL }} in the live workflow, or the {{ P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL }} placeholder injected at deploy) resolved to an empty string.

Root cause

The Discord webhook secret is blank in its source of truth. The most common cause is a webhook rotation that wrote empty values into SOPS (see the 2026-06-27 rotation row in docs/secrets-rotation-log.md, issue #1506). Chain of events:

  1. secrets/n8n-bms4.env.sops (and monitoring.env.sops, vps-h1.env.sops) end up with DISCORD_WEBHOOK_URL= / P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL= (key present, value empty).
  2. secrets-sync.yml deploys the decrypted file to /opt/p24-infra/bms-4/.env → empty value there too.
  3. n8n containers recreated after the deploy read the empty env var (N8N_BLOCK_ENV_ACCESS_IN_NODE=false, so $env access works — the value is simply blank).
  4. The “Send Discord” node builds an empty URL and the workflow errors on every Mezmo webhook.

Note: older containers started before the blanking (e.g. redis-exporter, audit-engine) may still hold a non-empty value in their environment. Do not harvest that value and restore it blindly — if the rotation deleted the old webhooks (it did in #1506), those stale values point at a 404 webhook.

Confirmation commands (no secret values printed)

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"   # worker: /home/claude-runner/.age/...
 
# 1. Source-of-truth value length (0 == blank == broken)
v=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
      | grep '^DISCORD_WEBHOOK_URL=' | cut -d= -f2-); echo "len=$(printf '%s' "$v" | wc -c)"; unset v
 
# 2. Deployed value on bms-4 (run on host)
grep -q '^DISCORD_WEBHOOK_URL=.' /opt/p24-infra/bms-4/.env && echo "set" || echo "BLANK"
 
# 3. Live container env length (0 == container has blank value)
docker exec bms-4-n8n-1 printenv DISCORD_WEBHOOK_URL | tr -d '\n' | wc -c
 
# 4. Confirm the failing node via the n8n API (key passed as header, never echoed)
N8N_KEY=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
            | grep '^BMS4_N8N_API_KEY=' | cut -d= -f2-)
curl -s -H "X-N8N-API-KEY: $N8N_KEY" \
  "https://n8n.bms-4.infra.zintegrowana.online/api/v1/executions?workflowId=Sc2AKCdslinnSvWs&status=error&limit=1&includeData=true" \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['data'][0]; e=d['data']['resultData']['error']; print(e['node']['name'], '->', e['message'])"
unset N8N_KEY

Fix (requires the human — the secret value is not recoverable by an agent)

The canonical fix is to re-populate the Discord webhook values in SOPS. An agent cannot do this: GH Secret values are write-only, .env.local lives on the dev workstation, and old webhooks may be deleted. Hand to radieu:

  1. From the dev workstation, get the current webhook URLs (created during the last rotation) from .env.local or the GitHub Secret DISCORD_WEBHOOK_URL.
  2. Restore them into each affected SOPS file using the safe SOPS edit pattern in CLAUDE.md (decrypt → edit temp → re-encrypt → canary decrypt → commit). Keys to fix:
    • secrets/n8n-bms4.env.sops: DISCORD_WEBHOOK_URL, P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
    • secrets/monitoring.env.sops: P24_DISCORD_INFRA_WEEKLY_WEBHOOK_URL, P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL, DISCORD_P24_ISSUES_WEBHOOK_URL
    • secrets/vps-h1.env.sops: DISCORD_WEBHOOK_URL
  3. Commit + PR to main. Merge triggers secrets-sync.yml, redeploying /opt/p24-infra/bms-4/.env.
  4. Recreate the n8n containers so they pick up the new value: 
    ssh root@54.36.123.110 "cd /opt/p24-infra/bms-4 && ./start.sh"   # start.sh sources SOPS env first
  5. Verify: re-run a Mezmo alert (or wait for the next one) and confirm executions succeed.

Hardening (optional — stops the failure spike even when the webhook is blank)

A monitoring/alerting workflow should not crash-loop just because one notification channel is unset. On the live workflow, set the Send Discord node’s error handling to Continue (using error output) — onError: continueRegularOutput in the node JSON. The parallel “IF GH Issue” → “Dedup GH Issue” path still creates GitHub issues for actionable alerts, so an empty/dead Discord webhook degrades gracefully instead of producing an N8nWorkflowFailureSpike.

n8n is the source of truth for workflows; n8n-workflows/*.json in the repo are one-way sanitized backups (see docs/n8n-operations.md). Apply the onError change in the n8n UI/API, then re-export the backup with scripts/export-n8n-workflows.sh.

Variant B — GitHub API 403 on issue create/comment (Dedup GH Issue)

Discovered: 2026-07-13 via alert issue #4084 (N8nWorkflowPersistentlyFailing, same workflow).

Symptom / trigger

  • N8nWorkflowPersistentlyFailing (or N8nWorkflowFailureSpike) fires for mezmo-alert-router.

  • Send Discord succeeds (webhook secret is populated); the failure is later in the chain.

  • The last executed node is Dedup GH Issue (a Code node, not an HTTP node) and the error is:

    NodeApiError / AxiosError: Request failed with status code 403

Root cause

The Dedup GH Issue code node calls the GitHub REST API to dedup/create issues for actionable Mezmo alerts. It authenticates with the token built in Prepare Alert:

const GH_TOKEN = 'token {{ GITHUB_PAT_ALL_WRITES }}';   // deploy-time placeholder → SOPS value

Two compounding faults:

  1. GITHUB_PAT_ALL_WRITES is empty/absent in secrets/n8n-bms4.env.sops (verified 2026-07-13, decrypt length 0). The live workflow therefore sends an empty/invalid GitHub credential → GitHub returns 403 on every write (POST /issues, POST /issues/{n}/comments).
  2. The write POSTs are unguarded. The code node wraps only the GET …/issues lookup in try/catch (fail-open); the POST create and comment calls are awaited directly, so a 403 on write throws → the node errors → the whole execution fails → the persistent-failure alert fires on every actionable Mezmo alert. (Non-actionable alerts, create_gh_issue !== 'yes', skip the API and do not fail — which is why only some Mezmo alert types trip it.)

Confirmation commands (no secret values printed)

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"   # worker: /home/claude-runner/.age/...
 
# 1. Token present? length 0 == missing == broken
v=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
      | grep '^GITHUB_PAT_ALL_WRITES=' | cut -d= -f2-); echo "len=$(printf '%s' "$v" | wc -c)"; unset v
 
# 2. Which node actually failed (should print: Dedup GH Issue -> ... 403)
N8N_KEY=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
            | grep '^BMS4_N8N_API_KEY=' | cut -d= -f2-)
curl -s -H "X-N8N-API-KEY: $N8N_KEY" \
  "https://n8n.bms-4.infra.zintegrowana.online/api/v1/executions?workflowId=Sc2AKCdslinnSvWs&status=error&limit=1&includeData=true" \
  | python3 -c "import sys,json; d=json.load(sys.stdin)['data'][0]['data']['resultData']; print(d.get('lastNodeExecuted'), '->', d['error']['message'])"
unset N8N_KEY

Fix

Two independent fixes — the first stops the crash-loop, the second restores GitHub issue creation:

  1. Resilience hardening (stops the alert; sys-admin / infra-task on bms-4). In the live Dedup GH Issue code node, wrap the create/comment POST calls in try/catch fail-open (matching the existing GET lookup) so a bad/missing GH token degrades gracefully — Discord still fires, the parallel path still runs, and the execution no longer errors. Apply live in the n8n UI/API (n8n is the source of truth), then re-export the backup with scripts/export-n8n-workflows.sh. Reference patch shape:

    } else {
  try {
    const created = await this.helpers.httpRequest({ /* POST …/issues */ });
    out.push({ json: { action: 'created', issue: created.number, title: j.gh_title } });
  } catch (e) {
    out.push({ json: { action: 'create_failed', title: j.gh_title, error: String(e).slice(0, 200) } });
  }
}

    (Alternatively / additionally set the code node’s onError: continueRegularOutput.)

  2. Restore the write token (real fix; secret-manager + human). GITHUB_PAT_ALL_WRITES must be a GitHub PAT with issues: write on radieu/p24-infra. An agent cannot mint it — hand to radieu/secret-manager: generate the PAT → add GITHUB_PAT_ALL_WRITES to secrets/n8n-bms4.env.sops (safe SOPS edit + canary) → PR → secrets-sync.yml deploys to /opt/p24-infra/bms-4/.env → recreate the n8n containers so the workflow re-injects the value. Verify: trigger a Mezmo alert and confirm the Dedup GH Issue node returns action: created.

GITHUB_PAT_ALL_WRITES blank in SOPS is likely fallout from the GitHub-token exposure/rotation churn tracked in docs/priorities.md (#4047 / #2824 family). Confirm the intended token identity before restoring.

Prevention

  • Post-rotation canary: after any Discord webhook rotation, assert every webhook key decrypts to a non-empty value before committing — extend docs/secrets-rotation-log.md rotation steps with: 
    for k in DISCORD_WEBHOOK_URL P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL; do
  v=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops | grep "^$k=" | cut -d= -f2-)
  [ -n "$v" ] || { echo "EMPTY: $k — do NOT commit"; exit 1; }
done
  • Prefer the resilience hardening above so a blank secret never turns into a workflow failure spike.
  • Variant B canary: the same rotation canary must cover GITHUB_PAT_ALL_WRITES in secrets/n8n-bms4.env.sops — a blank write token silently breaks GitHub issue creation for every actionable Mezmo alert.
  • Guard every external write in code nodes: any this.helpers.httpRequest write inside a Code node (not just the lookup) must be wrapped in try/catch — an unguarded POST turns a downstream service outage into an n8n crash-loop and a persistent-failure alert.

Escalation path

If the human is unavailable and alerts are being lost, apply the Hardening step to stop the spike (GitHub-issue alerts keep flowing), and leave issue #1788 open with the human-action label until the webhook secret is restored.

Graph View

Backlinks