p24-infra Docs

n8n-waha-monitor-401-credential-drift

5 min read

Playbook: n8n “WAHA Monitor” workflow persistently failing — HTTP 401 (n8n credential drift)

Trigger

Prometheus alert N8nWorkflowPersistentlyFailing fires for the bms-4 n8n workflow:

Workflow 'WAHA Monitor' (id=O86rjTu1Qfu83LBi) on n8n-bms4-exporter:9226
has recorded N failed executions in the last hour, continuously for 30+ minutes.

Discovered: 2026-06-30 (#2166).

Symptom

Every scheduled run (every 5 min) of WAHA Monitor ends in status='error'. The failing node is GET /status and the error is:

NodeApiError: Authorization failed - please check your credentials
401 - {"message":"Unauthorized","statusCode":401}

The workflow errors at the very first HTTP node (GET /status), so it never reaches the Is WORKING? → POST /reset self-heal branch. That is why the auto-reset never recovers it and the alert keeps firing.

Distinction from the GH-secret-drift playbook

This is not waha-healthcheck-401-gh-secret-drift.md. That one covers the GitHub Actions health-check.yml / waha-session-restart.yml workflows authenticating with the repo secret secrets.WAHA_API_KEY. This failure is the n8n scheduled workflow on bms-4, which authenticates with a stored n8n credential, not a GH secret and not the container env.

The WAHA Monitor nodes (GET /status and POST /reset) use the n8n credential waha-control-token (httpHeaderAuth, id nQ5zOi3Mdf5Emwzs) — sending X-Api-Key as a generic header-auth credential. (The git export under n8n-workflows/WAHA_Monitor_*.json is the older $env.WAHA_API_KEY header form and is stale vs the live definition.)

Root cause

The header value stored inside the n8n credential waha-control-token no longer matches the live WAHA key. When WAHA_API_KEY is rotated in secrets/vps-h1.env.sops (the canonical store, deployed to vps-h1), the n8n credential is not updated automatically — secrets-sync.yml syncs SOPS → server .env files only; it does not touch n8n credentials. So the stored credential drifts and every WAHA call from n8n returns 401.

Diagnose (on bms-4 — never print secret values)

# 1. Confirm the failing node + error on the latest execution:
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -t -A -c \
  "SELECT id,status,\"stoppedAt\" FROM execution_entity \
   WHERE \"workflowId\"='O86rjTu1Qfu83LBi' ORDER BY \"stoppedAt\" DESC LIMIT 5;"
# error data of one failing execution should contain: lastNodeExecuted='GET /status',
# httpCode 401, message 'Authorization failed - please check your credentials'.
 
# 2. Which credential the live nodes use (expect waha-control-token / nQ5zOi3Mdf5Emwzs):
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -t -A -c \
  "SELECT n->>'name', n->'credentials' \
   FROM (SELECT jsonb_array_elements(nodes::jsonb) n FROM workflow_entity \
         WHERE id='O86rjTu1Qfu83LBi') s WHERE n->'credentials' IS NOT NULL;"
 
# 3. Prove the CANONICAL key is valid (so the fix is "resync", not "WAHA down"):
export SOPS_AGE_KEY_FILE=~/.age/p24-infra-keys.txt
WK=$(sops -d --input-type dotenv --output-type dotenv secrets/vps-h1.env.sops \
  | grep '^WAHA_API_KEY=' | cut -d= -f2- | tr -d '"')
curl -s -o /dev/null -w '%{http_code}\n' -H "X-Api-Key: $WK" \
  https://waha2.vps-h1.infra.zintegrowana.online/api/server/status   # expect: 200
unset WK

If the canonical SOPS key returns 200 but the n8n executions return 401, the n8n credential has drifted → resync it (fix below).

Fix — resync the n8n credential waha-control-token (human action)

The n8n public API (v1) cannot update an existing credential’s data — only create/delete. So update it through the n8n editor UI (PROTECTED WAHA control path — do not script blind DB re-encryption):

  1. Open https://n8n.bms-4.infra.zintegrowana.online/ → Credentials → waha-control-token.
  2. Set the Header Auth value to X-Api-Key = the canonical WAHA_API_KEY from secrets/vps-h1.env.sops (read it with the SOPS one-liner above; never paste it into chat/logs).
  3. Save. Open WAHA Monitor and Execute Workflow once — GET /status should now return 200.

Alternative (if you must avoid the UI): create a fresh httpHeaderAuth credential via the n8n API, then repoint both GET /status and POST /reset nodes to the new credential id and re-import the workflow. More moving parts; the UI edit is preferred.

After the key is fixed — the session may still be FAILED

Fixing the 401 only clears the auth failure. If GET /status then reports the default session as FAILED/STOPPED (not WORKING), the workflow’s own POST /reset branch fires and WhatsApp needs a human pairing-code step on vps-h1 (the workflow Telegram-pings “Napisz gotowy gdy będziesz gotowy wpisać kod parowania”). See waha-session-stopped-after-reboot.md. Until the session is re-paired to WORKING, the WAHA gateway is not delivering, even though the n8n workflow itself no longer errors.

Verify resolution

# After resync (+ session re-pair), the next scheduled runs should be 'success':
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -t -A -c \
  "SELECT status,COUNT(*) FROM execution_entity \
   WHERE \"workflowId\"='O86rjTu1Qfu83LBi' AND \"startedAt\" > NOW() - INTERVAL '30 min' \
   GROUP BY status;"

The N8nWorkflowPersistentlyFailing alert auto-resolves once errors stop within the eval window.

Prevention

After any WAHA_API_KEY rotation, the rotation checklist must include “update the n8n waha-control-token credential” alongside the GH Actions secret resync. See docs/playbooks/static-api-key-incident-rotation.md (distribution is mandatory: SOPS → GH Secrets → live envs → containers → n8n credentials → Vercel → .env.local).

  • docs/playbooks/waha-healthcheck-401-gh-secret-drift.md — same 401, but GH Actions secret store
  • docs/playbooks/waha-session-stopped-after-reboot.md — session not WORKING after the key is fixed
  • docs/playbooks/n8n/n8n-bms4-stuck-execution-cleanup.md — n8n queue-mode execution diagnostics
  • docs/playbooks/static-api-key-incident-rotation.md — full key-rotation distribution

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="WAHA_CONTROL_TOKEN",
    result="success",  # "success" | "failed" | "skipped"
    detail="Credential drift fix — WAHA_CONTROL_TOKEN synced between SOPS, server env, and n8n credential",
    env="vps-h1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'WAHA_CONTROL_TOKEN', 'success', 'Credential drift fix — WAHA_CONTROL_TOKEN synced between SOPS, server env, and n8n credential', 'vps-h1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks