Playbook: N8nBms4ExporterDown — n8n BMS-4 API exporter scrape failing

Trigger

Prometheus alert N8nBms4ExporterDown fires:

n8n_bms4_scrape_ok == 0 for 5+ minutes
n8n-bms4-exporter:9226 returns metrics but with scrape_ok=0
Per-workflow name metrics absent from Grafana n8n dashboard

How to Confirm

# On vps-i1: check exporter logs
ssh root@217.154.82.162 "docker logs monitoring-n8n-bms4-exporter-1 --tail=50"
 
# Expected on failure: "BMS-4 scrape failed: 401 Client Error" or "KeyError: N8N_BMS4_API_KEY"
# Expected on success: "Collected BMS-4: N workflows (M active), 1h=X series, 24h=Y series"
 
# Check metrics directly
ssh root@217.154.82.162 "curl -s http://localhost:9226/metrics | grep n8n_bms4_scrape_ok"
# Healthy: n8n_bms4_scrape_ok{instance="n8n.bms-4.infra.zintegrowana.online"} 1
# Failing: n8n_bms4_scrape_ok{...} 0
 
# Verify n8n BMS-4 API is reachable from vps-i1
ssh root@217.154.82.162 "curl -s -o /dev/null -w '%{http_code}' https://n8n.bms-4.infra.zintegrowana.online/healthz"
# 200 = n8n is up; if not 200 — n8n BMS-4 itself is down (different issue)
 
# Verify the env var is set in monitoring .env on vps-i1
ssh root@217.154.82.162 "grep N8N_BMS4_API_KEY /opt/p24-infra/monitoring/.env | cut -c1-30"
# Should print first 30 chars of the variable (do NOT print the full value)

Common Root Causes and Fixes

Cause 1: N8N_BMS4_API_KEY missing from monitoring.env.sops (primary — PR #1150)

Symptom: Container logs show KeyError: 'N8N_BMS4_API_KEY' or container fails to start. grep N8N_BMS4_API_KEY /opt/p24-infra/monitoring/.env returns nothing.

Root cause: secrets/monitoring.env.sops did not contain N8N_BMS4_API_KEY. The secrets-sync.yml vps-i1 job only deploys monitoring.env.sops to vps-i1 — any key missing from that file will be absent from the deployed .env.

Fix (already applied in PR #1150): Verify N8N_BMS4_API_KEY is now in monitoring.env.sops:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "N8N_BMS4_API_KEY"

If missing — re-add it (see CLAUDE.md §Secrets for SOPS add procedure):

# Get value from n8n-bms4.env.sops (key is BMS4_N8N_API_KEY there)
# Follow CLAUDE.md SOPS add procedure; use --filename-override "secrets/monitoring.env.sops"
# when encrypting if path_regex causes "no matching creation rules found" error

Cause 2: N8N_BMS4_API_KEY is stale / rotated in n8n

Symptom: Container logs show 401 Client Error: Unauthorized. grep N8N_BMS4_API_KEY /opt/p24-infra/monitoring/.env returns a non-empty value. Alert N8nBms4ExporterAuthFailure is firing alongside N8nBms4ExporterDown (#3557) — this distinct alert (n8n_bms4_scrape_auth_failed == 1) means the failure is a 401/403, so it is this cause and not connectivity. A redeploy/secrets-sync will NOT fix it — the key must be regenerated in n8n and rotated in SOPS.

Fix:

Log in to n8n at https://n8n.bms-4.infra.zintegrowana.online → Settings → n8n API
Generate a new API key
Update secrets/n8n-bms4.env.sops (key BMS4_N8N_API_KEY) and secrets/monitoring.env.sops (key N8N_BMS4_API_KEY) with the new value — both must be identical
Commit and push to dev — secrets-sync.yml auto-deploys to both vps-i1 and bms-4
After deploy: restart only the exporter: docker compose up -d --no-deps n8n-bms4-exporter

Cause 3: n8n BMS-4 is unreachable from vps-i1

Symptom: curl https://n8n.bms-4.infra.zintegrowana.online/healthz from vps-i1 times out or returns 5xx.

Fix: This is an n8n/bms-4 issue, not an exporter issue. Check:

bms-4 Docker stack: ssh root@54.36.123.110 "docker compose ps" (in /opt/p24-infra/bms-4/)
Traefik logs on bms-4 for TLS or routing errors
See docs/servers/p4-ovh-bms-4-ns3101999-operations.md

Cause 4: vps-i1 monitoring stack restarted but exporter not yet running

Symptom: Alert fires right after vps-i1 restart. All monitoring containers were just restarted.

Fix: Wait for the 5-minute alert holddown to expire. If still firing after 10 min:

ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose ps n8n-bms4-exporter"
# If Exit/Restart: docker compose up -d --no-deps n8n-bms4-exporter && docker compose logs --tail=30 n8n-bms4-exporter

Cause 5: n8n main holds a dead Postgres pool after `bms-4-n8n-postgres-1` restarted (#6369)

Symptom (distinct from all above):

Exporter logs show 503 Server Error: Service Unavailable on /api/v1/workflows (NOT 401, NOT a connection timeout).
n8n_bms4_scrape_ok == 0 and n8n_bms4_scrape_auth_failed == 0 — so it is neither auth (Cause 2) nor a missing key (Cause 1).
From vps-i1, curl .../healthz returns 200 — n8n is “up” — but the API /api/v1/workflows is 503. The health endpoint does not exercise the DB pool, so it stays green while the API is broken.
On bms-4, docker logs bms-4-n8n-1 shows repeated Error: Cannot use a pool after calling end on the pool (TypeORM PostgresDriver).
bms-4-n8n-postgres-1 shows a recent Up N minutes while bms-4-n8n-1 shows Up 19 hours — i.e. Postgres was restarted/recreated (e.g. a docker compose up -d touched it) but the long-running n8n main process was not.

Root cause: When bms-4-n8n-postgres-1 restarts, the n8n worker containers auto-recover (they retry the DB connection — logs show Database connection recovered after N attempt(s)), but the n8n main process permanently ends its TypeORM connection pool and never rebuilds it, so every DB-backed API call (including GET /api/v1/workflows) returns 503 indefinitely.

Fix (bms-4 — local docker, no code/secret change needed):

cd /opt/p24-infra/bms-4   # PLAYBOOK: n8n/n8n-bms4-exporter-missing-key.md
docker compose restart n8n
# wait ~60-90s for n8n to boot, then confirm the API is DB-backed again (401 = pool alive, auth-gated):
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:5678/api/v1/workflows?limit=1   # expect 401, not 503

Only the main n8n service needs restarting — the workers self-heal. Within ~1 exporter cycle (~70s) n8n_bms4_scrape_ok returns to 1 and the alert resolves.

Prevention: whenever bms-4-n8n-postgres-1 is intentionally restarted/recreated, restart bms-4-n8n-1 in the same operation so the main process does not strand a dead pool.

Deploying the Fix to vps-i1 (manual trigger)

When vps-i1 is back up after a restart and secrets-sync.yml auto-run failed:

# From local workstation
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=vps-i1
# Then wait for it to complete:
gh run list --repo radieu/p24-infra --workflow secrets-sync.yml --limit 2

Or directly via SSH if CI is not available:

ssh root@217.154.82.162 "
  cd /opt/p24-infra && git pull && \
  cd monitoring && \
  docker compose up -d --no-deps n8n-bms4-exporter && \
  sleep 30 && \
  docker logs monitoring-n8n-bms4-exporter-1 --tail=20
"

Escalation Path

If cause is unclear: check full container logs with docker logs monitoring-n8n-bms4-exporter-1 --tail=100
If n8n API is unreachable: escalate to bms-4 ops, see n8n operations doc
If vps-i1 SSH unavailable: follow docs/playbooks/vps-i1-outage.md first

Prevention

N8N_BMS4_API_KEY is now in secrets/monitoring.env.sops (PR #1150) — will survive future vps-i1 restarts
The key in monitoring.env.sops must stay in sync with BMS4_N8N_API_KEY in secrets/n8n-bms4.env.sops — if n8n API key is rotated, both must be updated together
n8n_bms4_scrape_ok gauge is the canary — alert fires after 5 min of failure

Incident Log

Date	Issue	Duration	Root cause	Fix
2026-06-23 ~10:00 UTC	#1123	TBD	N8N_BMS4_API_KEY missing from monitoring.env.sops; also vps-i1 offline	PR #1150 adds key to monitoring SOPS; vps-i1 restart pending
2026-08-17 16:10 UTC	#6369	~25 min	Cause 5 — `bms-4-n8n-postgres-1` restarted ~16:10; n8n main (`bms-4-n8n-1`) kept a dead TypeORM pool (`Cannot use a pool after calling end on the pool`), so `/api/v1/workflows` returned 503 while `/healthz` stayed 200. Not auth (`scrape_auth_failed=0`), not a missing key.	`docker compose restart n8n` on bms-4 rebuilt the pool; `n8n_bms4_scrape_ok` back to 1 within one exporter cycle

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="n8n-bms4-exporter",
    result="success",  # "success" | "failed" | "skipped"
    detail="Missing n8n API key detected and restored — exporter config updated and container restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'n8n-bms4-exporter', 'success', 'Missing n8n API key detected and restored — exporter config updated and container restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

n8n-bms4-exporter-missing-key

Playbook: N8nBms4ExporterDown — n8n BMS-4 API exporter scrape failing

Trigger

How to Confirm

Common Root Causes and Fixes

Cause 1: N8N_BMS4_API_KEY missing from monitoring.env.sops (primary — PR #1150)

Cause 2: N8N_BMS4_API_KEY is stale / rotated in n8n

Cause 3: n8n BMS-4 is unreachable from vps-i1

Cause 4: vps-i1 monitoring stack restarted but exporter not yet running

Cause 5: n8n main holds a dead Postgres pool after `bms-4-n8n-postgres-1` restarted (#6369)

Deploying the Fix to vps-i1 (manual trigger)

Escalation Path

Prevention

Incident Log

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

n8n-bms4-exporter-missing-key

Playbook: N8nBms4ExporterDown — n8n BMS-4 API exporter scrape failing

Trigger

How to Confirm

Common Root Causes and Fixes

Cause 1: N8N_BMS4_API_KEY missing from monitoring.env.sops (primary — PR #1150)

Cause 2: N8N_BMS4_API_KEY is stale / rotated in n8n

Cause 3: n8n BMS-4 is unreachable from vps-i1

Cause 4: vps-i1 monitoring stack restarted but exporter not yet running

Cause 5: n8n main holds a dead Postgres pool after bms-4-n8n-postgres-1 restarted (#6369)

Deploying the Fix to vps-i1 (manual trigger)

Escalation Path

Prevention

Incident Log

Audit Log — Log to infra_operations

Graph View

Table of Contents

Backlinks

Cause 5: n8n main holds a dead Postgres pool after `bms-4-n8n-postgres-1` restarted (#6369)