Playbook: N8nBms4ExporterDown — n8n BMS-4 API exporter scrape failing

Trigger

Prometheus alert N8nBms4ExporterDown fires:

n8n_bms4_scrape_ok == 0 for 5+ minutes
n8n-bms4-exporter:9226 returns metrics but with scrape_ok=0
Per-workflow name metrics absent from Grafana n8n dashboard

How to Confirm

# On vps-i1: check exporter logs
ssh root@217.154.82.162 "docker logs monitoring-n8n-bms4-exporter-1 --tail=50"
 
# Expected on failure: "BMS-4 scrape failed: 401 Client Error" or "KeyError: N8N_BMS4_API_KEY"
# Expected on success: "Collected BMS-4: N workflows (M active), 1h=X series, 24h=Y series"
 
# Check metrics directly
ssh root@217.154.82.162 "curl -s http://localhost:9226/metrics | grep n8n_bms4_scrape_ok"
# Healthy: n8n_bms4_scrape_ok{instance="n8n.bms-4.infra.zintegrowana.online"} 1
# Failing: n8n_bms4_scrape_ok{...} 0
 
# Verify n8n BMS-4 API is reachable from vps-i1
ssh root@217.154.82.162 "curl -s -o /dev/null -w '%{http_code}' https://n8n.bms-4.infra.zintegrowana.online/healthz"
# 200 = n8n is up; if not 200 — n8n BMS-4 itself is down (different issue)
 
# Verify the env var is set in monitoring .env on vps-i1
ssh root@217.154.82.162 "grep N8N_BMS4_API_KEY /opt/p24-infra/monitoring/.env | cut -c1-30"
# Should print first 30 chars of the variable (do NOT print the full value)

Common Root Causes and Fixes

Cause 1: N8N_BMS4_API_KEY missing from monitoring.env.sops (primary — PR #1150)

Symptom: Container logs show KeyError: 'N8N_BMS4_API_KEY' or container fails to start. grep N8N_BMS4_API_KEY /opt/p24-infra/monitoring/.env returns nothing.

Root cause: secrets/monitoring.env.sops did not contain N8N_BMS4_API_KEY. The secrets-sync.yml vps-i1 job only deploys monitoring.env.sops to vps-i1 — any key missing from that file will be absent from the deployed .env.

Fix (already applied in PR #1150): Verify N8N_BMS4_API_KEY is now in monitoring.env.sops:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "N8N_BMS4_API_KEY"

If missing — re-add it (see CLAUDE.md §Secrets for SOPS add procedure):

# Get value from n8n-bms4.env.sops (key is BMS4_N8N_API_KEY there)
# Follow CLAUDE.md SOPS add procedure; use --filename-override "secrets/monitoring.env.sops"
# when encrypting if path_regex causes "no matching creation rules found" error

Cause 2: N8N_BMS4_API_KEY is stale / rotated in n8n

Symptom: Container logs show 401 Client Error: Unauthorized. grep N8N_BMS4_API_KEY /opt/p24-infra/monitoring/.env returns a non-empty value. Alert N8nBms4ExporterAuthFailure is firing alongside N8nBms4ExporterDown (#3557) — this distinct alert (n8n_bms4_scrape_auth_failed == 1) means the failure is a 401/403, so it is this cause and not connectivity. A redeploy/secrets-sync will NOT fix it — the key must be regenerated in n8n and rotated in SOPS.

Fix:

Log in to n8n at https://n8n.bms-4.infra.zintegrowana.online → Settings → n8n API
Generate a new API key
Update secrets/n8n-bms4.env.sops (key BMS4_N8N_API_KEY) and secrets/monitoring.env.sops (key N8N_BMS4_API_KEY) with the new value — both must be identical
Commit and push to dev — secrets-sync.yml auto-deploys to both vps-i1 and bms-4
After deploy: restart only the exporter: docker compose up -d --no-deps n8n-bms4-exporter

Cause 3: n8n BMS-4 is unreachable from vps-i1

Symptom: curl https://n8n.bms-4.infra.zintegrowana.online/healthz from vps-i1 times out or returns 5xx.

Fix: This is an n8n/bms-4 issue, not an exporter issue. Check:

bms-4 Docker stack: ssh root@54.36.123.110 "docker compose ps" (in /opt/p24-infra/bms-4/)
Traefik logs on bms-4 for TLS or routing errors
See docs/servers/p4-ovh-bms-4-ns3101999-operations.md

Cause 4: vps-i1 monitoring stack restarted but exporter not yet running

Symptom: Alert fires right after vps-i1 restart. All monitoring containers were just restarted.

Fix: Wait for the 5-minute alert holddown to expire. If still firing after 10 min:

ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose ps n8n-bms4-exporter"
# If Exit/Restart: docker compose up -d --no-deps n8n-bms4-exporter && docker compose logs --tail=30 n8n-bms4-exporter

Deploying the Fix to vps-i1 (manual trigger)

When vps-i1 is back up after a restart and secrets-sync.yml auto-run failed:

# From local workstation
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=vps-i1
# Then wait for it to complete:
gh run list --repo radieu/p24-infra --workflow secrets-sync.yml --limit 2

Or directly via SSH if CI is not available:

ssh root@217.154.82.162 "
  cd /opt/p24-infra && git pull && \
  cd monitoring && \
  docker compose up -d --no-deps n8n-bms4-exporter && \
  sleep 30 && \
  docker logs monitoring-n8n-bms4-exporter-1 --tail=20
"

Escalation Path

If cause is unclear: check full container logs with docker logs monitoring-n8n-bms4-exporter-1 --tail=100
If n8n API is unreachable: escalate to bms-4 ops, see n8n operations doc
If vps-i1 SSH unavailable: follow docs/playbooks/vps-i1-outage.md first

Prevention

N8N_BMS4_API_KEY is now in secrets/monitoring.env.sops (PR #1150) — will survive future vps-i1 restarts
The key in monitoring.env.sops must stay in sync with BMS4_N8N_API_KEY in secrets/n8n-bms4.env.sops — if n8n API key is rotated, both must be updated together
n8n_bms4_scrape_ok gauge is the canary — alert fires after 5 min of failure

Incident Log

Date	Issue	Duration	Root cause	Fix
2026-06-23 ~10:00 UTC	#1123	TBD	N8N_BMS4_API_KEY missing from monitoring.env.sops; also vps-i1 offline	PR #1150 adds key to monitoring SOPS; vps-i1 restart pending

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="n8n-bms4-exporter",
    result="success",  # "success" | "failed" | "skipped"
    detail="Missing n8n API key detected and restored — exporter config updated and container restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'n8n-bms4-exporter', 'success', 'Missing n8n API key detected and restored — exporter config updated and container restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

n8n-bms4-exporter-missing-key