p24-infra Docs

n8n-bms4-worker-metrics-not-bound

3 min read

Playbook — bms-4 n8n worker up but N8nWorkerDown (metrics HTTP server not bound)

Issue: #6874 · Server: bms-4 (54.36.123.110) · Service: n8n 2.26.3 queue-mode (main + 3 workers on host ports 5679/5680/5681 → container 5678) · Alert: N8nWorkerDown

Symptom

N8nWorkerDown fires for one worker instance (e.g. 54.36.123.110:5680) even though:

  • docker ps shows the worker container Up with the expected port mapping,
  • RestartCount=0, no OOM, no crash-storm (this is not N8nHighFailureRatio — see n8n-bms4-worker-crash-storm.md),
  • the worker is actively processing queue executions (Worker started execution … in logs) — queue work goes over Redis/Bull, not HTTP, so it is unaffected.

Only Prometheus scrapes fail: the worker’s embedded HTTP server (which serves /metrics and /healthz on container port 5678) failed to bind at startup — despite the log line n8n worker server listening on port 5678 printing normally. Observed once after a full-stack recreate (docker compose up -d) at a moment when one worker’s HTTP listener silently never came up.

Diagnosis (read-only, ~30s)

# 1. Which host port is dead? (000 = connection fails; 200 = healthy)
for p in 5679 5680 5681; do
  echo -n "port $p /metrics: "; curl -s -o /dev/null -w "HTTP %{http_code}\n" -m3 http://localhost:$p/metrics
done
 
# 2. Host socket IS listening (docker-proxy), so the container up-check passes but scrape 000s:
ss -ltn | grep -E ':567[89]|:568[01]'
 
# 3. Definitive: inside the bad worker, the :::5678 HTTP listener is MISSING.
#    A healthy worker shows BOTH ':::5678' (HTTP) and '127.0.0.1:5679' (JS Task Broker);
#    the broken one shows ONLY '127.0.0.1:5679'.
docker exec bms-4-n8n-worker-2-1 sh -c 'netstat -ltn 2>/dev/null | grep -E ":5678|:5679"'
docker exec bms-4-n8n-worker-1-1 sh -c 'netstat -ltn 2>/dev/null | grep -E ":5678|:5679"'   # healthy reference

Map host port → container name: 5679→worker-1, 5680→worker-2, 5681→worker-3.

Remediation — restart the single affected worker

No need to recreate the whole stack; a plain restart re-binds the HTTP server.

cd /opt/p24-infra/bms-4
docker compose restart n8n-worker-2        # substitute the affected worker
 
# Verify — HTTP server takes ~20–30s to bind after restart:
for i in $(seq 1 20); do
  code=$(curl -s -o /dev/null -w "%{http_code}" -m3 http://localhost:5680/metrics)
  echo "attempt $i: HTTP $code"; [ "$code" = "200" ] && break; sleep 2
done
docker exec bms-4-n8n-worker-2-1 sh -c 'netstat -ltn | grep :5678'   # expect ':::5678 LISTEN'

A healthy worker returns HTTP 200 with ~108 n8n_ metric lines. N8nWorkerDown auto-resolves on the next Prometheus scrape.

Notes

  • Because queue processing is unaffected, this alert does not indicate lost workflow executions — it is purely a metrics/observability gap for the affected worker while it was mis-bound.
  • If a restart does not re-bind :5678 (still HTTP 000 after ~40s), escalate to a full docker compose up -d --force-recreate of the workers, capturing logs first per the crash-storm playbook’s “capture evidence” step.

Graph View