p24-infra Docs

n8n-atrax-dns-resolution-failure

4 min read

Playbook: n8n Atrax workflow DNS resolution failure (EAI_AGAIN)

Covers the transient DNS failure mode of the Atrax-connected n8n workflows on bms-4 (e.g. atrax, kravag-scheduled-fleet-updates, id CCx9UMdphmGficDX). This is a distinct root cause from the Atrax credential expiry failure in atrax-credential-rotation.md — triage must tell them apart before labelling anything human-action.

First seen: #7548 (2026-10-07). Prior same-workflow alerts #7443 / #7452 were the credential class (invalid_grant / “User credentials have expired”); do not assume a new alert is a repeat of those without reading the actual node error.

Symptom

Prometheus alert N8nWorkflowFailureSpike / N8nWorkflowPersistentlyFailing fires for an Atrax workflow on 54.36.123.110:5678. The failing node (get_token, or a downstream httpRequest to tronik.atrax4.com) errors with:

NodeApiError: The DNS server returned an error, perhaps the server is offline
  ... getaddrinfo EAI_AGAIN tronik.atrax4.com

EAI_AGAIN = temporary name-resolution failure (retryable). This is not an auth error and not a 4xx from Atrax — the HTTP request never leaves the box because the hostname did not resolve.

Distinguish from credential expiry (do this first)

Node error in the failed executionRoot causeAction
getaddrinfo EAI_AGAIN / “DNS server returned an error”this playbook — transient DNSusually self-recovers; see below
HTTP 400 invalid_grant / “User credentials have expired”credential expiry (Tier 3)atrax-credential-rotation.md, label human-action
HTTP 401 invalid_tokendownstream of an expired tokensame as credential expiry
HTTP 5xx / 503 from AtraxAtrax-side outagetransient; self-recovers (see #7234)

Pull the real error (read-only, on bms-4) — never assume:

docker exec bms-4-n8n-postgres-1 sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U n8n -d n8n -tAF"|" -c \
  "SELECT id,status,\"startedAt\" FROM execution_entity \
   WHERE \"workflowId\"='"'"'CCx9UMdphmGficDX'"'"' ORDER BY \"startedAt\" DESC LIMIT 15;"'
# then, for a status='error' id, grep the stored error for the resolver code:
docker exec bms-4-n8n-postgres-1 sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U n8n -d n8n -tAc \
  "SELECT data FROM execution_data WHERE \"executionId\"=<ERR_ID>;"' \
  | grep -oiE 'EAI_AGAIN|ENOTFOUND|ETIMEDOUT|invalid_grant|invalid_token'

Confirm it is a DNS-timeout window (read-only)

The containers resolve via the Docker embedded resolver 127.0.0.11, which forwards to the host systemd-resolved (127.0.0.53). When the upstream authoritative DNS for the atrax4.com CNAME is slow (a host resolvectl query tronik.atrax4.com clocked 15 s during #7548), the embedded resolver’s forward times out (~5 s) and the container gets EAI_AGAIN. The failure is typically partial — some n8n containers resolve while others don’t, and it flaps:

# Node's resolver is what n8n uses — test each container
for c in bms-4-n8n-1 bms-4-n8n-worker-1-1 bms-4-n8n-worker-2-1 bms-4-n8n-worker-3-1; do
  printf '%s: ' "$c"
  docker exec "$c" node -e 'const t=Date.now();require("dns").lookup("tronik.atrax4.com",
    (e,a)=>console.log((e?"ERR "+e.code:"OK "+a)+" in "+(Date.now()-t)+"ms"))'
done
# Prove it is specific to this name (not a general container DNS outage):
docker exec bms-4-n8n-1 node -e 'require("dns").lookup("github.com",(e,a)=>console.log(e?e.code:"OK"))'
# Host-side: a slow (multi-second) answer here is the tell. A fresh public-resolver lookup is the control:
resolvectl query tronik.atrax4.com     # watch the "acquired ... in N s" line
dig +tries=1 +time=5 tronik.atrax4.com @8.8.8.8 +stats | grep -i 'Query time'

A general container DNS outage would fail github.com too; this failure mode fails only tronik.atrax4.com and only on some containers.

Resolution

  1. Default: wait — it self-recovers. EAI_AGAIN is transient. In #7548 the error window was ~11:06–11:30 UTC and executions resumed succeeding on their own once upstream resolution sped back up; the alert auto-closes when failures stop. Confirm recovery by watching the execution trend return to success:

    docker exec bms-4-n8n-postgres-1 sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U n8n -d n8n -tAF"|" -c \
  "SELECT date_trunc('"'"'minute'"'"',\"startedAt\") m, status, count(*) FROM execution_entity \
   WHERE \"workflowId\"='"'"'CCx9UMdphmGficDX'"'"' AND \"startedAt\" > now() - interval '"'"'30 min'"'"' \
   GROUP BY m,status ORDER BY m DESC;"'

    Do not label human-action and do not rotate credentials for this failure mode.

  2. If one container stays stuck (keeps timing out at ~5 s while siblings resolve) after the upstream has recovered, its embedded-resolver forward state is wedged. Clear it with a targeted restart of only that container (requires root / docker access; blast radius = that one container). In queue mode the schedule triggers run on the worker containers, so a wedged main bms-4-n8n-1 does not by itself block these workflows — weigh the UI/webhook interruption before restarting the main instance:

    # PLAYBOOK: n8n-atrax-dns-resolution-failure.md
ssh root@54.36.123.110 "cd /opt/p24-infra/bms-4 && docker compose restart n8n-worker-1"

  3. If the upstream itself is persistently slow/broken (every container + the host resolver fail to resolve atrax4.com for an extended period), that is an Atrax-side DNS provider problem — outside p24-infra. A durable mitigation would be pinning the IP via extra_hosts in bms-4/docker-compose.yml, but that is fragile if Atrax re-IPs and should only be a deliberate, human-reviewed change, not an alert-response reflex.

References

Graph View