p24-infra Docs

n8n-workflow-debugging-lessons

17 min read

n8n workflow debugging — general lessons

Reusable, general lessons for building or debugging any n8n workflow on the bms-4 instance — not tied to one workflow. Extracted from the #7009 “Dispo — Logistics Order Extraction” build (2026-09-14 → 2026-09-21), which hit all five in a single debugging session while getting the workflow to actually execute successfully after activation. See docs/logistics-order-extraction-pipeline-operations.md for that pipeline’s own design; this doc captures what generalizes.

Related: docs/playbooks/n8n-http-request-quirks.md, docs/playbooks/n8n-alert-router-empty-webhook.md (the pdf-parse/NODE_FUNCTION_ALLOW_EXTERNAL startup crash-loop class — a sixth #7009 finding, but general enough it already lives there as “Variant C”), .claude/commands/n8n-nodes.md.

1. Activation validates ALL credentials in a workflow, not just the trigger’s

Rule: n8n refuses to activate a workflow if any node’s credential is dead/misconfigured — even a node on a completely unrelated branch that the trigger never reaches on a given run. A single stale credential can block activation of an entire multi-purpose workflow.

Why: n8n’s activation check walks every node in the workflow graph and validates every credential reference, not just the ones reachable from the active trigger. This is a whole-workflow gate, not a per-branch one.

What happened (#7009): the Dispo Stage-1/Stage-2 branches were built additively inside the existing 57-node IMAP_Inbox_Sync__AI_Classifier workflow (82xi1IcZPf0Efj5z). Activating it failed — not because of anything in the new branches, but because the original classifier’s Fetch IMAP Emails node used a dead imap credential (sn3que6TvNPKAXbr, a plain Gmail account password; Gmail’s IMAP no longer accepts that auth mode). That one dead, never-exercised credential blocked activation of the whole workflow, Dispo branch included.

Fix applied: split the workflow. The Dispo branch (28 nodes, its own independent credentials: gmailOAuth2, openAiApi, aws, postgres, httpHeaderAuth) was confirmed to have zero cross-branch connections to the classifier’s 29 nodes, then extracted into its own standalone workflow, Dispo — Logistics Order Extraction, id 1PWIIWmodYYSYJNr. The original classifier workflow was restored to its original 29 nodes. Both now activate/fail independently — the classifier’s IMAP credential was later fixed with a fresh App Password, and both workflows are active: true as of 2026-09-17/21.

Actionable takeaway:

  • Before combining unrelated automation into one workflow “for convenience,” ask: does any node here use a credential I’m not confident is alive? If yes, keep them as separate workflows.
  • If you inherit a combined workflow that won’t activate, don’t assume the new branch you just added is the cause — check every credential in the whole graph, including old, unrelated, never-triggered nodes.
  • A useful diagnostic: GET /api/v1/workflows/{id} and cross-check every credentials.*.id against GET /api/v1/credentials (or the n8n UI’s credential list) for ones flagged as expired/ broken, before touching workflow logic at all.

2. httpRequest + nodeCredentialType: aws cannot sign requests to non-AWS S3 hosts

Rule: n8n’s generic httpRequest node with authentication: predefinedCredentialType / nodeCredentialType: aws only works against real *.amazonaws.com endpoints. It cannot sign requests to an S3-compatible-but-not-AWS host (Wasabi, MinIO, Backblaze B2, etc.), even though the AWS credential type exposes customEndpoints/s3Endpoint fields that look like they should support this.

Symptom:

AuthorizationHeaderMalformed ... Credential service is , but expected s3

Why: the SigV4 “service” (s3) inference for a custom endpoint is only implemented in n8n’s dedicated S3 node type (n8n-nodes-base.s3). The generic httpRequest node + AWS credential combination does not consult customEndpoints/s3Endpoint at all for this purpose — setting them on the credential is a silent no-op when the calling node is httpRequest, not s3.

What happened (#7009): Dispo Upload To Wasabi used httpRequest with an aws-type credential pointed at s3.eu-central-2.wasabisys.com via customEndpoints. Every execution failed with the error above. An initial fix attempt (setting the endpoint fields more explicitly on the credential) did nothing — confirming the fields are ignored for this node type.

Working pattern — sign the request manually in a Code node, then httpRequest with authentication: none. This repo already had this pattern live and proven in n8n-workflows/Brand_-_HeyGen_to_LinkedIn_Video_Pipeline_heygen-to-linkedin-v1.json’s “Sign Wasabi Upload” node before #7009 hit the same wall — the fix was to copy it, not invent a new approach. Current working version, from n8n-workflows/Dispo_Logistics_Order_Extraction_1PWIIWmodYYSYJNr.json’s Dispo Sign Wasabi Upload node (Code, runOnceForEachItem):

const crypto = require('crypto');
const ak = $env.WASABI_ACCESS_KEY || '';
const sk = $env.WASABI_SECRET_KEY || '';
const region = 'eu-central-2';
const bucket = 'p24-infra';
const oKey = $json.objectKey;
const host = `s3.${region}.wasabisys.com`;
const iso = (new Date()).toISOString();
const amzDate = iso.slice(0,4)+iso.slice(5,7)+iso.slice(8,10)+'T'+iso.slice(11,13)+iso.slice(14,16)+iso.slice(17,19)+'Z';
const ds = amzDate.slice(0,8);
const sha256 = 'UNSIGNED-PAYLOAD';
const cUri = `/${bucket}/${oKey}`;
const cHeaders = `host:${host}\nx-amz-content-sha256:${sha256}\nx-amz-date:${amzDate}\n`;
const sh = 'host;x-amz-content-sha256;x-amz-date';
const cr = ['PUT', cUri, '', cHeaders, sh, sha256].join('\n');
const cs = `${ds}/${region}/s3/aws4_request`;
const sts = ['AWS4-HMAC-SHA256', amzDate, cs, crypto.createHash('sha256').update(cr).digest('hex')].join('\n');
const h = (k,d,e) => crypto.createHmac('sha256', k).update(d,'utf8').digest(e||'buffer');
const sigK = h(h(h(h('AWS4'+sk, ds), region), 's3'), 'aws4_request');
const sig = h(sigK, sts, 'hex');
const auth = `AWS4-HMAC-SHA256 Credential=${ak}/${cs},SignedHeaders=${sh},Signature=${sig}`;
const putUrl = `https://${host}${cUri}`;
return [{ json: { ...$json, putUrl, auth, amzDate, sha256 }, binary: $input.item.binary }];

Downstream httpRequest node: authentication: none, method PUT, url: {{ $json.putUrl }}, headers set manually — Authorization: {{ $json.auth }}, x-amz-date: {{ $json.amzDate }}, x-amz-content-sha256: {{ $json.sha256 }} — body from the item’s binary data.

Actionable takeaway: for any Wasabi/non-AWS-S3 upload or download in n8n, skip the aws credential + httpRequest combination entirely. Go straight to the manual-SigV4 Code-node pattern above (WASABI_ACCESS_KEY/WASABI_SECRET_KEY env vars, already deployed on bms-4 — no new secret needed for a new workflow that follows this pattern).

3. exists treats an explicit JSON null as present — omit the key instead

Rule: n8n’s string-filter exists operator (used in IF/switch conditions) evaluates to true for a key whose value is JSON null. It only evaluates to false for a key that is truly absent from the object. A Code node that wants a downstream exists check to fail must omit the key from its returned json, never set it to null.

What happened (#7009): Dispo Fan Out Attachments produced one marker item per email that had zero attachments (so the email still survives the pipeline even with nothing to upload). The original version of this marker item explicitly set filename/mimetype/objectKey to null. A downstream exists check on objectKey (meant to route “has an attachment” vs. “no attachment” items) treated the null values as present, so zero-attachment emails were incorrectly routed into the Wasabi-upload branch, which then failed with URL parameter must be a string, got undefined trying to build a URL from an undefined objectKey.

Fix applied — in Dispo Fan Out Attachments, the marker item is built by spreading only the email metadata, never touching the attachment-only keys:

const out = [];
for (const item of $input.all()) {
  const meta = item.json;
  const binaryKeys = Object.keys(item.binary || {}).filter(k => k.startsWith('attachment_'));
  if (binaryKeys.length === 0) {
    out.push({ json: { ...meta } });   // no filename/mimetype/objectKey keys at all
    continue;
  }
  for (const key of binaryKeys) {
    const bin = item.binary[key];
    const filename = bin.fileName || key;
    const objectKey = `dispo-ecotrans/${meta.gmail_message_id}/${filename}`;
    out.push({ json: { ...meta, filename, mimetype: bin.mimeType || 'application/octet-stream', objectKey }, binary: { data: bin } });
  }
}
return out;

Actionable takeaway: when a Code node needs to signal “this field does not apply to this item” for a downstream exists gate, don’t write field: null — delete the key (or never add it). If you must merge an object that might carry nulls from elsewhere, strip them first: Object.fromEntries(Object.entries(obj).filter(([, v]) => v !== null)).

4. A SOPS-deployed env var isn’t live in n8n containers until it’s ALSO in docker-compose

Rule: having a key in a secrets/*.env.sops file that gets synced to a server is necessary but not sufficient. Every n8n/n8n-worker-* service in bms-4/docker-compose.yml has its own environment: block, and a var is only visible inside that container if it has an explicit - VAR=${VAR} line there too. Check all three layers before assuming a var is live:

  1. secrets/n8n-bms4.env.sops (or whichever file) — the key exists, canary-decrypts.
  2. bms-4/docker-compose.yml — the same key has a - VAR=${VAR} line in every n8n/ n8n-worker-* service’s environment: block (there are 4: n8n, n8n-worker-1/2/3).
  3. docker exec <container> printenv | grep VAR on the live server — confirms the previous two steps actually got applied (git pull && start.sh on bms-4 recreates the containers).

What happened (#7009): P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL was present in secrets/n8n-bms4.env.sops but had never been added to any of the 4 n8n service environment: blocks. The Dispo error-notification httpRequest nodes (the ones meant to post to Discord when something else fails) themselves failed with URL parameter must be a string, got undefined — because the URL they were reading was an unset env var. This masked the real, separate bug (the Wasabi SigV4 signing failure in lesson 2) behind a confusing secondary failure in the error handler itself; the real root cause only surfaced after inspecting raw execution data instead of trusting the top-level reported error.

Fix applied — added to all 4 environment: blocks in bms-4/docker-compose.yml:

- P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL=${P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL}  # #7009 -- dispo Stage-1/Stage-2 error-notification nodes (was missing entirely, silently masking real errors behind a URL-undefined failure)

then git pull && start.sh on bms-4 to recreate the containers.

Actionable takeaway:

  • Whenever you add a new env var to a SOPS file that an n8n Code/httpRequest node needs to read ($env.VAR or {{ $env.VAR }}), immediately also add it to every relevant service block in bms-4/docker-compose.yml in the same change — don’t treat these as separate steps done later.
  • When an error-notification/error-handler node is the thing that shows up as the reported failure, always suspect this class of bug first — a handler that fails on its own missing config is a classic way to hide the real upstream error. Pull the raw execution JSON (GET /api/v1/executions/{id}?includeData=true) rather than trusting the summary error.

5. Escape sequences can get mangled when generating jsCode through a shell

Rule: when a Code node’s jsCode string is generated by an external script (Python, PowerShell, etc.) invoked through a shell — especially python3 -c "..." or a heredoc nested inside other quoting — the literal two-character sequence \n inside a JS string literal (e.g. .join('\n')) can silently collapse into an actual newline byte. The resulting JSON is often still syntactically valid to write to disk, but the JS it contains breaks with a cryptic SyntaxError: Invalid or unexpected token — and n8n’s execution error data gives no node name or line number for a Code-node parse failure, making it very hard to localize.

What happened (#7009): the Wasabi manual-signing fix (lesson 2) was itself first shipped with this bug — the jsCode was built via a python3 -c "..." invocation nested inside bash, which collapsed the \n inside .join('\n') and similar string literals into real newlines. The signing node failed with an opaque syntax error and no indication of which node or line was at fault. Diagnosing it required comparing byte-for-byte against a known-good version instead of reading the error.

Mitigations:

  • Write the code-generation script to an actual .py/.ps1 file — never inline it via -c or a heredoc nested inside other shell quoting. A file’s contents aren’t subject to an extra layer of shell-escaping.
  • In Python, use a raw string (r"""...""") for any JS template containing backslash escapes, so \n in the template stays the literal two characters until it’s the JS engine’s turn to interpret it.
  • Before using the generated string, verify it directly: compare content.count(chr(10)) (actual newline bytes) against the count of literal \n occurrences you expect in the template — a mismatch means escaping was lost somewhere in the pipeline.
  • Round-trip-test the exact payload against a disposable scratch n8n workflow (webhook in, same Code node, PUT out) and confirm success before touching a production workflow — this is what caught the corrected version in #7009’s follow-up fix.
  • Prefer separate, explicit full-string templates over generic .replace("{PLACEHOLDER}", value) substitution for short JS snippets. A placeholder’s own {/} characters can collide with a real JS object-literal brace sitting right next to it in the template, and an empty/partial replacement can silently delete a brace the code actually needs — a second, distinct way for templating to produce broken JS that looks fine until it runs.

6. Use an isolated worktree for background/agent git work — even a “quick fix”

Rule: CLAUDE.md’s tmp/wt-{issue} worktree-isolation convention for background-agent git work applies to every git operation done outside an interactive foreground session — not just full “implement this issue” tasks. A session working directly in the shared default checkout (C:\devWork\p24-infra) can have its working tree and branch state changed out from under it by another concurrent process using the same checkout.

What happened: during the #7009 debugging chain, the shared checkout was used directly (not through an isolated tmp/wt-* worktree) across several quick sequential fix branches in the same session — fix/7009-dispo-runtime-errors, fix/5911-etop-technician-accounts-v2, fix/7009-wasabi-sigv4-signing, and others, interleaved with checkout/cherry-pick/commit operations on the one shared tree (visible in git reflog around 2026-09-21 18:00–18:46). Later in the same day, a commit from another automated process landed on the checkout mid-session, unrelated to the branch actively being worked. Because the checkout is shared, a “just one small fix” git operation is never actually isolated from whatever else might be running against the same directory at the same time.

Actionable takeaway:

  • Always create an isolated worktree before doing git work for background/agent tasks, per CLAUDE.md’s Agent Workflow — Worktree Branch Isolation: 
    git worktree add tmp/wt-{issue}-{slug} -b {branch-name} origin/main
    even when the change feels small enough to “just quickly commit in the main checkout.” The convention exists specifically because the shared checkout is not exclusive to one session. This playbook itself was written from tmp/wt-7009-playbook, not the shared checkout.
  • If you’re mid-session in the shared checkout and notice unexpected file/branch state (git status shows something you didn’t touch, git log shows a commit you didn’t make), stop, git stash/note what’s yours, and move the rest of the session into an isolated worktree before continuing — don’t assume you’re the only writer.

7. Inline error-output reporters must read the error via a Code node — $json alone loses it

Rule: when a node has onError: continueErrorOutput, the item delivered on the error output (main index 1) does not reliably expose the error in a form a downstream httpRequest reporter can read with a bare $json expression. Two independent traps combine to produce a content-free error report:

  1. The error lives in $json.error, and its type varies by failing-node category. A Code node’s error-output item carries $json.error as a string ("message [line N]"); an httpRequest/API node carries $json.error as an object ({message, description, name, code, status, ...}). A reporter that does JSON.stringify($json) without pulling .error out — or that reads $json.gmail_message_id expecting the original item’s fields — gets garbage, because what survives to the reporter is not the original item shape.
  2. A reporter chained after another httpRequest node reads that node’s response, not the error. If the topology is errored node → Discord (httpRequest) → GH Issue (httpRequest), the GH-Issue node’s $json is the Discord webhook’s response (empty / 204), so its body renders as {} and its title as gmail_message_id unknown. This is the #7429 bug: every Stage-2 error filed an empty ## Error\n{} GitHub issue (#7429, #7437) with zero diagnostic value — the real error (Bad request … file badly formatted or corrupted, images routed down the PDF path) was thrown away.

Working pattern — a Code “Build Report” normalizer node, fanned out to the reporters:

[errored nodes] --error(main#1)--> [Build Report (Code, runOnceForEachItem)]
                                        |-- main#0 --> [Discord httpRequest]   (reads $json.err_*)
                                        `-- main#0 --> [GH Issue httpRequest]  (reads $json.err_*)
  • The Code node is the only place that can robustly reach the error — a Code node downstream of any error output reliably sees $json.error (verified by round-trip probe for both Code and httpRequest failures, including a 2 MB attachment_pdf_b64 field — the full item survives to a Code node). It normalizes string-or-object into clean fields (err_title, err_body, gmail_message_id, has_detail, failed_node) and strips oversized fields so the report isn’t bloated by base64 payloads.
  • Fan the Build Report node out to Discord and GH Issue in parallel (both in the same main[0] array) so each reporter reads the normalizer’s clean output directly via $json — never chain reporter→reporter, or the second reads the first’s HTTP response (trap 2). This also makes the two reports independent (a Discord outage no longer blocks the GH issue).
  • Build the Code node’s jsCode from a committed .py/file, never a nested python3 -c — the .join('\n') escape-collapse of lesson 5 applies here too.

Diagnosing this class: the error-filing execution shows as success in the executions list (because continueErrorOutput lets the workflow complete), so you cannot filter by status=error. Pull GET /api/v1/executions/{id}?includeData=true and inspect the reporter node’s recorded input (runData[<reporter>][0].data.main[0][0].json) — an empty {} there is the signature.

Fixed (#7429, 2026-10-05): workflow 1PWIIWmodYYSYJNr on bms-4, added Dispo Stage{1,2} Error - Build Report Code nodes; applied live via the n8n REST API (the committed infra-src/n8n-workflows-export/ snapshot is a read-only daily export and picks the change up automatically).

8. Don’t assume an attachment’s content type from the branch it fell down — sniff it

Rule: when a workflow splits attachments by a coarse test (e.g. an IF on filename extension) and treats everything on the else-branch as one fixed type, any attachment that doesn’t match the IF — but also isn’t the assumed else-type — gets force-fed to a downstream API as the wrong MIME and fails. Detect the real content type from the binary’s own mimeType with a magic-byte fallback, branch the API payload on it, and guard genuinely-unsupported types away from the API call entirely.

What happened (#7450): the Dispo Stage-2 attachment path split only Excel vs. everything else, and the else-branch (Dispo Pass Through (PDF/other) → Dispo PDF To Base64 → Dispo Extract Order (PDF)) hardcoded the OpenAI /v1/responses call as { type: "input_file", file_data: "data:application/pdf;base64," + … }. A real inbound order with an image attachment (bare-UUID filename, no extension, attachment_is_excel: false, ~32 KB) was base64-encoded and sent as application/pdf bytes, so OpenAI rejected it with exactly the lesson-7 class of error surfaced by the new reporter:

Bad request — please check your parameters — The file you uploaded is badly formatted or corrupted.

Fix applied (live to 1PWIIWmodYYSYJNr on bms-4, n8n REST API, 2026-10-05):

  1. Dispo PDF To Base64 (Code) — resolve the binary with the lesson-2/§7376 getBinaryDataBuffer pattern, then sniff the first 12 bytes for a MIME: %PDF→application/pdf, FF D8 FF→image/jpeg, 89 50 4E 47→image/png, GIF8→image/gif, RIFF…WEBP→image/webp; fall back to the binary’s declared mimeType. Emit attachment_mime, a derived attachment_filename (synthesize an extension when the source filename has none), and a boolean attachment_supported.
  2. Dispo Extract Order (PDF) (httpRequest jsonBody) — choose the user-content block by MIME: image/* → { type: "input_image", image_url: "data:<mime>;base64,…" }, application/pdf → { type: "input_file", filename, file_data: "data:application/pdf;base64,…" }. gpt-4o-mini accepts both. Done with a single ternary inside the existing ={{ ({ … }) }} object expression — no IIFE — so the long default-prompt string literal is left byte-for-byte untouched (avoids the lesson-5 escape-collapse on re-authoring).
  3. New guard Dispo Is Supported Attachment? (IF v2.2) + Dispo Skip Unsupported Attachment (NoOp) between the Code node and Dispo Fetch Prompt (PDF) — attachment_supported TRUE continues to extraction, FALSE routes to the NoOp (silently dropped, visible in the execution graph), so docx / unidentifiable bytes never reach OpenAI and the error class cannot recur.

Actionable takeaways:

  • A two-way IF on a filename/extension is a router, not a type guarantee. The else-branch is “not-X”, never “therefore-Y” — read the real bytes before telling a downstream API what they are.
  • The IF boolean operator shape that works on n8n 2.2 is {"type":"boolean","operation":"true","singleValue":true} with leftValue: "={{ $json.flag }}" and rightValue: "" — copy it from a known-good node rather than hand-rolling it; a wrong operator silently routes everything to FALSE and would strand valid PDFs too.
  • Build the Code node’s jsCode from a committed file and node --check the extracted payload (wrap in (async function(){ … })) before the PUT — same lesson-5 discipline; it catches a typo without a scratch workflow. When touching only part of an httpRequest expression, string-replace that sub-object instead of re-typing the whole jsonBody — the untouched parts can’t regress.
  • The live PUT is the real validation (n8n applies it as one transaction); a 502 from the gateway means n8n was mid-restart and the change did not land — re-GET to confirm node count before retrying, don’t assume a partial apply.

See also

  • docs/logistics-order-extraction-pipeline-operations.md — the pipeline this session’s lessons were extracted from.
  • .claude/commands/n8n-nodes.md — node-type reference; httpRequest/Code node gotchas.
  • docs/playbooks/n8n-http-request-quirks.md — other httpRequest-specific quirks.
  • docs/playbooks/n8n-alert-router-empty-webhook.md — the NODE_FUNCTION_ALLOW_EXTERNAL eager-require()-at-startup crash-loop class (Variant C), hit by the same #7009 session (pdf-parse/DOMMatrix, #7078).

Graph View

Backlinks