p24-infra Docs

n8n-workflow-export

11 min read

Playbook — n8n workflow export (p24-n8n-workflow-export)

Service: scripts/n8n-workflow-export.sh, run nightly (02:00 UTC) by p24-n8n-workflow-export.timer on bms-4 (oneshot p24-n8n-workflow-export.service, user claude-runner).

What it does: fetches all n8n workflows from the bms-4 n8n API (https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows), writes each as n8n-backups/workflows/{id}-{name}.json, and commits + pushes any change to origin dev. N8N_API_KEY is read from /opt/p24-infra/bms-4/.env (key name only — never echoed).

Logs: journalctl -u p24-n8n-workflow-export.service -n 50.

Incident — “failed at line 94 (exit 1)” (#5823, 2026-08-07)

Symptom

The error handler filed [Infra] n8n-workflow-export — failed at line 94 with a Discord embed. Line 94 was the python3 heredoc that saves each workflow. The preceding count line (then line 88) had already succeeded, so the response was valid JSON.

Root cause

The response’s workflow collection was extracted with data.get('data', data). When the API returns a dict that lacks a data key (an error/envelope object such as {"message": ...} returned with HTTP 200 — which curl -sf does not treat as a failure), the , data default returned the whole dict:

  • the count line did len(<dict>) → counted keys → passed;
  • the save loop did for wf in <dict>: → iterated the string keys → wf.get('id') raised AttributeError: 'str' object has no attribute 'get' → exit 1 at line 94.

A {"data": null} response would instead hit for wf in None → TypeError — same class of crash.

Secondary defect: export BACKUP_DIR sat after the heredoc, so the python subprocess never received it and only worked because the hardcoded default matched.

Fix

scripts/n8n-workflow-export.sh now, in a single python pass:

  • extracts data['data'] explicitly (no , data fallback), tolerating a bare list;
  • treats null/missing-list as 0 workflows (exit 0 — an empty instance is not an error);
  • fails loudly with a clear, secret-free diagnostic (keys only, never values) when the response is a dict without a data key, or when the collection is a non-list — instead of crashing cryptically or silently backing up 0 workflows (which would look healthy while masking a broken export);
  • skips any non-dict entry defensively;
  • exports BACKUP_DIR before the heredoc.

If it fires again

  1. Pull the journal: journalctl -u p24-n8n-workflow-export.service -n 50. The python error now names the exact response shape (e.g. dict without a 'data' key (keys: ['code', 'message'])).
  2. keys: ['code', 'message'] / a non-list data ⇒ the API call itself failed — check the n8n container health on bms-4 and whether N8N_API_KEY in /opt/p24-infra/bms-4/.env is still valid (rotation playbook: docs/playbooks/n8n/n8n-bms4-api-key-rotation.md). This is a server/credential issue, not a script bug.
  3. A JSON-decode error ⇒ the endpoint returned non-JSON (proxy/HTML error page) — check Caddy / the n8n reverse proxy on bms-4.

Incident — “failed at line 157 (exit 1)” (#5890, 2026-08-08)

Symptom

The error handler filed [Infra] n8n-workflow-export — failed at line 157 with a Discord embed — but with no cause: the issue body said only “failed at line 157”, so triage required an SSH to bms-4 to read the journal. Line 157 was the closing ) of the SAVE_OUTPUT=$(… | python3 … ) command substitution.

Root cause

This is the **same failure class as 5823 — the n8n API returned a shape the python step rejects (error envelope / non-list / non-JSON), so it exited 1 with the secret-free diagnostic added in #5823. But that diagnostic goes to stderr → the journal, and the command substitution captured stdout only. With set -euo pipefail the substitution’s non-zero exit tripped the generic ERR trap, which reported the substitution’s own line (157) with no detail. Every recurrence also landed on a different line number (94 → 157), so the auto-filed issues never deduped.

Fix

scripts/n8n-workflow-export.sh now:

  • runs the python step under if ! SAVE_OUTPUT=$(… 2>"$SAVE_ERR_FILE" …), capturing its stderr, and on failure hands that diagnostic to on_error as a new optional DETAIL argument — so the Discord embed and the GH issue body now carry Detail: <exact response shape> (e.g. dict without a 'data' key (keys: ['code', 'message']));
  • builds the Discord JSON payload with python3 -c json.dumps so quotes/newlines in the diagnostic can’t produce invalid JSON and drop the alert;
  • uses real newlines (not literal \n) in the GH issue body;
  • passes the env-file-missing / N8N_API_KEY-missing messages through the same DETAIL arg (they were previously set into MSG and then silently overwritten by on_error).

The python parse/save logic is unchanged — the happy path is byte-for-byte identical.

If it fires again

Read the Discord embed / GH issue body first — the Detail: line now names the cause. Only pull the journal if you need the full context. Then apply the same triage as the #5823 “If it fires again” steps above (error-envelope / non-list ⇒ n8n health + N8N_API_KEY; JSON-decode ⇒ Caddy / reverse proxy). Both are server/credential conditions on bms-4, not script bugs.

Incident — “failed at line 185 (exit 1)” — transient empty response (#5913, 2026-08-09)

Symptom

The error handler filed [Infra] n8n-workflow-export — failed at line 185 at 02:04 UTC with Detail: ERROR: n8n API returned a non-JSON response: Expecting value: line 1 column 1 (char 0). char 0 = the RESPONSE body was empty. Because curl -sf had exited 0, the n8n reverse proxy returned an HTTP 2xx with an empty body — a momentary blip right after the 02:00 nightly run.

Root cause

Not a parse bug — the same server/proxy transient class the #5823 / #5890 “If it fires again” steps already describe (JSON-decode ⇒ Caddy / reverse proxy). The remaining defect was that the fetch had no retry: a single empty/non-JSON response hard-failed the whole export and fired a Discord alert + GH issue. This was the third recurrence of the class (line 94 → 157 → 185), so the noise was recurring on every brief n8n hiccup.

Fix

scripts/n8n-workflow-export.sh now wraps the fetch in a bounded retry (_n8n_fetch_with_retry, default N8N_FETCH_ATTEMPTS=4, N8N_FETCH_RETRY_DELAY_S=5):

  • each attempt accepts only a non-empty, valid-JSON body (_n8n_valid_json);
  • retries cover both a curl-level failure (network / 5xx via -f) and a 2xx-with-empty/ non-JSON body;
  • a transient blip self-heals (logged to the journal as a breadcrumb, no alert);
  • only a sustained failure (all attempts exhausted) reaches on_error, whose Detail: line then reads n8n API fetch failed after N attempt(s): … — a real outage, worth investigating;
  • the response body is never printed, so N8N_API_KEY (a -H arg only) cannot leak.

The retry helpers are unit-tested by scripts/tests/test_n8n_workflow_export.sh, which sources the script with N8N_EXPORT_LIB_ONLY=1 (a source-guard that loads the functions without running the export) and drives them with a mocked _n8n_fetch_once. The python parse/save block is unchanged.

If it fires again

The alert now means the fetch failed N times in a row — a genuine sustained condition. Triage as before: check n8n container health on bms-4, N8N_API_KEY validity, and Caddy / the reverse proxy. To tune sensitivity, adjust N8N_FETCH_ATTEMPTS / N8N_FETCH_RETRY_DELAY_S (env, read by the script).

Incident — “failed at line 268 (exit 1)” — the real root cause: the body never reached the parser (#6084, 2026-08-11)

Symptom

Same Detail: ERROR: n8n API returned a non-JSON response: Expecting value: line 1 column 1 (char 0) — but this time the #5913 fetch-retry was already deployed, so a transient empty fetch could no longer be the cause. The retry validates JSON before returning; a sustained fetch failure reaches on_error at the fetch site, not line 268 (the save step).

Root cause (the one behind ALL of #5823 / #5890 / #5913 too)

The save step was written as:

echo "$RESPONSE" | python3 - << 'PYEOF'
    data = json.load(sys.stdin)   # <-- always empty
    ...
PYEOF

python3 - << 'PYEOF' binds stdin to the heredoc program. The leading echo "$RESPONSE" | pipe is therefore silently discarded: python3 - reads its program from the heredoc, and sys.stdin is that same (now fully-consumed) stream — so json.load(sys.stdin) always read an empty stream and raised Expecting value: line 1 column 1 (char 0) on every run, regardless of what the API returned.

Proof it never worked: n8n-backups/workflows/ held 0 files since the script’s inception. The #5823 (error-envelope handling), #5890 (actionable diagnostics) and #5913 (fetch retry) fixes all targeted symptoms downstream/upstream of a parser that never saw the body — the char 0 error is emitted by json.load before any of that logic runs. curl was healthy the whole time; the “non-JSON response” was our own empty pipe, not n8n’s.

Fix

scripts/n8n-workflow-export.sh now hands the body to python via a temp file (path in RESPONSE_FILE), not a pipe the heredoc discards:

RESPONSE_FILE=$(mktemp); printf '%s' "$RESPONSE" > "$RESPONSE_FILE"
SAVE_OUTPUT=$(RESPONSE_FILE="$RESPONSE_FILE" python3 - << 'PYEOF'
    with open(os.environ['RESPONSE_FILE'], encoding='utf-8') as _fh:
        data = json.loads(_fh.read())
    ...
PYEOF
)

A temp file (not an env-var string) is required because the payload is ~1.8 MB, well over the 128 KB MAX_ARG_STRLEN an env var can hold. scripts/tests/test_n8n_workflow_export.sh Test 6 guards both halves: the discarded-pipe construct must not return, and the extracted save program must parse a body and write one file per workflow.

Credential-scrub gate added at the same time (#4691)

Because the save step now writes real files for the first time, an unscrubbed commit would ship any token an author typed into an n8n HTTP Request node header straight into git — exactly what scrub-n8n-export.py exists to prevent (the two GitHub-Actions backups already gate on it). The on-server export now runs the same gate before committing: scrub-n8n-export.py --write redacts in place, then --check re-verifies and exits 1 on any residual finding, so the script fails loudly and does not commit rather than push a secret. The scrub report is names + rule ids only (never a value), so it is safe to fold into the alert.

Redundancy note (for human follow-up, not fixed here)

There are now three n8n-workflow backups to git: .github/workflows/n8n-backup.yml (→ n8n-workflows/), .github/workflows/n8n-workflow-snapshot.yml (→ infra-src/n8n-workflows-export/), and this on-server timer (→ n8n-backups/workflows/, pushed to dev). The on-server one duplicates the GH-Actions coverage. Consider retiring the timer (or the GH-Actions jobs) in a follow-up — left as-is here to keep the fix scoped to stopping the failure.

If it fires again

A line-268 failure now means the API genuinely returned a shape the parser rejects (error envelope, non-list, or truly non-JSON) — the Detail: names it. A scrub-gate failure means a new hardcoded secret appeared in a live n8n workflow node; the real fix is converting that node to an n8n credential reference (see scrub-n8n-export.py docstring), not editing this script.

Incident — “failed at line 334 (exit 1)” — non-fast-forward push to dev (#6180, 2026-08-12)

Symptom

The error handler filed [Infra] n8n-workflow-export — failed at line 334 at 02:01 UTC. Line 334 was git push origin "HEAD:$BRANCH" ($BRANCH=dev) — the final push step.

Root cause

This was the first time the push step ever ran. Every prior incident (#5823 → #5890 → #5913 → #6084) died in the fetch or save step, so the save step never produced a commit to push. Once #6084 fixed the parser, the save step wrote real files, git diff --cached was non-empty, and execution reached the push for the first time — where the original design was broken:

  • The script did cd "$REPO_DIR" (/opt/p24-infra — the shared live checkout on bms-4, which tracks main), git add/git commit (committing the backup onto local main), then git push origin HEAD:dev.
  • dev is historical-only and has long diverged from main (see CLAUDE.md §Branching), so the local main-tip commit is not a descendant of origin/dev → git rejects the push as a non-fast-forward → exit 1.
  • Secondary defect: committing onto the shared main checkout pollutes bms-4’s live working tree with commits that never belong on main.

Fix

scripts/n8n-workflow-export.sh now commits+pushes via a new _n8n_commit_and_push helper that builds the commit in a throwaway detached git worktree checked out at origin/dev, so the commit is always a fast-forward child of the current remote tip and the shared $REPO_DIR tree / index / HEAD are never touched. Exported files are copied (not moved) into the worktree, so BACKUP_DIR is preserved for the next run; a stale worktree from a crashed run is worktree pruned first and the worktree is removed on every exit path. A git failure now hands its one-line cause to on_error as DETAIL (same actionable-diagnostic contract as 6084) instead of a bare “failed at line N”. The helper is defined above the N8N_EXPORT_LIB_ONLY guard so test_n8n_workflow_export.sh Test 7 drives it against a throwaway remote (diverged dev) and asserts: the push is a fast-forward that does not clobber dev history, the shared checkout stays on main with an unchanged HEAD and no leftover worktree, and a no-change re-run is idempotent.

If it fires again

A line-334 (_n8n_commit_and_push) failure now carries a Detail: line. git fetch origin dev failed or a push failure ⇒ check bms-4 → origin connectivity / the claude-runner git credentials and that origin/dev still exists — this path can no longer be a non-fast-forward by construction, so a push rejection means a genuinely diverged/rewritten remote dev, worth a human look. Note the standing redundancy flag from #6084: three separate jobs back n8n workflows to git; retiring this on-server timer (or the two GH-Actions jobs) remains an open follow-up.

Graph View