p24-infra Docs

n8n-backup-secret-scrubbing

10 min read

n8n backup — hardcoded credential scrubbing

Owner: dev-coder (script + workflows) · sys-admin/infra-task (live n8n fixes) · secret-manager (rotation) Related: #4691, #4888, #5707, static-api-key-incident-rotation.md Added: 2026-07-30 · mojibake guard added 2026-08-01 (#4888) · jsCode credential-literal rule added 2026-08-06 (#5707)

What this covers

Two GitHub Actions export live n8n workflows as JSON and commit them:

WorkflowExports toSchedule
.github/workflows/n8n-backup.ymln8n-workflows/daily 04:00 UTC
.github/workflows/n8n-workflow-snapshot.ymlinfra-src/n8n-workflows-export/daily 04:00 UTC

n8n stores an HTTP Request node’s headers as plain {name, value} pairs in the workflow document. So if anyone types a token straight into an Authorization header in the n8n UI instead of attaching a credential, that token is in the exported JSON and gets committed on the next nightly run.

That is exactly what happened in #4691: Bearer sb_… service-role keys across 4 Supabase-targeting workflows, a Bearer gho_… GitHub OAuth token, an opaque static key for the p24-auth CF Worker, and scraped Cookie: session jars for the Kravag portal — all in git.

scripts/scrub-n8n-export.py is the gate that stops this recurring.

How the gate works

Both workflows now run, after the export and before git add:

python3 scripts/scrub-n8n-export.py --write --json /tmp/n8n-scrub-report.json <export-dir>/
python3 scripts/scrub-n8n-export.py --check <export-dir>/       # hard verification
  • --write redacts in place. The backup still commits — a redacted export is useful, a leaked token is not. This is deliberately not a blocking check: the live workflow is the thing that is wrong, and failing the nightly backup every night until someone fixes n8n would just mean no backups.
  • --check runs afterwards as a hard gate. If anything credential-shaped survived the scrub the job fails and nothing is committed.
  • When --write finds anything, n8n-backup.yml fires the repo’s standard error path — a Discord embed on P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL plus a rolling GitHub issue titled [Infra] n8n — hardcoded credentials in live workflow nodes (commented on, not re-opened, each night).

What gets redacted

  1. Sensitive header values — Authorization, Proxy-Authorization, Cookie, Set-Cookie, X-Api-Key, apikey, X-N8N-Api-Key, X-Auth-Token, token, secret, and friends (full list: SENSITIVE_HEADERS in the script).
  2. Known secret shapes anywhere in the document — including inside jsCode, request bodies and URLs: sb_secret_…, gh[pousr]_…, github_pat_…, sk-ant-…, sk-…, 3-segment JWTs, AGE-SECRET-KEY-1…, AIzaSy…, xox[baprs]-…, glpat-…, SG.…, re_…, Mailgun key-<32hex>, AKIA…, Discord webhook tokens, and scheme://user:password@host basic-auth.
  3. jsCode credential-named literals (#5707) — a quoted string literal bound (= or :) to a credential-named identifier inside a Code node’s jsCode, e.g. const PROM_PASS = '<value>'. This is the class shapes (2) and headers (1) both miss: an opaque static credential (a Prometheus basic-auth password, a bespoke service secret) matches no known token format and is not a header, so before this rule it shipped into the nightly export undetected. The identifier is matched by its word-parts (PROM_PASS → pass, apiKey → api+key, authToken → token), never as a substring — so compass, passenger, tokenizer are not treated as credentials (_CRED_WORDS in the script). See What is deliberately left alone for the value guards that keep it from firing on templated / benign values.

What is deliberately left alone

  • n8n expressions. =Bearer {{ $credentials.token }} is already correct — the {{ … }} segments are preserved verbatim and only the literal text around them is scanned. Clobbering an expression would silently break the live workflow on a restore.
  • credentials blocks. They hold {id, name} references only, never values.
  • Obvious placeholders. YOUR_API_KEY_HERE, CHANGEME, xxxxxxxx, REDACTED_… and similar stay readable — a reviewer should be able to tell an unconfigured node from a scrubbed one.
  • Short literals. Anything under 16 characters, so application/json, Bearer, no-cache and return=representation are never touched.
  • jsCode value guards (#5707). The credential-named-literal rule (item 3 above) fires only when the assigned value actually looks like credential material. It skips: an n8n template '{{ PROMETHEUS_BASIC_AUTH_PASSWORD }}' (the correct, hand-maintained canonical form — left byte-identical); stand-in placeholders (your-password-here, CHANGEME); a URL value (://, handled by the basic-auth rule instead); a whitespace-bearing string (UI copy / a sentence, not a token); and anything under 4 characters. This is what lets the rule scrub the resolved-value export (*_<id>.json) without touching the templated twin (mezmo-alert-router.json).

Properties worth knowing

  • Idempotent. Re-scrubbing an already-scrubbed file finds nothing and writes nothing, so the nightly backup does not churn a diff every night.
  • Minimal diff. The script patches the raw file text rather than re-serialising, so indentation, key order, trailing newline and UTF-8 BOM are all preserved. One committed export was originally written by PowerShell’s ConvertTo-Json (4-space indent, ": " separators); re-serialising it would have produced a ~10k-line diff around a 6-line redaction. If the textual patch cannot produce a provably clean document, the script falls back to a full re-serialisation — a big diff is acceptable, shipping a secret is not.
  • The report never contains a value. It carries file / workflow / node / header names and a rule id only. That is load-bearing: the report is pasted into Actions logs, Discord embeds and GitHub issue bodies. test_report_never_contains_the_secret and test_finding_dataclass_has_no_value_field guard it.

Responding to a finding

The nightly issue tells you which workflow and which node. The backup already protected git; what is left is the live misconfiguration.

  1. Fix the source in n8n (role: sys-admin / infra-task on bms-4 — needs n8n API access). Open the named node, delete the hardcoded header, and attach a credential instead:

    • generic API keys → Header Auth credential (httpHeaderAuth)
    • a supported vendor → the predefined credential type for it (predefinedCredentialType)

    Verify with a read-back through the n8n API that only that node changed.

  2. Decide on rotation (role: secret-manager). The token was in a git-tracked file, so anyone with repo read access could have taken it, and redacting the working tree does not rewrite history. Follow static-api-key-incident-rotation.md. Rotation is a secret-manager call, never a dev-issue one — see secret-manager-request.md.

  3. Re-run the backup (gh workflow run n8n-backup.yml --repo radieu/p24-infra) and confirm the scrub reports clean.

Restore semantics — read this before restoring from JSON

The committed JSON exports are a structure backup: nodes, connections, parameters and settings. They are not, and were never meant to be, a credential backup. After the scrub, a node that relied on a hardcoded header will import with REDACTED_CREDENTIAL in place of the token — by design.

Full-fidelity restore paths, in order of preference:

  1. The Postgres dump — n8n-backup.yml also dumps the n8n database to s3://p24-infra/n8n/postgres-<date>.sql.gz on Wasabi. That is the real disaster-recovery artefact and it is unaffected by the scrub.
  2. The n8n credential store — credentials live there, referenced by id from the workflow JSON.
  3. SOPS — secrets/n8n-bms4.env.sops for the n8n instance’s own environment.

Mojibake signature guard (#4888)

A second, independent rule set runs alongside the credential gate in --check mode only. It catches a different failure mode entirely: encoding corruption, not a secret leak.

What happened

An ad-hoc Windows/PowerShell edit session during the 4694 credential-hardening pass round-tripped workflow JSON through a CP1250 (Windows Central European) decode of already-UTF-8-encoded bytes. That mangles every non-ASCII character in the touched workflows — reproduced exactly by "płatności".encode("utf-8").decode("cp1250"), which yields "pĹ‚atnoĹ›ci". The corruption landed in the live n8n database and was then mirrored into git by the nightly backup, undetected, because nothing was checking for it.

This is the same root-cause family as the documented Windows/PowerShell encoding hazard for SOPS files (docs/playbooks/sops-windows-crlf.md, CLAUDE.md §Secrets) — an ASCII-locale Windows tool touching UTF-8 content without an explicit encoding.

Why it’s a separate rule from SECRET_PATTERNS

  • Different failure class. A credential finding means a token leaked; a mojibake hit means a string is corrupted. Confusing the two on a red CI run sends on-call down the wrong path (rotating a credential fixes nothing here — the live n8n string needs repair).
  • Different remediation. Credential findings are fixed by converting the node to an n8n credential reference. Mojibake hits are fixed by repairing the corrupted string directly in the live n8n instance (sys-admin/infra-task role, not dev-coder) — see format_mojibake_report()’s output and MojibakeHit.describe().
  • Different report shape need. Keeping it a distinct type (MojibakeHit, not a Finding with a special rule id) keeps format_report() — and its no-secret-value contract for credential findings — completely unchanged.

How it works

find_mojibake_signatures(text) -> list[str] scans raw file text for a fixed, exact list of known CP1250-as-UTF-8 double-encoding byte signatures (MOJIBAKE_SIGNATURES in the script) — currently Ĺ‚ (ł), Ĺ› (ś), Ä… (ą), Ĺ„ (ń), Ăł (ó). Each is the literal substring produced by encoding the legitimate Polish character as UTF-8 and mis-decoding those bytes as cp1250.

This is deliberately not a general non-ASCII heuristic. A broader rule (e.g. “any non-ASCII byte sequence that looks unusual”) would false-positive on ordinary Polish/German UTF-8 text — płatności, Kontoauszüge and similar are legitimate and correct, and must never be flagged. Only the exact signatures already confirmed corrupted in the wild are checked. Adding a new signature later: append it to MOJIBAKE_SIGNATURES with the same “encode as UTF-8, decode as cp1250” derivation, plus a synthetic (non-real-value) parametrized test case in test_scrub_n8n_export.py.

scan_mojibake(path) runs this per file during --check only (never --write — this script does not attempt to repair the corruption, only detect it). A hit makes the “Verify export is clean” step fail exactly like a credential finding does, and the --json report gains a mojibake_signatures field (file + signature only, no surrounding text — the same no-value contract as findings, even though these are encoding artifacts rather than secrets).

Responding to a mojibake hit

  1. Do not rotate anything — this is not a credential leak.
  2. Identify the correct spelling by checking against the live source of truth (e.g. the Pinbox24 category list for a costCategory field) — sys-admin/infra-task role.
  3. Repair the string directly in the live n8n instance, not just in git — git is a downstream mirror; the next nightly backup overwrites a git-only edit.
  4. Read the repaired value back from n8n to confirm the write took, then re-run python3 scripts/scrub-n8n-export.py --check <export-dir>/ and confirm it reports clean.
  5. When editing n8n content from a Windows/PowerShell session, always write with explicit UTF-8: [System.IO.File]::WriteAllText($path, $content, [System.Text.UTF8Encoding]::new($false)) — never rely on the console/locale default encoding.

Running it locally

# Report what is hardcoded, without writing (exit 1 if anything found)
python3 scripts/scrub-n8n-export.py --check n8n-workflows/
 
# Redact in place
python3 scripts/scrub-n8n-export.py --write n8n-workflows/
 
# Machine-readable report (no values) — also includes mojibake_signatures
python3 scripts/scrub-n8n-export.py --check --json /tmp/report.json n8n-workflows/
 
# Tests
python -m pytest scripts/tests/test_scrub_n8n_export.py -v

--check runs both gates: the credential scan and the mojibake-signature scan (see above). Either one finding fails the run.

Exit codes: 0 clean (or --write completed), 1 findings in --check mode (credential gate or mojibake gate), 2 usage/IO error.

Adding a new detection rule

  1. Add the header name to SENSITIVE_HEADERS, the shape to SECRET_PATTERNS, or — for a credential bound to a variable inside jsCode — the credential word to _CRED_WORDS, in scripts/scrub-n8n-export.py.
  2. Add a case to the matching @pytest.mark.parametrize block in scripts/tests/test_scrub_n8n_export.py — use a synthetic token, never a real one.
  3. Run python3 scripts/scrub-n8n-export.py --check n8n-workflows/ against the committed corpus and confirm the new rule does not fire on expressions or benign values.

When adding a pattern, prefer a slightly over-eager rule to a slightly under-eager one: redacting an extra base64 blob costs a reviewer nothing, missing a token costs a rotation.

Graph View

Backlinks