Playbook: n8n per-resource Redis singleton lock (INCR + EXPIRE + DEL)

Reusable pattern for any n8n scheduleTrigger workflow whose real-world run time can exceed the trigger interval, and whose runs would corrupt shared state (queues, folders, mailboxes, inboxes, dedup tables) if two runs of the SAME resource execute concurrently.

Origin: issue #4008 — the W4 Direct API Ingestion workflow (bms-4, id=MUjApqruo6H88ebw) has three scheduleTriggers (min :20 / :30 / :40) on separate folders (rechnungen / gutshrifts / standard). On 2026-07-12 the rechnungen backlog took ~70 min, so the next :20 fired while the first was still running, both walked the same folder, and 97.7% of the resulting 427 rows landed on processStatus 300 (“duplicates to confirm”). Backend dedup prevented double-booking, but the ops cost of clearing 300s was large. This pattern prevents the overlap at the source.


When to use this pattern

Apply if all of the following are true. Skip if any is false.

QuestionAnswer to apply
Is the trigger a scheduleTrigger (or webhook that can fire back-to-back on the same resource)?yes
Can one run legitimately take longer than the interval between two consecutive fires?yes
Do two overlapping runs on the SAME resource corrupt state (duplicates, wrong ordering, over-billing)?yes
Is there already a Redis instance the workflow can reach?yes
Would the ops cost of a mid-run crash be smaller than the ops cost of an overlap?yes

If NO to any → use a plain scheduleTrigger with a longer interval, or a queue-mode SET NX EX inside a Code node, or delegate to a message broker. This pattern is specifically for the “same-folder overlap on schedule” shape.

Concrete cases in this stack that qualify:

  • W4 Direct API Ingestion — folder-scoped runs (this issue).
  • Any bulk-mail sender that reads a “to-send” queue table on a fixed schedule.
  • Any cron-driven export that touches a per-tenant working directory.

Design in one paragraph

Each run acquires a lock keyed on the resource it will touch (w4ingest:lock:rechnungen etc.) with INCR. INCR is atomic in Redis — exactly one caller sees 1. If a run sees 1 it owns the lock, sets a TTL (EXPIRE key 600), and proceeds. If a run sees N ≥ 2 it means another run already owns the lock; the loser exits immediately (NoOp) and does NOT touch the counter or the TTL (otherwise it would release the winner’s lock early). The winner heartbeats EXPIRE key 600 on every unit of work (per file / per record) so a long run does not expire mid-flight, and calls DEL key on the “done” branch. If the run crashes, the TTL is the only cleanup — the next scheduled fire finds INCR returning 1 again once the TTL elapses, and takes over cleanly.

Why the key is per-resource, not global: three folders that do NOT share state should still run in parallel. Only the same folder blocks itself.

Why INCR and not SET NX EX: the n8n Redis node exposes INCR / EXPIRE / DEL as first-class operations but does NOT expose a compound SET NX EX — that would require a Code node using ioredis, which the n8n Code sandbox blocks. INCR gives the same atomic “exactly one caller sees 1” guarantee without any Code node.


Node schema (three Redis nodes + one IF, per resource branch)

[scheduleTrigger] ─► [Redis INCR w4ingest:lock:<folder>] ─► [IF: value == 1?]
                                                             │
                     ┌───────────────────────────────────────┴──────────────┐
                     │ TRUE  (this run OWNS the lock)                       │ FALSE (loser)
                     ▼                                                      ▼
        [Redis EXPIRE key 600]                                      [NoOp — end branch]
                     ▼                                              (do NOT touch counter/TTL)
        [existing search / loop nodes]
                     │ (inside loop, per file)
                     ▼
        [Redis EXPIRE key 600]  ◄── heartbeat
                     │
                     ▼
        [existing rest of loop body]
                     │ (loop "done" branch)
                     ▼
        [Redis DEL key] ─► [end]

Node count added per resource branch: 3 Redis operations + 1 IF + 1 NoOp = 5 nodes, plus 1 in-loop heartbeat Redis node.


Node parameters

All Redis nodes use n8n-nodes-base.redis and the bms-4 Redis credential already in the n8n vault (queue-mode Redis; do not create a new one).

1. Acquire — INCR

{
  "operation": "incr",
  "key": "w4ingest:lock:rechnungen"
}

Response shape: { "value": 1 } on first run, { "value": 2 } (or higher) on losers.

2. Gate — IF

{
  "conditions": {
    "number": [
      {
        "value1": "={{ $json.value }}",
        "operation": "equal",
        "value2": 1
      }
    ]
  }
}
  • TRUE branch → continue to EXPIRE → search → loop.
  • FALSE branch → NoOp → end. Do NOT DECR, DEL, or touch TTL in the loser branch. Any of those would release the winner’s lock while the winner is still running.

3. Set TTL — EXPIRE (winner only, once, right after IF-TRUE)

{
  "operation": "expire",
  "key": "w4ingest:lock:rechnungen",
  "ttl": 600
}

TTL of 600 s (10 min) is the “crash-recovery window” — how long a stalled workflow blocks the next attempt. If the workflow runs on a 60-min interval and a normal run is 5–70 min, 600 s is short enough that a crash mid-run does not stall the next scheduled fire beyond one interval, and long enough that a healthy run does not expire between heartbeats.

4. Heartbeat — EXPIRE (inside loop body, per file/record)

Same node parameters as (3). Placed after (or before) the per-file work node so every processed unit resets the TTL to 600 s. In practice the winner’s TTL is refreshed every ~15 s (one file), so a run of 70 min stays locked for its entire duration; only a genuine crash lets the TTL count down.

5. Release — DEL (winner “done” branch only)

{
  "operation": "delete",
  "key": "w4ingest:lock:rechnungen"
}

Placed on the loop’s “done” output — the branch that fires after the last file. Not on the loop iteration output.


Naming convention for the lock key

<workflow-slug>:lock:<resource-id>
  • <workflow-slug> — short, stable identifier for the workflow (not the workflow ID, which changes if the workflow is duplicated). Examples: w4ingest, w3digest, mail-cron.
  • <resource-id> — the smallest thing that must not overlap with itself. Folder name, tenant id, mailbox slug. NOT the trigger ID and NOT the execution ID.

Three folders in W4 ingest → three keys:

  • w4ingest:lock:rechnungen
  • w4ingest:lock:gutshrifts
  • w4ingest:lock:standard

Rechnungen can run for 70 min while gutshrifts runs its 20 min — they do not block each other. Only rechnungen blocks rechnungen.


Rationale — INCR vs SET NX EX

FeatureSET NX EX (would-be ideal)INCR + EXPIRE (this pattern)
Atomic “first caller wins”yesyes (INCR is atomic)
Sets TTL in the same opyesno — second call needed
TTL race window (crash between INCR and EXPIRE)nonesmall: if the workflow crashes AFTER INCR but BEFORE EXPIRE, the key has no TTL and never expires
Requires n8n Code node with ioredisyes — n8n Redis node has no compound SET NX EXno — pure n8n Redis node ops
Works in n8n Code sandboxno (require('ioredis') blocked)yes

The INCR-then-EXPIRE race is the only real downside. Mitigations:

  1. The two nodes execute back-to-back in-process; a crash between them is possible but rare.
  2. A watchdog n8n workflow (out of scope for this pattern) can SCAN for keys matching <workflow-slug>:lock:* with TTL == -1 and set a TTL — belt-and-braces if this becomes a real problem.
  3. Alternatively, wrap the two ops in a Redis MULTI/EXEC transaction using two n8n Redis nodes chained — but the n8n Redis node has no MULTI/EXEC UI, so this again requires a Code node.

For the workflows this pattern targets, the crash-between-INCR-and-EXPIRE window is milliseconds against a 10-minute TTL — the trade is favorable.


Crash-TTL handling — the failure modes

ScenarioWhat happensRecovery
Winner runs normally, DEL fires on done branchLock released immediately.Next trigger sees INCR → 1 and runs.
Winner crashes AFTER EXPIRE set but BEFORE doneLock still held. Heartbeats stop. TTL counts down 600 s.Next trigger within 10 min sees INCR → 2+ and NoOps; a trigger after 10 min sees INCR → 1 and runs. Backlog delayed by ≤10 min.
Winner crashes BETWEEN INCR and EXPIREKey exists with no TTL. Never expires.Detected via watchdog OR by a human noticing “every scheduled run is a NoOp” — redis-cli DEL <key> unblocks.
Loser accidentally DELsWinner’s lock released mid-run; the next scheduled fire enters as a second winner → OVERLAP (the exact bug this pattern prevents).Prevention: loser branch has ONLY NoOp. Any Redis op in the loser branch is a design bug — flag in review.
Loser accidentally EXPIREs / DECRsSame as above — releases winner’s lock or extends it wrongly.Same prevention: NO Redis ops in loser branch.
Redis restarts mid-runKey gone. Next INCR returns 1. If the winner is still running, it will still EXPIRE and DEL, but the next scheduled fire is a new winner too → OVERLAP.Redis restarts are rare; if this is a concern, use Redis persistence (appendonly yes) or accept the once-per-redis-restart overlap.

The dominant failure mode is #2 (winner crashes) — that’s the case the TTL is designed for. Everything else is a design-review concern: keep the loser branch pure NoOp, add a heartbeat in the loop, put DEL only on the done branch.


Applying this to the W4 Direct API Ingestion workflow (worked example)

Workflow: MUjApqruo6H88ebw on bms-4.

Current structure per branch:

[scheduleTrigger :20] ─► [search rechnungen folder] ─► [loop over files] ─► [end]

Modified structure per branch:

[scheduleTrigger :20]
    ▼
[Redis INCR w4ingest:lock:rechnungen]
    ▼
[IF value == 1]
    ├── TRUE ─► [Redis EXPIRE …:rechnungen 600]
    │            ▼
    │        [search rechnungen folder]
    │            ▼
    │        [loop over files]
    │            │
    │            ├── iteration ─► [Redis EXPIRE …:rechnungen 600]
    │            │                    ▼
    │            │                [existing per-file work]
    │            │                    ▼ (back to loop)
    │            │
    │            └── done ─► [Redis DEL …:rechnungen] ─► [end]
    │
    └── FALSE ─► [NoOp] ─► [end]

Repeat identically for gutshrifts (:30) and standard (:40) — same shape, only the key suffix and folder name differ.

Rollout steps (for the follow-up implementation issue, NOT this PR)

  1. Confirm the workflow is active:false before editing (it currently is — safe edit window).
  2. Snapshot current JSON: GET /rest/workflows/MUjApqruo6H88ebw → commit to n8n-workflows/w4-direct-api-ingestion.pre-4008.json so a rollback is one file away.
  3. Add the 5 lock nodes + 1 heartbeat per branch (× 3 branches) via the n8n REST API.
  4. Export the modified JSON to n8n-workflows/w4-direct-api-ingestion.json — this is the first time this workflow lives in git.
  5. Test with a manual webhook trigger on the rechnungen branch:
    • First POST → confirm winner: INCR → 1, EXPIRE set, search runs.
    • Second POST WHILE first is still running → confirm loser: INCR → 2, NoOp, execution finishes in <2 s.
    • redis-cli TTL w4ingest:lock:rechnungen should show a decreasing value that jumps back to ~600 s at every heartbeat.
  6. Kill the winner mid-run (docker restart n8n-worker) → confirm TTL counts down to zero within 10 min and the next scheduled fire wins.
  7. Re-enable the workflow only after all three tests pass. Choose a window between :00–:19 or :41–:59 to avoid stepping on the real triggers.
  8. If any test is red → leave the workflow active:false and file a bug back to this playbook (the pattern is at fault, not the workflow).

Testing & verification (single-document webhooks)

Origin: #4135 — the repair round where lock_hb_* heartbeat keys were dropping items and gutshrifts/standard stopped processing. Same visible symptom as the #6281 credential outage below. That repair added a single-document test webhook per category so a change can be smoke-tested end-to-end without waiting for the real :20 / :30 / :40 cron.

⚠ FIRST-CHECK — 403 “Forbidden — perhaps check your credentials?” on a Drive download node = a SUBFOLDER in the intake path

Origin: #6xxx — 2026-08-13 rechnungen outage. A user created a subfolder (gesendet) inside the rechnungen intake folder. The search_* node ran in Google Drive “Files and Folders” mode with no folder-type filter, so it returned the folder as an item, the download_* node tried to “download” it, and Google returned 403 fileNotExportable (“Export only supports Docs Editors files”) — which n8n surfaced as the misleading “Forbidden — perhaps check your credentials?”.

This is NOT an OAuth / scope / credential / ownership problem. Do not chase re-auth, re-scope, or credential rotation. The download credential is fine. The bug is the search node listing folders.

Diagnosis (fastest): list the intake folder — parentId = '<intakeFolderId>' — and look for any item with mimeType = 'application/vnd.google-apps.folder'. If one exists, that is the item the download node is choking on.

Permanent fix: set every Drive fileFolder search node to Files-only so folders are never returned. In the node’s filter set whatToSearch: "files" (UI: Filter → What to Search → Files). This appends mimeType != 'application/vnd.google-apps.folder' to the Drive query and changes nothing else (same folder id, same name filter, same limit/returnAll). If a node uses the query-string method instead, AND-combine mimeType != 'application/vnd.google-apps.folder' into the existing query.

Scope — must be applied to ALL 6 search nodes: the 3 production search nodes (search_rechnungen, search_gutshrifts, search_standard) and their 3 test variants (test_search_rechnungen, test_search_gutshrifts, test_search_standard). A test_ node left in Files+Folders mode reintroduces the bug on the next test run.

Verification — folder-exclusion test (two-part, on the rechnungen branch):

  1. Single-file test: ensure one normal PDF is directly in the rechnungen intake folder, fire test-w4-ingest-rechnungen, and read the execution: download_rechnungen must emit real binary bytes (application/pdf, non-zero size) with no 403, lock_hb_rechnungen shows 1 item, a regRecords row is written, status success.
  2. Subfolder test: create a temporary subfolder (e.g. __test_folder_exclusion__) inside the rechnungen intake folder, fire the webhook again, and read the execution: the test_search_rechnungen node must return 0 items, download_rechnungen must not run and must not 403, status success. Then DELETE the temporary subfolder — never leave a test artifact in a production intake folder. (A responseMode:lastNode webhook may return HTTP 500 “No item to return was found” here — that is expected/benign when the search legitimately finds nothing; the execution status is still success.)

Prerequisite — the Redis credential must actually authenticate (WRONGPASS check)

Do this FIRST, before any ingestion test. The lock nodes silently fail closed if the n8n Redis credential is stale — the exact #6281 outage: after a bms-4 REDIS_PASSWORD rotation the compose env + requirepass were updated but the stored n8n credential (xjD48jTskJlCyRY0, “bms-4 queue Redis”) was not, so every run threw WRONGPASS invalid username-password pair, the lock never acquired, and all three folders went silent with no obvious error to the client.

The REDIS_PASSWORD has 5 consumers — a rotation must update all five:

  1. secrets/n8n-bms4.env.sops (SOPS source of truth)
  2. bms-4/.env (delivered by secrets-sync.yml)
  3. bms-4/docker-compose.yml — requirepass + QUEUE_BULL_REDIS_PASSWORD
  4. Redis requirepass reload
  5. n8n stored credential xjD48jTskJlCyRY0 ← the one rotations miss

Canonical fix for consumer #5 (do NOT hand-roll a REST PATCH):

# on bms-4 — reads current REDIS_PASSWORD from the deployed .env, updates the n8n vault
python3 bms-4/sync-n8n-credentials.py --cred REDIS_PASSWORD

See docs/n8n-credential-portability.md. Verify the credential is live before testing:

# healthy → PONG (password value never printed)
REDIS_PW=$(grep "^REDIS_PASSWORD=" /opt/p24-infra/bms-4/.env | cut -d= -f2-)
docker exec bms4-redis redis-cli -a "$REDIS_PW" PING; unset REDIS_PW

The three test webhooks (one document each, limit:1)

Each webhook drives the same category branch as its cron trigger but caps the search at a single document (test_search_<cat> node, limit:1) so it processes exactly one file:

WebhookCategory branchLock key
test-w4-ingest-rechnungenRechnungen (:20)w4ingest:lock:rechnungen
test-w4-ingest-gutshriftsGutshrifts (:30)w4ingest:lock:gutshrifts
test-w4-ingest-standardStandard (:40)w4ingest:lock:standard

Trigger one (POST, no body needed):

curl -s -o /dev/null -w "%{http_code}\n" \
  -X POST https://n8n.<bms4-host>/webhook/test-w4-ingest-gutshrifts

Pass criteria — a green single-document run must show ALL of:

  • Webhook returns HTTP 200 with a fileId + w4RecordId in the response.
  • The source file is archived/moved out of the inbox folder (not left for a re-pick).
  • redis-cli EXISTS lock_hb_<cat> (or the category lock key) shows 1 item during the run — 0 means the lock never acquired (WRONGPASS or a broken IF gate; go back to the prerequisite check above).
  • A w4_db.regRecords row was created with recordMainDocument populated.
  • No error_flow red alert on the Discord error webhook within 5 minutes of the run.
  • The search_* / test_search_* node is Files-only (filter.whatToSearch: "files") so a stray subfolder in the intake path is never returned as a downloadable item. See the folder-exclusion FIRST-CHECK callout at the top of this section.

Run all three (rechnungen / gutshrifts / standard) — a credential or lock regression usually takes out all three at once, but the per-category webhooks isolate which branch is broken.

When to run this

  • After any REDIS_PASSWORD rotation (the #6281 class — always re-test all three).
  • After any edit to the lock nodes or category branches.
  • When a client reports “folders not processing” — this is the fastest confirmation that the fix landed, instead of waiting up to an hour for the next :20 / :30 / :40 cron tick.
  • After a download node throws 403 “Forbidden — perhaps check your credentials?” — run the two-part folder-exclusion test (single-file + subfolder) from the FIRST-CHECK callout above, not a credential rotation.

Checklist — reviewer / self-review

Before merging a workflow that uses this pattern:

  • Every scheduleTrigger branch has an INCR → IF gate immediately after it.
  • Lock key is <workflow-slug>:lock:<resource-id> — stable across workflow renames.
  • Winner branch: EXPIRE key 600 executes exactly once, right after the IF-TRUE output.
  • Winner branch: EXPIRE key 600 heartbeat inside the loop, per unit of work.
  • Winner branch: DEL key on the loop’s DONE output only (not the iteration output).
  • Loser branch: contains a NoOp and nothing else — no Redis operations at all.
  • Three separate keys if three resources — never a shared key across parallel resources.
  • TTL matches the expected max healthy run duration × 1.2 (buffer). 600 s for W4 ingest; adjust up for slower workflows, never down below ~2× heartbeat interval.
  • The Redis credential is the existing bms-4 queue-mode Redis credential — do not create a new one and do not put the connection string in a node parameter.

A separate n8n workflow can SCAN 0 MATCH <workflow-slug>:lock:* every 5 min and alert on:

  • Any key with TTL == -1 (INCR-EXPIRE race) → likely stale.
  • Any key with TTL < 30 AND the owning workflow has an execution in running state older than the TTL was set for → likely a heartbeat gap.

Route to the existing Discord webhook P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL per the p24-infra error-notification standard.

This is not required for the pattern to be correct — it is defensive monitoring for the two crash modes the TTL alone does not fix (Redis restart while winner running; crash between INCR and EXPIRE).


Intake-subfolder 403 — “check your credentials?” is NOT a credential problem

Origin: #6281 / #6291 — root cause finally isolated 2026-08-13. The W4 Direct API Ingestion workflow (MUjApqruo6H88ebw) download_* Google Drive nodes threw 403 "Forbidden — perhaps check your credentials?" and ingestion stalled. This looked identical to the #6281 Redis WRONGPASS outage above and to an OAuth expiry, so time was lost re-authing credentials that were already valid.

DIAGNOSTIC ORDER — read this first

When a download_* node returns 403 "Forbidden — perhaps check your credentials?", CHECK FOR A SUBFOLDER IN THE INTAKE PATH FIRST — before touching OAuth, re-auth, or scopes. The “check your credentials?” text is n8n’s generic wrapper and is misleading here. Confirm by listing the intake folder for children with mimeType = application/vnd.google-apps.folder; if one exists, that is the cause.

Root cause

The search_* nodes ran in “Files and Folders” mode. When someone creates a subfolder inside an intake folder to organise things (e.g. a gesendet folder for sent invoices), the search returns that subfolder as an item. The downstream download_* node then tries to export a folder. Google Drive rejects exporting a folder with the real error "Export only supports Docs Editors files" (reason: fileNotExportable) — and n8n masks that behind the generic 403 "Forbidden — perhaps check your credentials?" string. So a purely organisational subfolder created by an operator silently takes the whole branch down, disguised as a credential failure.

The fix (applied 2026-08-13)

Set every search_* node AND its test_search_* variant to whatToSearch: "files" (Files only, not Files and Folders). n8n then queries mimeType != 'application/vnd.google-apps.folder', so subfolders are never returned as items and can never reach a download node. Applied to all six nodes:

Cron search nodeTest-webhook variant
search_rechnungentest_search_rechnungen
search_gutshriftstest_search_gutshrifts
search_standardtest_search_standard

This makes subfolders permanently invisible to the search, so this class of outage cannot recur regardless of what operators create in the intake folder.

SILENT-SKIP CAVEAT — operators must NOT use subfolders

The search is non-recursive. Files placed inside a subfolder are never found and never processed — no error, silently skipped. Operators must drop invoices directly in the intake folder, never in a subfolder. A file in a subfolder is invisible to ingestion.

Test procedure (both must pass before trusting the fix)

Both tests use the single-document test webhooks documented above. The Redis singleton lock short-circuits test runs to lock_noop_* if the lock is already held, so free the lock first each time (DEL w4ingest:lock:rechnungen on bms-4-redis-1).

  1. Single-file test — free the lock, POST the test-w4-ingest-rechnungen webhook. The execution must reach download_rechnungen, download 1 item, no error, and end at lock_del_rechnungen. (Verified: exec 322620.)
  2. Subfolder test — create a temp subfolder inside the intake folder, free the lock, POST the same webhook. test_search_rechnungen must return 0 items and download_rechnungen must never be reached (the folder is excluded, no 403). (Verified: exec 322653.) Under the old Files-and-Folders behaviour this returned the folder as 1 item and produced the 403.

See also

  • docs/playbooks/n8n-bms4-redis-password-rotation.md — bms-4 REDIS_PASSWORD rotation across all 5 consumers (incl. this workflow’s stored credential) + the #4135 single-doc verification test.
  • docs/pinbox24/pinbox24-workflow-engine.md — inventory of W3/W4 ingest workflows.
  • docs/playbooks/n8n/n8n-workflow-creation.md — n8n workflow authoring conventions.
  • docs/playbooks/n8n/n8n-bms4-stuck-execution-cleanup.md — how to unstick a workflow whose lock is legitimately stale.