Playbook: n8n per-resource Redis singleton lock (INCR + EXPIRE + DEL)
Reusable pattern for any n8n scheduleTrigger workflow whose real-world run time can exceed
the trigger interval, and whose runs would corrupt shared state (queues, folders, mailboxes,
inboxes, dedup tables) if two runs of the SAME resource execute concurrently.
Origin: issue #4008 — the W4 Direct API
Ingestion workflow (bms-4, id=MUjApqruo6H88ebw) has three scheduleTriggers (min :20 / :30 /
:40) on separate folders (rechnungen / gutshrifts / standard). On 2026-07-12 the rechnungen
backlog took ~70 min, so the next :20 fired while the first was still running, both walked the
same folder, and 97.7% of the resulting 427 rows landed on processStatus 300
(“duplicates to confirm”). Backend dedup prevented double-booking, but the ops cost of clearing
300s was large. This pattern prevents the overlap at the source.
When to use this pattern
Apply if all of the following are true. Skip if any is false.
| Question | Answer to apply |
|---|---|
Is the trigger a scheduleTrigger (or webhook that can fire back-to-back on the same resource)? | yes |
| Can one run legitimately take longer than the interval between two consecutive fires? | yes |
| Do two overlapping runs on the SAME resource corrupt state (duplicates, wrong ordering, over-billing)? | yes |
| Is there already a Redis instance the workflow can reach? | yes |
| Would the ops cost of a mid-run crash be smaller than the ops cost of an overlap? | yes |
If NO to any → use a plain scheduleTrigger with a longer interval, or a queue-mode SET NX EX
inside a Code node, or delegate to a message broker. This pattern is specifically for the
“same-folder overlap on schedule” shape.
Concrete cases in this stack that qualify:
- W4 Direct API Ingestion — folder-scoped runs (this issue).
- Any bulk-mail sender that reads a “to-send” queue table on a fixed schedule.
- Any cron-driven export that touches a per-tenant working directory.
Design in one paragraph
Each run acquires a lock keyed on the resource it will touch (w4ingest:lock:rechnungen
etc.) with INCR. INCR is atomic in Redis — exactly one caller sees 1. If a run sees 1
it owns the lock, sets a TTL (EXPIRE key 600), and proceeds. If a run sees N ≥ 2 it means
another run already owns the lock; the loser exits immediately (NoOp) and does NOT touch
the counter or the TTL (otherwise it would release the winner’s lock early). The winner
heartbeats EXPIRE key 600 on every unit of work (per file / per record) so a long run
does not expire mid-flight, and calls DEL key on the “done” branch. If the run crashes,
the TTL is the only cleanup — the next scheduled fire finds INCR returning 1 again once
the TTL elapses, and takes over cleanly.
Why the key is per-resource, not global: three folders that do NOT share state should still run in parallel. Only the same folder blocks itself.
Why INCR and not SET NX EX: the n8n Redis node exposes INCR / EXPIRE / DEL as first-class
operations but does NOT expose a compound SET NX EX — that would require a Code node using
ioredis, which the n8n Code sandbox blocks. INCR gives the same atomic
“exactly one caller sees 1” guarantee without any Code node.
Node schema (three Redis nodes + one IF, per resource branch)
[scheduleTrigger] ─► [Redis INCR w4ingest:lock:<folder>] ─► [IF: value == 1?]
│
┌───────────────────────────────────────┴──────────────┐
│ TRUE (this run OWNS the lock) │ FALSE (loser)
▼ ▼
[Redis EXPIRE key 600] [NoOp — end branch]
▼ (do NOT touch counter/TTL)
[existing search / loop nodes]
│ (inside loop, per file)
▼
[Redis EXPIRE key 600] ◄── heartbeat
│
▼
[existing rest of loop body]
│ (loop "done" branch)
▼
[Redis DEL key] ─► [end]
Node count added per resource branch: 3 Redis operations + 1 IF + 1 NoOp = 5 nodes, plus 1 in-loop heartbeat Redis node.
Node parameters
All Redis nodes use n8n-nodes-base.redis and the bms-4 Redis credential already in the n8n
vault (queue-mode Redis; do not create a new one).
1. Acquire — INCR
{
"operation": "incr",
"key": "w4ingest:lock:rechnungen"
}Response shape: { "value": 1 } on first run, { "value": 2 } (or higher) on losers.
2. Gate — IF
{
"conditions": {
"number": [
{
"value1": "={{ $json.value }}",
"operation": "equal",
"value2": 1
}
]
}
}- TRUE branch → continue to
EXPIRE→ search → loop. - FALSE branch →
NoOp→ end. Do NOTDECR,DEL, or touch TTL in the loser branch. Any of those would release the winner’s lock while the winner is still running.
3. Set TTL — EXPIRE (winner only, once, right after IF-TRUE)
{
"operation": "expire",
"key": "w4ingest:lock:rechnungen",
"ttl": 600
}TTL of 600 s (10 min) is the “crash-recovery window” — how long a stalled workflow blocks the next attempt. If the workflow runs on a 60-min interval and a normal run is 5–70 min, 600 s is short enough that a crash mid-run does not stall the next scheduled fire beyond one interval, and long enough that a healthy run does not expire between heartbeats.
4. Heartbeat — EXPIRE (inside loop body, per file/record)
Same node parameters as (3). Placed after (or before) the per-file work node so every processed unit resets the TTL to 600 s. In practice the winner’s TTL is refreshed every ~15 s (one file), so a run of 70 min stays locked for its entire duration; only a genuine crash lets the TTL count down.
5. Release — DEL (winner “done” branch only)
{
"operation": "delete",
"key": "w4ingest:lock:rechnungen"
}Placed on the loop’s “done” output — the branch that fires after the last file. Not on the loop iteration output.
Naming convention for the lock key
<workflow-slug>:lock:<resource-id>
<workflow-slug>— short, stable identifier for the workflow (not the workflow ID, which changes if the workflow is duplicated). Examples:w4ingest,w3digest,mail-cron.<resource-id>— the smallest thing that must not overlap with itself. Folder name, tenant id, mailbox slug. NOT the trigger ID and NOT the execution ID.
Three folders in W4 ingest → three keys:
w4ingest:lock:rechnungenw4ingest:lock:gutshriftsw4ingest:lock:standard
Rechnungen can run for 70 min while gutshrifts runs its 20 min — they do not block each other. Only rechnungen blocks rechnungen.
Rationale — INCR vs SET NX EX
| Feature | SET NX EX (would-be ideal) | INCR + EXPIRE (this pattern) |
|---|---|---|
| Atomic “first caller wins” | yes | yes (INCR is atomic) |
| Sets TTL in the same op | yes | no — second call needed |
| TTL race window (crash between INCR and EXPIRE) | none | small: if the workflow crashes AFTER INCR but BEFORE EXPIRE, the key has no TTL and never expires |
Requires n8n Code node with ioredis | yes — n8n Redis node has no compound SET NX EX | no — pure n8n Redis node ops |
| Works in n8n Code sandbox | no (require('ioredis') blocked) | yes |
The INCR-then-EXPIRE race is the only real downside. Mitigations:
- The two nodes execute back-to-back in-process; a crash between them is possible but rare.
- A watchdog n8n workflow (out of scope for this pattern) can
SCANfor keys matching<workflow-slug>:lock:*withTTL == -1and set a TTL — belt-and-braces if this becomes a real problem. - Alternatively, wrap the two ops in a Redis MULTI/EXEC transaction using two n8n Redis nodes chained — but the n8n Redis node has no MULTI/EXEC UI, so this again requires a Code node.
For the workflows this pattern targets, the crash-between-INCR-and-EXPIRE window is milliseconds against a 10-minute TTL — the trade is favorable.
Crash-TTL handling — the failure modes
| Scenario | What happens | Recovery |
|---|---|---|
| Winner runs normally, DEL fires on done branch | Lock released immediately. | Next trigger sees INCR → 1 and runs. |
Winner crashes AFTER EXPIRE set but BEFORE done | Lock still held. Heartbeats stop. TTL counts down 600 s. | Next trigger within 10 min sees INCR → 2+ and NoOps; a trigger after 10 min sees INCR → 1 and runs. Backlog delayed by ≤10 min. |
Winner crashes BETWEEN INCR and EXPIRE | Key exists with no TTL. Never expires. | Detected via watchdog OR by a human noticing “every scheduled run is a NoOp” — redis-cli DEL <key> unblocks. |
Loser accidentally DELs | Winner’s lock released mid-run; the next scheduled fire enters as a second winner → OVERLAP (the exact bug this pattern prevents). | Prevention: loser branch has ONLY NoOp. Any Redis op in the loser branch is a design bug — flag in review. |
Loser accidentally EXPIREs / DECRs | Same as above — releases winner’s lock or extends it wrongly. | Same prevention: NO Redis ops in loser branch. |
| Redis restarts mid-run | Key gone. Next INCR returns 1. If the winner is still running, it will still EXPIRE and DEL, but the next scheduled fire is a new winner too → OVERLAP. | Redis restarts are rare; if this is a concern, use Redis persistence (appendonly yes) or accept the once-per-redis-restart overlap. |
The dominant failure mode is #2 (winner crashes) — that’s the case the TTL is designed for. Everything else is a design-review concern: keep the loser branch pure NoOp, add a heartbeat in the loop, put DEL only on the done branch.
Applying this to the W4 Direct API Ingestion workflow (worked example)
Workflow: MUjApqruo6H88ebw on bms-4.
Current structure per branch:
[scheduleTrigger :20] ─► [search rechnungen folder] ─► [loop over files] ─► [end]
Modified structure per branch:
[scheduleTrigger :20]
▼
[Redis INCR w4ingest:lock:rechnungen]
▼
[IF value == 1]
├── TRUE ─► [Redis EXPIRE …:rechnungen 600]
│ ▼
│ [search rechnungen folder]
│ ▼
│ [loop over files]
│ │
│ ├── iteration ─► [Redis EXPIRE …:rechnungen 600]
│ │ ▼
│ │ [existing per-file work]
│ │ ▼ (back to loop)
│ │
│ └── done ─► [Redis DEL …:rechnungen] ─► [end]
│
└── FALSE ─► [NoOp] ─► [end]
Repeat identically for gutshrifts (:30) and standard (:40) — same shape, only the key suffix and folder name differ.
Rollout steps (for the follow-up implementation issue, NOT this PR)
- Confirm the workflow is
active:falsebefore editing (it currently is — safe edit window). - Snapshot current JSON:
GET /rest/workflows/MUjApqruo6H88ebw→ commit ton8n-workflows/w4-direct-api-ingestion.pre-4008.jsonso a rollback is one file away. - Add the 5 lock nodes + 1 heartbeat per branch (× 3 branches) via the n8n REST API.
- Export the modified JSON to
n8n-workflows/w4-direct-api-ingestion.json— this is the first time this workflow lives in git. - Test with a manual webhook trigger on the rechnungen branch:
- First POST → confirm winner:
INCR → 1, EXPIRE set, search runs. - Second POST WHILE first is still running → confirm loser:
INCR → 2, NoOp, execution finishes in <2 s. redis-cli TTL w4ingest:lock:rechnungenshould show a decreasing value that jumps back to ~600 s at every heartbeat.
- First POST → confirm winner:
- Kill the winner mid-run (
docker restart n8n-worker) → confirmTTLcounts down to zero within 10 min and the next scheduled fire wins. - Re-enable the workflow only after all three tests pass. Choose a window between :00–:19 or :41–:59 to avoid stepping on the real triggers.
- If any test is red → leave the workflow
active:falseand file a bug back to this playbook (the pattern is at fault, not the workflow).
Testing & verification (single-document webhooks)
Origin: #4135 — the repair round where
lock_hb_* heartbeat keys were dropping items and gutshrifts/standard stopped processing.
Same visible symptom as the #6281 credential
outage below. That repair added a single-document test webhook per category so a change can
be smoke-tested end-to-end without waiting for the real :20 / :30 / :40 cron.
⚠ FIRST-CHECK — 403 “Forbidden — perhaps check your credentials?” on a Drive download node = a SUBFOLDER in the intake path
Origin: #6xxx — 2026-08-13 rechnungen outage. A user created a subfolder (
gesendet) inside the rechnungen intake folder. Thesearch_*node ran in Google Drive “Files and Folders” mode with no folder-type filter, so it returned the folder as an item, thedownload_*node tried to “download” it, and Google returned403 fileNotExportable(“Export only supports Docs Editors files”) — which n8n surfaced as the misleading “Forbidden — perhaps check your credentials?”.This is NOT an OAuth / scope / credential / ownership problem. Do not chase re-auth, re-scope, or credential rotation. The download credential is fine. The bug is the search node listing folders.
Diagnosis (fastest): list the intake folder —
parentId = '<intakeFolderId>'— and look for any item withmimeType = 'application/vnd.google-apps.folder'. If one exists, that is the item the download node is choking on.Permanent fix: set every Drive
fileFoldersearch node to Files-only so folders are never returned. In the node’sfiltersetwhatToSearch: "files"(UI: Filter → What to Search → Files). This appendsmimeType != 'application/vnd.google-apps.folder'to the Drive query and changes nothing else (same folder id, same name filter, same limit/returnAll). If a node uses the query-string method instead, AND-combinemimeType != 'application/vnd.google-apps.folder'into the existing query.Scope — must be applied to ALL 6 search nodes: the 3 production search nodes (
search_rechnungen,search_gutshrifts,search_standard) and their 3 test variants (test_search_rechnungen,test_search_gutshrifts,test_search_standard). Atest_node left in Files+Folders mode reintroduces the bug on the next test run.Verification — folder-exclusion test (two-part, on the rechnungen branch):
- Single-file test: ensure one normal PDF is directly in the rechnungen intake folder, fire
test-w4-ingest-rechnungen, and read the execution:download_rechnungenmust emit real binary bytes (application/pdf, non-zero size) with no 403,lock_hb_rechnungenshows 1 item, aregRecordsrow is written, statussuccess.- Subfolder test: create a temporary subfolder (e.g.
__test_folder_exclusion__) inside the rechnungen intake folder, fire the webhook again, and read the execution: thetest_search_rechnungennode must return 0 items,download_rechnungenmust not run and must not 403, statussuccess. Then DELETE the temporary subfolder — never leave a test artifact in a production intake folder. (AresponseMode:lastNodewebhook may return HTTP 500 “No item to return was found” here — that is expected/benign when the search legitimately finds nothing; the execution status is stillsuccess.)
Prerequisite — the Redis credential must actually authenticate (WRONGPASS check)
Do this FIRST, before any ingestion test. The lock nodes silently fail closed if the n8n
Redis credential is stale — the exact #6281
outage: after a bms-4 REDIS_PASSWORD rotation the compose env + requirepass were updated but
the stored n8n credential (xjD48jTskJlCyRY0, “bms-4 queue Redis”) was not, so every run
threw WRONGPASS invalid username-password pair, the lock never acquired, and all three folders
went silent with no obvious error to the client.
The REDIS_PASSWORD has 5 consumers — a rotation must update all five:
secrets/n8n-bms4.env.sops(SOPS source of truth)bms-4/.env(delivered bysecrets-sync.yml)bms-4/docker-compose.yml—requirepass+QUEUE_BULL_REDIS_PASSWORD- Redis
requirepassreload - n8n stored credential
xjD48jTskJlCyRY0← the one rotations miss
Canonical fix for consumer #5 (do NOT hand-roll a REST PATCH):
# on bms-4 — reads current REDIS_PASSWORD from the deployed .env, updates the n8n vault
python3 bms-4/sync-n8n-credentials.py --cred REDIS_PASSWORDSee docs/n8n-credential-portability.md. Verify the credential is live before testing:
# healthy → PONG (password value never printed)
REDIS_PW=$(grep "^REDIS_PASSWORD=" /opt/p24-infra/bms-4/.env | cut -d= -f2-)
docker exec bms4-redis redis-cli -a "$REDIS_PW" PING; unset REDIS_PWThe three test webhooks (one document each, limit:1)
Each webhook drives the same category branch as its cron trigger but caps the search at a
single document (test_search_<cat> node, limit:1) so it processes exactly one file:
| Webhook | Category branch | Lock key |
|---|---|---|
test-w4-ingest-rechnungen | Rechnungen (:20) | w4ingest:lock:rechnungen |
test-w4-ingest-gutshrifts | Gutshrifts (:30) | w4ingest:lock:gutshrifts |
test-w4-ingest-standard | Standard (:40) | w4ingest:lock:standard |
Trigger one (POST, no body needed):
curl -s -o /dev/null -w "%{http_code}\n" \
-X POST https://n8n.<bms4-host>/webhook/test-w4-ingest-gutshriftsPass criteria — a green single-document run must show ALL of:
- Webhook returns HTTP 200 with a
fileId+w4RecordIdin the response. - The source file is archived/moved out of the inbox folder (not left for a re-pick).
-
redis-cli EXISTS lock_hb_<cat>(or the category lock key) shows 1 item during the run — 0 means the lock never acquired (WRONGPASS or a broken IF gate; go back to the prerequisite check above). - A
w4_db.regRecordsrow was created withrecordMainDocumentpopulated. - No
error_flowred alert on the Discord error webhook within 5 minutes of the run. - The
search_*/test_search_*node is Files-only (filter.whatToSearch: "files") so a stray subfolder in the intake path is never returned as a downloadable item. See the folder-exclusion FIRST-CHECK callout at the top of this section.
Run all three (rechnungen / gutshrifts / standard) — a credential or lock regression usually takes out all three at once, but the per-category webhooks isolate which branch is broken.
When to run this
- After any
REDIS_PASSWORDrotation (the #6281 class — always re-test all three). - After any edit to the lock nodes or category branches.
- When a client reports “folders not processing” — this is the fastest confirmation that the
fix landed, instead of waiting up to an hour for the next
:20 / :30 / :40cron tick. - After a download node throws 403 “Forbidden — perhaps check your credentials?” — run the two-part folder-exclusion test (single-file + subfolder) from the FIRST-CHECK callout above, not a credential rotation.
Checklist — reviewer / self-review
Before merging a workflow that uses this pattern:
- Every
scheduleTriggerbranch has anINCR→IFgate immediately after it. - Lock key is
<workflow-slug>:lock:<resource-id>— stable across workflow renames. - Winner branch:
EXPIRE key 600executes exactly once, right after the IF-TRUE output. - Winner branch:
EXPIRE key 600heartbeat inside the loop, per unit of work. - Winner branch:
DEL keyon the loop’s DONE output only (not the iteration output). - Loser branch: contains a
NoOpand nothing else — no Redis operations at all. - Three separate keys if three resources — never a shared key across parallel resources.
- TTL matches the expected max healthy run duration × 1.2 (buffer). 600 s for W4 ingest; adjust up for slower workflows, never down below ~2× heartbeat interval.
- The Redis credential is the existing bms-4 queue-mode Redis credential — do not create a new one and do not put the connection string in a node parameter.
Alerting on stale locks (optional but recommended)
A separate n8n workflow can SCAN 0 MATCH <workflow-slug>:lock:* every 5 min and alert on:
- Any key with
TTL == -1(INCR-EXPIRE race) → likely stale. - Any key with
TTL < 30AND the owning workflow has an execution inrunningstate older than the TTL was set for → likely a heartbeat gap.
Route to the existing Discord webhook P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL per the
p24-infra error-notification standard.
This is not required for the pattern to be correct — it is defensive monitoring for the two crash modes the TTL alone does not fix (Redis restart while winner running; crash between INCR and EXPIRE).
Intake-subfolder 403 — “check your credentials?” is NOT a credential problem
Origin: #6281 /
#6291 — root cause finally isolated
2026-08-13. The W4 Direct API Ingestion workflow (MUjApqruo6H88ebw) download_* Google Drive
nodes threw 403 "Forbidden — perhaps check your credentials?" and ingestion stalled. This
looked identical to the #6281 Redis WRONGPASS outage above and to an OAuth expiry, so time was
lost re-authing credentials that were already valid.
DIAGNOSTIC ORDER — read this first
When a download_* node returns 403 "Forbidden — perhaps check your credentials?", CHECK
FOR A SUBFOLDER IN THE INTAKE PATH FIRST — before touching OAuth, re-auth, or scopes. The
“check your credentials?” text is n8n’s generic wrapper and is misleading here. Confirm by
listing the intake folder for children with mimeType = application/vnd.google-apps.folder; if
one exists, that is the cause.
Root cause
The search_* nodes ran in “Files and Folders” mode. When someone creates a subfolder
inside an intake folder to organise things (e.g. a gesendet folder for sent invoices), the
search returns that subfolder as an item. The downstream download_* node then tries to
export a folder. Google Drive rejects exporting a folder with the real error
"Export only supports Docs Editors files" (reason: fileNotExportable) — and n8n masks that
behind the generic 403 "Forbidden — perhaps check your credentials?" string. So a purely
organisational subfolder created by an operator silently takes the whole branch down, disguised
as a credential failure.
The fix (applied 2026-08-13)
Set every search_* node AND its test_search_* variant to whatToSearch: "files"
(Files only, not Files and Folders). n8n then queries
mimeType != 'application/vnd.google-apps.folder', so subfolders are never returned as items
and can never reach a download node. Applied to all six nodes:
| Cron search node | Test-webhook variant |
|---|---|
search_rechnungen | test_search_rechnungen |
search_gutshrifts | test_search_gutshrifts |
search_standard | test_search_standard |
This makes subfolders permanently invisible to the search, so this class of outage cannot recur regardless of what operators create in the intake folder.
SILENT-SKIP CAVEAT — operators must NOT use subfolders
The search is non-recursive. Files placed inside a subfolder are never found and never processed — no error, silently skipped. Operators must drop invoices directly in the intake folder, never in a subfolder. A file in a subfolder is invisible to ingestion.
Test procedure (both must pass before trusting the fix)
Both tests use the single-document test webhooks documented above. The Redis singleton lock
short-circuits test runs to lock_noop_* if the lock is already held, so free the lock first
each time (DEL w4ingest:lock:rechnungen on bms-4-redis-1).
- Single-file test — free the lock, POST the
test-w4-ingest-rechnungenwebhook. The execution must reachdownload_rechnungen, download 1 item, no error, and end atlock_del_rechnungen. (Verified: exec 322620.) - Subfolder test — create a temp subfolder inside the intake folder, free the lock, POST
the same webhook.
test_search_rechnungenmust return 0 items anddownload_rechnungenmust never be reached (the folder is excluded, no 403). (Verified: exec 322653.) Under the old Files-and-Folders behaviour this returned the folder as 1 item and produced the 403.
See also
docs/playbooks/n8n-bms4-redis-password-rotation.md— bms-4REDIS_PASSWORDrotation across all 5 consumers (incl. this workflow’s stored credential) + the #4135 single-doc verification test.docs/pinbox24/pinbox24-workflow-engine.md— inventory of W3/W4 ingest workflows.docs/playbooks/n8n/n8n-workflow-creation.md— n8n workflow authoring conventions.docs/playbooks/n8n/n8n-bms4-stuck-execution-cleanup.md— how to unstick a workflow whose lock is legitimately stale.