Playbook — bms-4 Redis rotation must also update the n8n stored credential xjD48jTskJlCyRY0
Trigger: The W4 Direct API Ingestion workflow (bms-4, id=MUjApqruo6H88ebw, runs :20/:30/:40
hourly on the three folders rechnungen / gutshrifts / standard) starts failing on every run
at its Redis singleton-lock node with:
WRONGPASS invalid username-password pair
The three category folders stop processing entirely (each is gated by a Redis lock —
w4ingest:lock:{rechnungen,gutshrifts,standard} — acquired via INCR before any work; see
n8n-redis-singleton-lock.md). Executions show status='error';
a handful may show success only because they short-circuited before the lock node.
This is distinct from pinbox24-bms1-redis-wrongpass.md
(that is V32_REDIS_* on bms-1 / W3 — a different Redis, a different server). This one is the
bms-4 n8n Redis and the same rotation-vs-consumer gap class as
n8n-supabase-credential-rotation.md.
Root cause (incident 2026-08-13, #6281)
REDIS_PASSWORD on bms-4 was rotated (Redis container recreated with the new password; SOPS +
on-server .env updated). n8n’s BullMQ queue picked up the new value immediately because it
reads QUEUE_BULL_REDIS_PASSWORD from the container env at start — so PING was PONG, the queue
was healthy, and the outage looked like “everything is fine except this one workflow”.
But the W4 ingestion workflow’s Redis lock nodes do not use the BullMQ env var. They authenticate with a separately-stored n8n credential:
| Credential id | Name | Type | Field |
|---|---|---|---|
xjD48jTskJlCyRY0 | bms-4 queue Redis (w4 singleton lock) | redis | password |
n8n credentials embed the secret inside the credential object (encrypted with the n8n instance key).
secrets-sync.yml does not touch n8n credentials — exactly like the Supabase-key case. So the
rotation updated the compose env var and every BullMQ consumer, but left xjD48jTskJlCyRY0 holding
the old password → WRONGPASS on INCR → the lock never acquires → nothing processes.
Fix — update the stored credential to the current REDIS_PASSWORD
Run on bms-4 (has docker + the age key). Never print the password — fingerprints only.
# 1. Verify the current password is live (PING → PONG) before writing anything.
RP=$(grep -m1 "^REDIS_PASSWORD=" /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"'\''\r')
docker exec bms-4-redis-1 redis-cli -a "$RP" PING # must print PONG
# 2. Export the stored credential decrypted, patch ONLY the password field, re-import.
docker exec bms-4-n8n-1 n8n export:credentials --id=xjD48jTskJlCyRY0 --decrypted --output=/tmp/cred.json
docker cp bms-4-n8n-1:/tmp/cred.json /tmp/cred.json
NEWKEY="$RP" python3 -c "import json,os; d=json.load(open('/tmp/cred.json')); c=d[0] if isinstance(d,list) else d; c['data']['password']=os.environ['NEWKEY']; json.dump(d,open('/tmp/cred.json','w'))"
chmod 0644 /tmp/cred.json # container 'node' uid must be able to read it (#2070)
docker cp /tmp/cred.json bms-4-n8n-1:/tmp/cred.json
docker exec bms-4-n8n-1 n8n import:credentials --input=/tmp/cred.json
# 3. Cleanup plaintext, unset the value.
rm -f /tmp/cred.json; docker exec bms-4-n8n-1 rm -f /tmp/cred.json; unset RPNo n8n restart is needed — credentials are decrypted from the DB per execution.
Verify (fingerprint match, then a green run)
# The credential's stored password fingerprint must now equal the env fingerprint.
docker exec bms-4-n8n-1 n8n export:credentials --id=xjD48jTskJlCyRY0 --decrypted --output=/tmp/v.json
CRED_FP=$(docker exec bms-4-n8n-1 node -e 'const f=require("fs"),c=require("crypto");const d=JSON.parse(f.readFileSync("/tmp/v.json"));const o=Array.isArray(d)?d[0]:d;process.stdout.write(c.createHash("sha256").update(o.data.password||"").digest("hex").slice(0,12))')
docker exec bms-4-n8n-1 rm -f /tmp/v.json
ENV_FP=$(grep -m1 '^REDIS_PASSWORD=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"'\''\r' | sha256sum | cut -c1-12)
[ "$CRED_FP" = "$ENV_FP" ] && echo "credential current ✅" || echo "STILL STALE ❌"Then confirm the next scheduled run (:20/:30/:40) reaches success and the lock node no longer
WRONGPASSes (query execution_entity for workflowId='MUjApqruo6H88ebw').
Prevention
Whenever REDIS_PASSWORD is rotated on bms-4, updating the compose env var is not sufficient —
also run the fix above for xjD48jTskJlCyRY0. This is now flagged on the REDIS_PASSWORD row of
../credential-rotation-180d.md. If more n8n workflows adopt the
singleton-lock pattern with their own stored Redis credential, add each one here.
See also
n8n-redis-singleton-lock.md— the lock pattern that consumes this credential.n8n-supabase-credential-rotation.md— same rotation-vs-consumer gap class.../pinbox24-bms1-redis-wrongpass.md— the W3/bms-1 Redis WRONGPASS (different server, do not confuse).