Scope: rotating the self-hosted bms-4 Redis password (REDIS_PASSWORD, SOPS
secrets/n8n-bms4.env.sops). This is a sys-admin / infra-task operation (needs n8n API +
SOPS read + bms-4 SSH) — a dev-issue worker cannot execute it. Origin: the 2026-08-13 client
outage (#6283, live fix
#6281).
Why this playbook exists. A REDIS_PASSWORD rotation updated SOPS → .env →
docker-compose env → container restart, but did not update the n8n stored credentialxjD48jTskJlCyRY0 (“bms-4 queue Redis”). That credential is not env-driven — it lives in n8n’s
encrypted PostgreSQL vault. Result: WRONGPASS on every W4 Direct API Ingestion run →
rechnungen/gutshrifts/standard folders silently stopped ingesting. Same rotation-vs-consumer gap
class as the Mailgun/Wasabi misses this cycle. This is the analog of the
GRAFANA_ADMIN_PASSWORD “stored in its own SQLite DB” gap in
credential-rotation-180d.md.
The complete consumer list — a rotation MUST hit all 5
#
Consumer
How it gets the new value
Missable?
1
secrets/n8n-bms4.env.sops
SOPS source of truth — edited by secret-manager
no (it is the source)
2
bms-4/.env
secrets-sync.yml decrypts + SCPs it to /opt/p24-infra/bms-4/.env
no (automated)
3
bms-4/docker-compose.yml runtime
redis requirepass (${REDIS_PASSWORD}) and every n8n service’s QUEUE_BULL_REDIS_PASSWORD — applied on docker compose up -d (recreate)
no (automated by secrets-sync.yml)
4
redis server requirepass reload
docker compose up -d --no-deps redis restarts redis with the new .env
no (automated)
5
n8n stored credential xjD48jTskJlCyRY0 (“bms-4 queue Redis”, type n8n-nodes-base.redis)
python3 bms-4/sync-n8n-credentials.py --cred REDIS_PASSWORD — NOT env-driven
YES — this is the step that was missing
Consumers 2–4 are handled by secrets-sync.yml’s sync-bms-4 job. Consumer 5 is the trap:
the stored credential is decoupled from .env and must be pushed into the n8n vault explicitly.
Automation status (#6283)
REDIS_PASSWORD is now a phase-2 entry in bms-4/sync-n8n-credentials.py’s CREDENTIALS
manifest, PATCHed by credential id (xjD48jTskJlCyRY0). secrets-sync.yml’s sync-bms-4 job
already runs python3 sync-n8n-credentials.py (no args → phase 1+2) after n8n is confirmed
healthy, so consumer 5 now updates automatically on every deploy/rotation. The manual
--cred REDIS_PASSWORD step below remains the verification / out-of-band fallback (e.g. when a
rotation was applied without a full secrets-sync.yml run, or the automated sync logged a WARNING
because n8n was briefly down).
Rotation procedure
Order matters: rotate the value everywhere the server reads it before pushing the stored
credential, so redis and the stored credential agree.
Rotate the SOPS value (secret-manager role): update REDIS_PASSWORD in
secrets/n8n-bms4.env.sops. Never print the value. Canary-decrypt before commit.
Deploy — merge triggers secrets-sync.yml, or run it manually:
gh workflow run secrets-sync.yml --repo radieu/p24-infra
Its sync-bms-4 job: writes /opt/p24-infra/bms-4/.env, recreates redis (new requirepass),
waits for the redis healthcheck, recreates the 4 n8n services (they reconnect BullMQ with the
new QUEUE_BULL_REDIS_PASSWORD), waits for the n8n API to return 200, then runs
sync-n8n-credentials.py (consumer 5).
Confirm redis auth (read-only, never prints the password):
Push / verify the stored credential (consumer 5, idempotent):
ssh root@54.36.123.110 'cd /opt/p24-infra && python3 bms-4/sync-n8n-credentials.py --cred REDIS_PASSWORD'# expect: Summary: {'created': 0, 'updated': 1, 'skipped': 0, ...}# dry run first if unsure: ... --cred REDIS_PASSWORD --dry-run → "would PATCH"
Run the post-rotation verification below. Do not consider the rotation done until a single
document flows end-to-end through all 3 W4 ingest branches.
Post-rotation verification — #4135 single-document ingest smoke test
The lock-race test in n8n-redis-singleton-lock.md proves the
INCR/EXPIRE/DEL lock is correct; it does not prove the redis credential still authenticates or
that a real document flows found → uploaded → archived. This section is that end-to-end test, using
the per-branch single-document test webhooks from
#4135 (each limit:1, so exactly one real file
is exercised per branch):
test-w4-ingest-rechnungen
test-w4-ingest-gutshrifts
test-w4-ingest-standard
For each of the 3 branches, POST the test webhook and confirm all of the following:
Check
Pass condition
HTTP status
200 (a 500 "No item to return" / WRONGPASS = FAIL — stored credential still stale)
fileId
present in the response (Google Drive file was found)
w4RecordId
present (record was created in W4)
File archived
the source file moved to the archive folder
Redis heartbeat
exactly 1 item at lock_hb_<cat> (the expire heartbeat passed the item through)
regRecords row
one new register row for the document
No red alert
no Discord P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL alert fires within 5 min
# Substitute the real webhook base + secret from n8n; never echo secret values.for cat in rechnungen gutshrifts standard; do echo "=== test-w4-ingest-$cat ===" curl -s -o /dev/null -w "%{http_code}\n" -X POST \ "https://n8n.bms-4.infra.zintegrowana.online/webhook/test-w4-ingest-$cat"done
STOP-and-report rule: if any test webhook returns 0 items / non-200, or a branch does not
produce fileId + w4RecordId + archived file, STOP — do not declare the rotation complete.
Re-run consumer 5 (--cred REDIS_PASSWORD), re-check redis PING, and file/escalate per the
error-notification standard if it still fails.
Guardrails
Read-only where possible; never print the REDIS_PASSWORD value — reference the key name only.
Verify PONG before and after (step 3).
The stored credential PATCH is atomic and queue-safe: in-flight executions finish with the old
value, queued/future executions get the new one — no queue drain or workflow pause needed
(see the sync-n8n-credentials.py header, “QUEUE SAFETY”).