Both env files are in plaintext on bms-1 — NOT in SOPS (Part C covers fixing this).
Origin: incident #2376 leaked RABBIT_MQ_URL + REDIS_PASSWORD in a chat transcript; rotation
tracked in #2378.
⚠️ UPDATE 2026-07-11 (#3712/#3714): Part B below (OVH DBaaS Redis rotation) is OBSOLETE.
Both v32-prod and v42-prod have migrated off the shared OVH DBaaS Redis kr40258-001 onto
their own local Redis containers on bms-1 (redis-v32 and redis-v42 respectively, each
in its own docker-compose stack under infra-src/pinbox24/w3/ and infra-src/pinbox24/w4/).
They are independent instances now — rotating one no longer affects the other. The OVH
instance itself is unused and flagged for a decommission decision in docs/priorities.md.
Part B’s OVH-specific steps are kept below for historical reference only; §Part B′ replaces
it for any future Redis password rotation.
First read docs/playbooks/static-api-key-incident-rotation.md (the master rotation playbook).
This file is the backend-specific supplement.
Each backend and its Redis container are defined in the same docker-compose file
(infra-src/pinbox24/w3/docker-compose.yml, infra-src/pinbox24/w4/docker-compose.yml — mirror
these against whatever is actually deployed on bms-1 before trusting them) and both read
REDIS_PASSWORD from the sameenv_file (backend-environment.env) — rotating the password
is a same-file edit for both the backend and its Redis container, not a separate credential store.
RabbitMQ (RABBIT_MQ_URL) is consumed only by v42-prod (v32-prod uses a different queue setup).
Why this must run from the Windows dev session (not a bms-4 worker)
backend-environment.env on bms-1 (v42-prod only) + container recreate
bms-4 → bms-1 SSH is unreachable (rc 255)
REDIS_PASSWORD (local redis-v32 / redis-v42, since 2026-07-11)
backend-environment.env on bms-1, per platform (see §Part B′)
same backend-environment.env + a recreate of both the app container and its Redis container
bms-4 → bms-1 SSH unreachable
Hazard (RabbitMQ): changing the password on bms-4 immediately breaks the running v42-prod
connection. Update bms-1 and recreate back-to-back in a maintenance window.
Hazard (Redis): since the 2026-07-11 local-Redis cutover, W3 and W4 Redis instances are
independent — rotating one no longer affects the other. It still invalidates sessions on
that platform only. See §Part B′ for the current procedure and its two hazards (docker restart not reloading env_file; hand-written docker run dropping bind mounts).
Execute this playbook from a Windows dev session with C:\Users\konar\.ssh\id_ed25519.
OVH API access for REDIS_PASSWORD (Tier 2 → autonomous)
The OVH_APPLICATION_KEY / OVH_APPLICATION_SECRET / OVH_APPLICATION_CONSUMER_KEY are in
secrets/monitoring.env.sops. The consumer key must be validated once in the OVH portal before
the API can manage Redis.
One-time setup (if CK not yet validated or has wrong permissions):
# Run on bms-4: scripts/ovh_new_ck.py# Returns validationUrl — human opens it, approves, then new CK goes into monitoring.env.sops
After validation, the rotation is fully autonomous via:
POST /cloud/project/{projectId}/database/redis/{clusterId}/user/{userId}/credentials/reset
OVH discovery script:/tmp/ovh_discover_redis2.py on bms-4 (or re-generate from session history).
Requires SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt when running as root.
Have the safe-extraction pattern ready (read one key into $VAR, never echo).
Part A — RABBIT_MQ_URL (RabbitMQ user pinbox24 on bms-4)
Do A1–A4 back-to-back — the gap between A1 and A3 is the outage window.
A1 — change the RabbitMQ password (on bms-4; pick a strong value, never echo it):
# PLAYBOOK: v42-prod-credential-rotation.mdNEW_RMQ=$(openssl rand -base64 30 | tr -d '/+=' | head -c 32)docker exec bms-4-rabbitmq-1 rabbitmqctl change_password pinbox24 "$NEW_RMQ"# keep $NEW_RMQ in the shell only; do NOT print it
A2 — build the new RABBIT_MQ_URL (amqp URI with the new password; host/port/vhost unchanged
from the old value — read the old URI structure from bms-1 without printing it):
# PLAYBOOK: v42-prod-credential-rotation.md# On bms-1: extract old URI structure, substitute only the password segment.
A3 — update backend-environment.env on bms-1 + recreate backend:
# PLAYBOOK: v42-prod-credential-rotation.mdssh root@94.23.26.113 'set -e cd /path/to/v42-prod # dir holding docker-compose.yml + backend-environment.env # in-place replace ONLY the RABBIT_MQ_URL= line with the new URI (no echo of value) docker-compose up -d --force-recreate backend'
If not connected: re-check the URI password segment and vhost; recreate again.
Part B — REDIS_PASSWORD (OVH DBaaS Redis kr40258-001) — HISTORICAL, OBSOLETE since 2026-07-11
Do not follow this section for a new rotation — see §Part B′ below. Kept only so the OVH
API mechanics (ovh_redis_rotate.py, task-based changePassword flow) aren’t lost; that flow
was never confirmed to reliably complete (issue #3714 — the API accepted the change and
returned a task ID, but neither the old nor the new password worked afterward, and there was no
reliable way to query task completion for this legacy “Private Database” product). Both
backends are off this instance now, so the question is moot unless OVH DBaaS is un-decommissioned.
Affects BOTH v42-prod AND v32-prod. Both must be updated and restarted before the window ends.
B1 — reset Redis password via OVH API (on bms-4; result stored in file, never echoed):
# PLAYBOOK: v42-prod-credential-rotation.md# Run on bms-4 — reads OVH creds from monitoring.env.sops, outputs new password to /tmp/new_redis_pass (chmod 600)SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt python3 /opt/p24-infra/scripts/ovh_redis_rotate.py# Script: discovers project/cluster/user IDs, calls credentials/reset, writes password to /tmp/new_redis_pass
B2 — SCP new password to Windows (never print it):
# PLAYBOOK: v42-prod-credential-rotation.mdscp -i C:\Users\konar\.ssh\id_ed25519 root@54.36.123.110:/tmp/new_redis_pass $env:TEMP\new_redis_pass.txt$env:NEW_REDIS_PASS = (Get-Content $env:TEMP\new_redis_pass.txt).Trim()Remove-Item $env:TEMP\new_redis_pass.txt # clean up local temp# also clean up on bms-4ssh -i C:\Users\konar\.ssh\id_ed25519 root@54.36.123.110 'rm -f /tmp/new_redis_pass'
B3 — update BOTH env files on bms-1 (awk in-place, no echo of value):
If either fails: check REDIS_PASSWORD in the env file and the container logs for auth errors.
Part B′ — REDIS_PASSWORD (local redis-v32 / redis-v42 containers) — current procedure
Since 2026-07-11, both backends run their own local Redis container. Rotating one does not
affect the other — run this independently per platform. Both redis-vNN and the app container
read REDIS_PASSWORD from the samebackend-environment.env, so a single edit covers both.
⚠️ Two hazards discovered the hard way during 3714 — read before running anything:
docker restart <container> does NOT reload env_file. Container env is baked in at
docker create/docker run time and does not change on restart/stop/start. If you edit
backend-environment.env and only docker restart redis-vNN, the container keeps its OLD
REDIS_PASSWORD and the app will get WRONGPASS forever. You must fully recreate the
container (stop, rename-aside — never delete outright — then a fresh docker run/
docker-compose up -d) for the new password to take effect. This applies to the app container
too.
Never hand-write a docker run recreate command from memory or from an old playbook copy.
The docker run example that used to live in Part B above only carried --network and
--env-file — it silently dropped 7 bind-mounted hotfix patches (twilioService.js,
mongoose.js, filesUpload.js, excel_import.js, pushNotification.helper.js,
crudService.js, and /var/log) plus the --restart unless-stopped policy on v32-prod.
These patches are load-bearing production hotfixes, not optional. (One exception, for accuracy:
excel_import.js is not a p24-infra hotfix — it is a stock, git-tracked file in
pinbox24/p24-v-3.2 whose mount is a redundant no-op, since the image copy is byte-identical
(#4708). The rule below is unchanged: reproduce every mount, because reconstructing the set
from memory is exactly how the load-bearing five get dropped.) Before recreating any
container, always get its real current config from the source of truth:
Prefer the actual compose file for that service (infra-src/pinbox24/w3/docker-compose.yml,
infra-src/pinbox24/w4/docker-compose.yml) and docker-compose up -d --force-recreate <service> — compose knows the full desired state (mounts, network, restart policy, env_file)
and re-reads env_file correctly. This is the preferred method.
If compose can’t be used, extract the live config first (structural fields only, never
dump full docker inspect output — that includes env values):
and reconstruct the docker run from those real values — not from this playbook’s memory of
what the config “should” be.
B′1 — generate a new password locally (no OVH round-trip needed):
# PLAYBOOK: v42-prod-credential-rotation.mdAdd-Type -AssemblyName System.Web$env:NEW_REDIS_PASS = [System.Web.Security.Membership]::GeneratePassword(28, 0) -replace '[^a-zA-Z0-9]',''# keep in $env:NEW_REDIS_PASS only; never print it
B′2 — update REDIS_PASSWORD in backend-environment.env on bms-1 (stdin-piped, never on
the command line — a value on the ssh/ awk command line risks landing in shell history or
ps aux on the remote host):
# PLAYBOOK: v42-prod-credential-rotation.md# Write a small remote script (no secret in it) via Write + scp — NOT via PowerShell pipe-to-ssh,# which silently prepends a UTF-8 BOM to piped native-command input and corrupts the shebang/value.# See docs/playbooks/sops-windows-crlf.md for the same BOM class of bug in a different context.$env:NEW_REDIS_PASS | ssh -i C:\Users\konar\.ssh\id_ed25519 root@94.23.26.113 'sh /root/update_redis_pw.sh'
B′3 — recreate the Redis container AND the app container (compose preferred, see hazard #2
above). Verify byte-length of the updated line before recreating anything (REDIS_PASSWORD= +
28 chars + newline = 44 bytes for a 28-char password — adjust for your generated length):
Also check for WRONGPASS in the same window — a Redis is connected line alone does not
prove auth succeeded; it only means the TCP connection was established. Confirm no auth error
follows it, and ideally confirm real traffic (e.g. GET /api/i18n/langs 200 for W3).
B′5 — if it fails, roll back, don’t guess forward. Rename the working container aside
(never delete) before recreating; if the recreate is unhealthy, stop it, rename the broken one
aside, and rename the original back + docker start it. Diagnose from a stable state, not
while production is degraded.
Part C — Move bms-1 backend secrets into SOPS (remove plaintext)
Both backend-environment.env files store credentials in plaintext — root cause of the leak.
After A + B succeed, encrypt them:
Create secrets/pinbox24-w4.env.sops (V42_* keys) and secrets/pinbox24-w3.env.sops (V32_* keys)
from both v32 and v42 env files. Use the SOPS+age workflow:
Windows: [System.IO.File]::WriteAllText + UTF8Encoding($false).
Temp files MUST be named *.env.sops inside secrets/. Canary decrypt before git add.
Add a secrets-sync job or documented manual SCP so bms-1 env files are rendered from SOPS
on every redeploy, not hand-edited.
Add/update dev_r_services rows for both backends; set last_rotated.
Close-out
Update rotation-log entries: pending → completed for RABBIT_MQ_URL and REDIS_PASSWORD.
Confirm old RabbitMQ password is no longer accepted; old Redis password revoked at OVH.
Update dev_r_services.last_rotated / next_due for both backends.
Recreate-not-restart incident: docs/playbooks/mailgun-mongodb-stale-credential-hang.md — a docker restart after a credential change keeps the stale value; always --force-recreate every consumer container