w3-redis-local-migration

W3 Redis: OVH DBaaS → local redis-v32 container migration

Status: COMPLETE — all three phases done. Phase 1 (investigation) + Phase 2 (additive provisioning) landed via 3717 (PR #3729). Phase 3 (cutover) was subsequently authorized and executed live on 2026-07-11 (PRs #3732 + #3744) — V32_REDIS_HOST/PORT/PASSWORD in secrets/pinbox24-w3.env.sops were repointed from the OVH DBaaS (kr40258-001.dbaas.ovh.net:35689) to the local redis-v32 container, v32-prod was redeployed and verified connected (“Redis is connected with host - redis-v32” / “Redis is ready”), and all five W3 containers passed health checks. W3 and W4 now run at architecture parity — both on local Redis containers, zero production dependency on the OVH-managed DBaaS. See §“Phase 3 — completed” below. The remaining follow-up (decommissioning the now-unused OVH kr40258-001 instance) is tracked separately and is not blocking; the exposed-password concern (#3714) was already remediated by an independent rotation.

Note (history preserved): the Phase 1/2 sections below were written before Phase 3 ran and describe the pre-cutover state (live config still pointing at OVH, redis-v32 password accidentally coupled to V32_REDIS_PASSWORD, etc.). They are kept for the investigation record — read them as historical, superseded by the Phase 3 section at the end.

Related: #3712 (this migration, Phases 1–3), #3717 (Phase 1+2 investigation), #3906 (Phase 3 cutover request — filed on a stale drifted-config read after the cutover had already completed; see §“Phase 3 — completed”), #3467 (original W3 Redis DBaaS discovery), #3481 (superseded approach — OVH DBaaS rotation automation, paused), #3714 (incident — V32_REDIS_PASSWORD exposed during Phase 1 investigation, since rotated independently), docs/playbooks/w3-w4-rotation-orchestration.md.

---

Background

scripts/sops-reset-redis.ps1’s W3 branch has always assumed a local redis-v32 Docker container on bms-1 (unchanged since #3256), but #3467 (2026-07-09) confirmed W3’s real production Redis is an OVH-managed Private Database (kr40258-001, Valkey/Redis 7.2, kr40258-001.dbaas.ovh.net:35689) — there was no local container, so the rotation script’s W3 branch always failed closed (No such container) without ever actually rotating the real credential.

Rather than build new OVH-DBaaS-specific rotation automation (no straightforward rotation API, unlike Wasabi/Mailgun — scripts/ovh_redis_rotate.py exists but isn’t wired into the orchestrator), the decision (per #3712) is to provision a local Redis container for W3 mirroring W4’s already- working redis-v42 pattern, and migrate W3’s app config to use it. Once done, sops-reset-redis.ps1’s existing W3 logic becomes correct as originally designed, with zero new rotation-automation code needed.

---

Phase 1 — Investigation findings (read-only)

1. What W3 actually stores in Redis

Source: app-backend/helperFunction/crudService.js on bms-1 (/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/app-backend/) — the only file in the entire app-backend tree that references Redis (confirmed via a recursive case-insensitive grep excluding node_modules; the three files flagged by incident #3562 (config/env.config.js, controllers/auth.js, config/mailgun.js) were checked for presence only — none of them reference redis at all).

53:const redis = require("redis");
84:helperService.redisClient = redis.createClient(process.env.REDIS_PORT, process.env.REDIS_HOST);
88:if (process.env.REDIS_PASSWORD) { helperService.redisClient.auth(process.env.REDIS_PASSWORD); }
599:  const tempDoc = await helperService.redisGetData(`v3:register:${regId}`)
607:  await helperService.redisSetExData(`v3:register:${regId}`, JSON.stringify(doc), 5*60);
1920: const cacheDoc = await helperService.redisGetData(`v3:process:${processId}`);
1929:  await helperService.redisSetExData(`v3:process:${processId}`, JSON.stringify(doc), 60*5);
3884-3921: redisGetData / redisSetData / redisSetExData helpers (get/set/setex wrappers)

Finding: Redis is used exclusively as a short-TTL cache (v3:register:* and v3:process:* keys, both explicitly set with a 5-minute TTL via setex/redisSetExData). There is no connect-redis or express-session package in package.json or node_modules, and no other file in the app touches Redis at all — no session store, no queue, no long-lived cache.

2. Runtime usage pattern (connection count / memory / key count) — NOT completed this pass

This part of Phase 1 was not completed. An attempt to run read-only INFO/DBSIZE against the OVH DBaaS instance (using the authorized V32_REDIS_HOST/PORT/PASSWORD from secrets/pinbox24-w3.env.sops) hit a PowerShell-pipe-to-SSH BOM-injection bug that caused the AUTH command to fail and the password to be echoed back in redis-cli’s own error text — this became a credential-exposure incident (see #3714) and the check was abandoned mid-attempt rather than retried. See docs/playbooks/ssh-agent-pattern.md §5 for the root-cause writeup and the safe alternative method a future session should use to complete this specific check (unauthenticated PING/INFO first, or file-redirect into ssh instead of a PowerShell \| pipe).

3. Cold-start assessment

Given the source-code evidence, a cold-start (empty local Redis, no data migration) appears acceptable. The only data W3 stores in Redis are two categories of 5-minute-TTL cache entries (v3:register:*, v3:process:*) used to avoid recomputing register/process documents within a short window — not user sessions, not queues, not anything that would force-logout users or lose durable state on cutover. Worst case on a cold cutover: a handful of in-flight register/process requests recompute their cache entry once instead of hitting a warm cache, with no user-visible failure. This is not a live-traffic confirmation (the runtime usage-pattern check in §2 above was not completed), so treat this as a strong-but-not-fully-verified assessment — a human authorizing Phase 3 cutover should weigh this source-code-only finding accordingly, and a future session should complete the §2 check (safely) before or during a Phase 3 authorization decision if higher confidence is wanted.

---

Phase 2 — Provisioning status

redis-v32 container already exists on bms-1 (not newly created by this work)

docker ps -a on bms-1 (2026-07-11) shows redis-v32 already running (Up 12 hours at investigation time), auto-created as a side effect of infra-src/pinbox24/w3/docker-compose.yml’s existing redis: service block (already checked into this repo, container_name redis-v32, image redis:7-alpine) plus the docker-compose up -d redis step already present in secrets-sync.yml’s W3 force-recreate path (triggered by the 2026-07-11 00:07 UTC secrets-sync run for the generic-V32-key-sync fix, commit 38255c83/09bd74a3). This work did not create the container — it was already there when Phase 2 investigation began.

Structural verification (read-only docker inspect, scoped fields only — no env dump):

Field	redis-v32	redis-v42 (reference)
Image	redis:7-alpine	redis:7-alpine
Restart policy	unless-stopped	unless-stopped
Persistence	anonymous volume → /data	anonymous volume → /data
Network	test-net only (matches compose file: W3 backend/redis are test-net-only)	prod-v-4-net + test-net

Structurally, redis-v32 already mirrors redis-v42’s pattern correctly.

Health verification (read-only, credential-free): docker exec redis-v32 redis-cli ping (unauthenticated) returned NOAUTH Authentication required. — this confirms the container is up, responding to the Redis protocol, and enforcing password auth, without ever needing to know or use the password.

Known gap — the container’s current auth password is NOT yet independently generated

redis-v32’s command (redis-server --requirepass "$REDIS_PASSWORD") sources REDIS_PASSWORD from backend-environment.env via env_file: — and that file’s REDIS_PASSWORD is populated by secrets-sync.yml’s generic “strip V32_ prefix” sync from the live V32_REDIS_PASSWORD (currently the OVH DBaaS password). In other words: today, redis-v32’s own auth password is accidentally identical to the live OVH DBaaS production Redis password — not a deliberate assignment, just a side effect of the generic key-sync fix plus the pre-existing compose service definition.

This was not corrected in this pass, for two reasons:

1. Attempting to change the running container’s requirepass would require either (a) reading the current value from backend-environment.env to AUTH before CONFIG SET — explicitly out of the credential sources authorized for this work (only secrets/pinbox24-w3.env.sops was authorized, and reading backend-environment.env was separately denied by the orchestrating session’s permission classifier), or (b) recreating the container standalone outside the Docker-Compose-managed lifecycle — which would very likely be silently reverted by the next secrets-sync.yml run’s docker-compose up -d redis step (compose reconciles the running container back to its env_file-sourced config on drift).
2. Staging a brand-new, separate key (V32_REDIS_LOCAL_PASSWORD_PENDING, intended to be generated via New-RandomHex 20 matching sops-reset-redis.ps1’s own convention) was attempted and blocked, not completed. A delegated secret-manager sub-agent’s write to secrets/pinbox24-w3.env.sops was denied by the orchestrating session’s permission classifier: secret-store writes require genuine named human consent, which an agent-relayed delegation chain does not satisfy — even for a net-new, inert, non-production key with zero live consumers. No write occurred; secrets/pinbox24-w3.env.sops is unchanged by this work (verified: still 27 keys, V32_REDIS_LOCAL_PASSWORD_PENDING absent, existing V32_REDIS_HOST/PORT/PASSWORD untouched). This step needs a human running the Update-SopsKey command directly in an interactive session — see docs/sops-templates/pinbox24-w3.keys’s placeholder comment for the exact key name/value convention to use.

Practical implication: today’s accidental password coupling (redis-v32’s requirepass == current V32_REDIS_PASSWORD) is actually structurally what Phase 3 cutover would eventually want — it’s just premature (the live V32_REDIS_HOST/PORT still point at OVH, so nothing currently authenticates to redis-v32 using that shared value in production). A Phase 3 authorization decision should treat “generate a genuinely fresh, independent password for redis-v32 and update both the container and V32_REDIS_PASSWORD together, atomically, as part of the actual cutover” as the correct sequencing — not as two separate steps — since decoupling redis-v32’s password from backend-environment.env today, only to recouple them at cutover, would fight the compose-managed lifecycle for no lasting benefit.

Also directly relevant: #3714 documents that the live V32_REDIS_PASSWORD value (the one redis-v32 currently happens to share) was itself exposed during this investigation and needs rotating by a separately-authorized session — whoever picks up #3714’s rotation should be aware that rotating it will also change what redis-v32 currently answers to (until Phase 3 gives it its own independent value), though this has no live consumer today so it is not itself blocking.

---

What Phase 3 (cutover — NOT done here, NOT authorized here) would need

1. Explicit, separately-named human authorization (per #3712’s own hard scope limit).
2. A human first needs to add V32_REDIS_LOCAL_PASSWORD_PENDING (or decide to skip straight to a fresh value at cutover time instead) — this step was blocked in #3712’s own pass, see Phase 2 above. Decide redis-v32’s final password at cutover time — either promote that staged value once it exists, or generate fresh at that point — and set it on the container via a proper docker-compose-lifecycle-respecting update (updating backend-environment.env’s REDIS_PASSWORD source, i.e. the live V32_REDIS_PASSWORD in SOPS, in the same pass as the container update) so it doesn’t get silently reverted.
3. Update V32_REDIS_HOST → redis-v32 (or redis, the compose network alias) and V32_REDIS_PORT → 6379 in secrets/pinbox24-w3.env.sops, alongside the password.
4. Restart v32-prod to pick up the new connection target (the specific action #3712 explicitly forbids without separate authorization).
5. Given Phase 1’s cold-start assessment (§3 above) is source-code-only and not runtime-traffic- confirmed, consider completing the Phase 1 §2 runtime check (safely, per the fixed method in docs/playbooks/ssh-agent-pattern.md §5) as part of Phase 3 planning for higher confidence, though the current evidence (5-minute-TTL cache only, no sessions) is already fairly strong.
6. Once cutover is live and verified, sops-reset-redis.ps1’s existing W3 branch becomes correct and functional for the first time — no code change needed there.

---

Phase 3 — completed (2026-07-11)

The cutover described above as “NOT done here” was subsequently authorized and executed. This section records the actual outcome; the Phase 1/2 sections above are the pre-cutover investigation record and are superseded here.

What was done (per #3712’s final comment, PRs #3732 + #3744)

• secrets/pinbox24-w3.env.sops — V32_REDIS_HOST/V32_REDIS_PORT/V32_REDIS_PASSWORD repointed from the OVH DBaaS (kr40258-001.dbaas.ovh.net:35689) to the local redis-v32 container (PR #3732).
• Deploy + verify — v32-prod redeployed; logs confirmed “Redis is connected with host - redis-v32” / “Redis is ready”; redis-v32 PONG confirmed; all five W3 containers healthy; W3 HTTP health returned 200.
• Durability fix (PR #3744) — the cutover deploy first failed because the manually-staged redis-v32-environment.env lived inside a GitLab-CI-managed build dir and was wiped by an intervening pipeline run (briefly took v32-prod-reso/v32-prod-socket offline; v32-prod itself stayed up, consistent with Phase 1’s finding that W3 Redis is a cache with a MongoDB fallback). Fixed durably: the env file is now regenerated from SOPS on every secrets-sync run, immune to future CI wipes.

Result: W3 and W4 are at architecture parity — both run local Redis containers with zero production dependency on the OVH-managed DBaaS.

Verified current state (2026-07-11, this session, #3906)

• secrets/pinbox24-w3.env.sops live values: V32_REDIS_HOST = redis-v32, V32_REDIS_PORT = 6379, and V32_REDIS_PASSWORD key absent — redis-v32 runs passwordless, matching the redis-v42 precedent (the W3 app only calls redisClient.auth() when REDIS_PASSWORD is set).

sops-reset-redis.ps1 W3 branch — current behaviour (item 4 of #3906)

The script’s W3 branch is gated on V32_REDIS_PASSWORD existing in the SOPS file:

$oldW3Redis = Get-Kv $w3Lines "V32_REDIS_PASSWORD"
if ($oldW3Redis) { <rotate redis-v32> } else { "V32_REDIS_PASSWORD not found in W3 SOPS — skipping W3 Redis" }

Because the passwordless cutover removed V32_REDIS_PASSWORD from the file, the W3 branch now takes the skip path — a clean no-op rather than the old fail-closed No such container error. This is correct: there is no password to rotate on a passwordless container, so no rotation is required. No script change is needed. If a future decision gives redis-v32 a dedicated password (e.g. bringing it in line with a Tier-2 rotation policy), add V32_REDIS_PASSWORD back to the SOPS file and set the container’s requirepass atomically in the same pass — at which point this branch resumes rotating as originally designed.

About issue #3906

#3906 asked to design and execute the Phase 3 cutover, on the premise that “W3’s v32-prod still connects to the external OVH-managed Redis.” That premise was already stale when the issue was filed — the cutover had completed the same day (#3712 final comment; SOPS verified above). The most likely cause is the same drifted-static-config-file read that #3906 itself flags for W4: an on-server backend-environment.env still showing the old OVH host/port even though the live SOPS config and running container had already moved to redis-v32. No further cutover work is required; #3906 is superseded by the already-completed #3712 Phase 3.