p24-infra Docs

w3-w4-redeploy-idempotency

9 min read

Playbook: W3/W4 Redeploy Idempotency — making fixes survive a CI redeploy

Created: 2026-07-18 Source issue: radieu/p24-infra#4258 — [PLAN] W3/W4 redeploy not idempotent — patches lost on CI redeploy Owner: sys-admin / secret-manager on bms-1 (W3/W4 stack — see w3-w4-stack-operations.md) Status: tracking / coordination runbook. This document is the design-phase deliverable for the recurrence class; the individual fixes it sequences live in their own issues.

One-line rule of this runbook: on the Pinbox24 W3 (v32-prod) / W4 (v42-prod) stack, a redeploy restores last-committed git / image state and the SOPS-synced env, and reverts everything else. If a fix does not live in one of those three places, assume the next redeploy silently deletes it. “Fixed on the box” is not fixed.

See also: pinbox24-w3-w4-health-verification.md — if you’re here because a fix that worked yesterday looks broken again, run that playbook’s quick checklist first to confirm the symptom, then come back here for the redeploy-survival model.

1. What this is — the failure class

W3 and W4 on bms-1 redeploy through GitLab CI (image rebuild + push to ECR → docker compose pull + force-recreate) with env files re-injected from SOPS. Any change applied outside committed git/image state or the synced SOPS env is silently reverted on the next redeploy — with no error, no alert, and no diff, because the redeploy is doing exactly what it was told to do: reproduce the last-committed state.

This is a structural regression class, not a series of one-off mistakes. Three confirmed instances this session all share one shape:

“A redeploy restores last-committed git/image state and reverts fixes applied elsewhere.”

The point of this runbook is to make that shape visible, give a hard checklist for “will this survive the next redeploy?”, and record the ordered dependency chain that closes the class for good.

2. The redeploy model — what a redeploy actually restores

A “redeploy” on this stack is any of:

Every one of those rebuilds container state from a declared source. Nothing that only exists in running-container memory or in a hand-edited on-box file that no declared source reproduces survives.

The four layers of state — and which survive a redeploy

#LayerDeclared source (survives)Reverted on redeploy if the fix lives only…
1App codeThe image built from the GitLab repo (p24-v-3.2 / p24-back-ts)…as a manual edit inside the running container’s filesystem
2Persistent patchespersistent-patches/* files, volume-mounted at runtime (NOT baked into the image — Gap 8, 2026-07-11) and tracked in docs/bms-1/docker-compose-{w3,w4}.yml…as an edit to the mounted target inside the container instead of to the source patch file, or if the bind-mount is dropped on force-recreate
3Secrets / envSOPS (secrets/pinbox24-w3.env.sops, pinbox24-w4.env.sops, pinbox24-backends.env.sops, bms-servers.env.sops) → synced to the on-disk .env by secrets-sync.yml…as a hand-edited on-disk env value that a hardcoded plaintext copy in the deploy script re-clobbers (root cause #2855)
4Config / capabilityansible role + the git-tracked compose file…as an on-box cron/toggle/flag that no ansible role or compose file reproduces

Layers 3 and 4 are where the class bites. Layer 3 has a contested source of truth: SOPS is authoritative in principle, but the CI/image and the bms-1/GitLab deploy scripts still carry stale plaintext copies that win on redeploy, so credential rotations don’t stick. Layer 4 has no source of truth for on-box fixes — crons and toggles applied by hand are simply lost on rebuild.

3. The three confirmed lost patches

#What was lostLayerWhy it revertedFix / tracking
#4169W4 Mailgun idempotent-sync guard3 (secrets)The upsert_key guard shipped for W3 on 07-09 (see mailgun-mongodb-stale-credential-hang.md), but W4’s secrets-sync step never got it → v42-prod redeploy re-clobbered the Mailgun key → all W4 password resets down againParity fix #4170 (P2)
#3905W3 env value (rotated credential re-exposed)3 (secrets)Recurs (#3614 / #3066 / #3750 / #3216) because root cause #2855 — hardcoded creds in the bms-1/GitLab deploy scripts — was never fixed; every redeploy re-injects the old plaintext, so rotations don’t stickRotate after #2855 lands — see chain below
#3738W3/W4 Mongo rotation cron4 (config)Disabled 07-05 for a heredoc bug that is since fixed (#2639 / #2672) but never re-enabled, and never codified in ansible → a redeploy-adjacent capability silently lostCodify in ansible; re-enable per user decision (deferred to 2026-07-28+)

All three are the same bug wearing different clothes: the fix lived somewhere a redeploy does not reproduce.

4. Root causes

No single source of truth survives a redeploy for layers 3 and 4:

  1. Secrets (layer 3) — contested authority. SOPS is authoritative by policy, but the CI/image and the bms-1/GitLab deploy scripts carry stale plaintext copies that win on redeploy. Until the injection source is fixed (#2855), any W3/W4 rotation is burned on the next CI run.
  2. Config (layer 4) — no authority at all. Fixes applied on the box (crons, toggles, feature flags) are not codified in ansible or the image, so a rebuild has nothing to reproduce them from.

The general principle both violate: a redeploy is a pure function of its declared sources. Any state not reachable from those sources is, by construction, ephemeral.

5. The idempotency invariant

A fix is redeploy-safe if and only if re-running the deploy from its declared sources reproduces the fix byte-for-byte.

Concretely, before you consider any W3/W4 change “done”, it must land in exactly one of:

  • App/patch behaviour → a committed change to the GitLab repo or a persistent-patches/* source file that is bind-mounted by the git-tracked docker-compose-{w3,w4}.yml. Never an in-container edit.
  • A secret value → the SOPS file and every plaintext copy the deploy path reads must be eliminated or made to defer to SOPS (this is what #2855 fixes). Rotating a key without closing the plaintext-injection path is not a fix — it is a countdown.
  • A capability/config → an ansible role or the git-tracked compose file. Never a bare on-box cron.

If a change can’t be expressed as one of those three, it is not redeploy-safe and must not be recorded as resolved — track it as tech-debt with the reason.

6. Ordered dependency chain — the hardening effort

This is the sequence that closes the class. Order matters: rotating exposed keys before the injection source is fixed just burns the fresh value on the next redeploy.

① #2855  Fix the injection source FIRST
         Strip hardcoded creds from the bms-1 / GitLab deploy scripts so redeploy reads only SOPS.
         └─ Until this lands, ANY W3 rotation is burned on the next redeploy.
                 │
                 ▼
② #4170  Port W3 → W4 parity
         upsert_key + sha256-verify into the W4 secrets-sync step so a v42-prod redeploy
         cannot clobber synced W4 secrets (the guard W3 already has; #4169's lost patch).
                 │
                 ▼
③ #3738  Codify the Mongo rotation cron in ansible
         Not on-box. Re-enable per the user decision (deferred to 2026-07-28+).
                 │
                 ▼
④ #3216 (W4) / #3905 (W3)  ONLY NOW rotate the exposed keys
         With ①–③ in place the fresh value survives the next CI run. Doing this earlier
         means the rotation dies on the next redeploy.
StepIssueRoleGate before starting
①#2855sys-admin + human (GitLab MR merge)GitLab runner up (#2690); GITLAB_ADMIN_PAT group access (#3775)
②#4170sys-admin (secrets-sync.yml)① merged
③#3738sys-admin (ansible)User re-enable decision (revisit 2026-07-28+)
④#3216 (W4) · #3905 (W3)secret-manager①–③ complete

7. Preventive patterns (the guards each step installs)

These are the concrete techniques the chain codifies — reuse them for any new W3/W4 fix.

7.1 --force-recreate, never restart (layers 1–3)

docker restart reuses the environment baked in at container creation and does not re-read the host env file — a rotated credential on disk is ignored until the container is recreated. Every env-sensitive deploy/rotation step must docker compose up -d --force-recreate (or the rename-old + up -d pattern). This is the exact flag whose absence caused the #4169 mailgun hang.

7.2 upsert_key + sha256 verify (layer 3)

A secrets-sync step must (a) upsert each key idempotently rather than assume prior state, and (b) verify the value actually loaded by comparing the on-disk hash against the running container’s:

# Read-only, prints no secret value — hashes must match after a force-recreate.
grep "^KEY_NAME=" /path/to/on-disk.env | sha256sum
docker exec <container> sh -c "cat /proc/1/environ | tr '\0' '\n' | grep '^KEY_NAME=' | sha256sum"
# Differing hashes = the container is running a stale credential → recreate, don't restart.

This is the W3 guard (#4169) that #4170 ports to W4.

7.3 Codify in ansible / compose, never on-box (layer 4)

A cron, toggle, or flag applied by hand does not survive a rebuild. It must live in an ansible role or the git-tracked docker-compose-{w3,w4}.yml. #3738’s cron is the standing example.

7.4 Persistent patches stay bind-mounted, source in git (layer 2)

Patches are volume-mounted at runtime (Gap 8) and their source files are tracked. Edit the persistent-patches/* source, never the in-container target, and confirm the mount survived a force-recreate:

docker inspect <container> --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{println}}{{end}}'

8. Verification — before and after any W3/W4 redeploy

Before applying a fix, classify it against §5. If it is not expressible as one of the three declared sources, stop — it is not redeploy-safe.

After any redeploy (CI, nightly, or manual force-recreate), confirm the fix survived:

  • Secrets — on-disk env hash == running-container env hash for every rotated key (§7.2). No plaintext copy in the deploy script still shadows SOPS (§6 ①).
  • Patches — every persistent-patches/* bind-mount is present (docker inspect … .Mounts) and the patched behaviour is live.
  • Config — the cron/toggle/flag is present because ansible/compose declares it, not because someone re-applied it by hand.
  • App — the deployed image tag matches the intended build (GitLab pipeline history records the exact merged-<date>-<sha> tag).

If any box fails, the fix was not redeploy-safe — re-express it per §5 and re-file, do not re-apply on the box.

9. Error reporting (repo standard)

If any step in this chain fails, report before working around — do not silently patch on the box:

  1. Discord embed via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL.
  2. GitHub issue in radieu/p24-infra labelled bug.

Graph View

Backlinks