rotation-scripts-audit-2026-07-08

scripts/rotate/ audit — mongosh use-statement, silent-no-op & propagation bugs

Issue: #3416 Audit date: 2026-07-09 Trigger: 2026-07-08 W3 incident (docs/pinbox24/incident-2026-07-08-w3-wasabi-mongo-cascading-regression.md). w3_app_mongodb_password.sh / w4_app_mongodb_password.sh began their mongosh payload with use <db> — a mongosh REPL shell-helper, not valid JavaScript. In mongosh --file (non-interactive) mode that silently aborts the script with no output and exit 0, so every later statement never ran and the rotation appeared to work but never did. Fixed in PR #3411.

This audit swept every other script in scripts/rotate/ (plus scripts/sops-sync-receiver.py) for the same class of “looks tested but silently no-ops” bugs:

1. Interpreter-mode mismatch — an interactive-only helper run in non-interactive mode (mongosh use, redis-cli inline, psql \-commands under -c, …).
2. Silent-failure / error-swallowing on the operation that actually matters (|| true, 2>/dev/null, unchecked exit codes, catch-and-continue), as opposed to on best-effort logging.
3. Incomplete propagation — does the script update every file/container/consumer of the credential it rotates, or only some? (The w3_app s3-environment.env gap is tracked separately in #3415 Task 4.)
4. SOPS/dotenv serialization — could a malformed line ({-prefix, CRLF/BOM, un-escaped value) be written, and could the canary decrypt pass falsely? (The invalid dotenv input line: { failure is tracked in #3415 Task 1.)

Method

• Static read of every script against the four criteria above (fan-out audit, then per-finding re-verification against monitoring/docker-compose.yml and scripts/sops-sync-receiver.py).
• Cross-checked docs/secrets-rotation-log.md and docs/playbooks/secret-manager-rotation-log.md for a COMPLETED entry naming each script (evidence of a confirmed end-to-end run).
• No live rotation was executed. Every rotator here operates on a production credential; the issue explicitly says to flag scripts that can’t be safely tested without touching a live secret rather than silently skip them. Those are marked NEEDS-LIVE-TEST below — no silent gaps.

Results

Script	Verdict	Confirmed prior run?	Finding
template.sh	✅ VERIFIED-CLEAN	reference template	Sound reference pattern; every || true is on best-effort logging, every critical op is || fail N.
w3_app_mongodb_password.sh	✅ FIXED (#3411)	yes (post-fix)	use w3_db → db.getSiblingDB("w3_db"). s3-environment propagation gap tracked in #3415.
w4_app_mongodb_password.sh	✅ FIXED (#3411)	yes (post-fix)	use w4_db → db.getSiblingDB("w4_db").
gf_rendering_renderer_token.sh	🔴 BUG — FIXED here	no	Restarted non-existent compose service grafana-image-renderer (real name renderer).
pdf_service_api_key.sh	🔴 BUG — FIXED here	no	Restarted only pdf-service; second consumer p24-infra-mcp kept the stale key.
rotate-now.ps1	🟠 BUG — FIXED here	one-off (#2620)	git commit exit never checked → false-success no-op. GH-secrets loop mismatch flagged (not changed).
update-sops-keys.ps1	🟠 BUG — FIXED here	one-off (#2620)	Same unchecked git commit. GH-secrets skip-when-GH_TOKEN-unset flagged (not changed).
n8n-bms4-api-key.ps1	✅ VERIFIED-CLEAN	—	Set-StrictMode -Latest + EAP=Stop; canary-before-move correct; propagation hand-off documented.
sops-sync-receiver.py	✅ VERIFIED-CLEAN	—	invalid dotenv input line: { is defended in depth (dict unwrap + double-encoded-string unwrap + explicit startswith("{") reject before write). Not a general flaw. See residual note below.
n8n-upsert-credential.js	✅ VERIFIED-CLEAN	—	No swallowed catches on the critical path; awaited DB writes; top-level .catch exits 1. One SUSPECT: no decrypt read-back to prove n8n can use the credential.
ovh-api-credentials.js	🟡 NEEDS-LIVE-TEST	no	Good pattern — verifies new creds with a signed GET /me (HTTP 200) before write, revokes old app after. Selectors self-declared unverified vs live UI; /me doesn’t exercise the granted scope.
discord-bot-token.js	🟡 NEEDS-LIVE-TEST	no	Has real capture guards (token length ≥ 50) so it won’t silently no-op, but MFA-submit / reset are not verified against Discord’s API.
ovh-api-credentials.sh	🟡 FLAG (latent)	no	Payload omits gh_secret_name/gh_secrets → receiver silently skips GH-Secret sync for OVH_INFRA_CLIENT_ID/SECRET. Latent (no GH consumer today). Not changed — see below.
discord-account-password.js	🟡 FLAG-FOR-HUMAN	no	Submit success inferred from absence of an on-screen error element; no re-auth with the new password before it overwrites the old → lockout risk.
mailgun-api-key.js	🟡 FLAG-FOR-HUMAN	no	Confirm-reset click is optional/swallowed and capture greedily grabs the first long string on the page → can capture the OLD key and exit 0. Highest silent-no-op risk of the JS set.
sentry-auth-token.js	🟡 FLAG-FOR-HUMAN	no	Scope checkboxes checked via .check().catch(()=>{}) (may silently miss) → wrong-scoped token declared success; old tokens revoked before the new one is validated (no rollback).
n8n-bms4-api-key.js	🟡 FLAG-FOR-HUMAN	no	Submit never confirmed; new key never test-called before overwrite; “Strategy D” full-page JWT scan can capture a stale key.
n8n-update-credential-playwright.js	🟡 FLAG-FOR-HUMAN	no	Clicks save, can’t catch n8n’s auto-dismissing toast, prints “verify manually” and still exits 0. Textbook silent no-op.

Fixes applied in this PR

1. gf_rendering_renderer_token.sh — wrong compose service name (highest severity)

The apply step ran docker compose up -d --no-deps grafana grafana-image-renderer. The compose service is named renderer (monitoring/docker-compose.yml:179); grafana-image-renderer is only the image tag (:181). docker compose up with an unknown service name fails the entire command, so — after the inline sed had already rewritten the token in .env — the run aborted: .env held the new token, neither grafana nor renderer was restarted, and the receiver dispatch (SOPS + GH-Secret sync) never ran. This is the closest analog to the #3411 mongosh incident: it looks like it ran, but leaves grafana/renderer on mismatched tokens and rendering broken. Fix: grafana-image-renderer → renderer in both the apply and the verify (docker compose ps).

2. pdf_service_api_key.sh — missed second consumer

PDF_SERVICE_API_KEY is consumed by two services: pdf-service (docker-compose.yml:481) and p24-infra-mcp (:510, a client that authenticates to pdf-service with the same key). The script restarted only pdf-service, so p24-infra-mcp kept the old key in its running env until an unrelated restart and its calls would 401 after rotation. Same incomplete-propagation class as the w3_app gap (#3415). Fix: restart both — docker compose up -d --no-deps pdf-service p24-infra-mcp.

3. rotate-now.ps1 + update-sops-keys.ps1 — unchecked git commit

Both run under Set-StrictMode -Off + $ErrorActionPreference="Continue" and never checked $LASTEXITCODE after git commit; a failed commit (nothing staged, hook rejection) fell straight through to git push + Write-Host "Pushed" and reported success. Fix: check the exit code after both commit and push; Write-Error + exit 1 on failure.

Flagged — NOT fixed here (needs human decision / live test)

• Playwright browser-automation rotators (discord-account-password.js, mailgun-api-key.js, sentry-auth-token.js, n8n-bms4-api-key.js, n8n-update-credential-playwright.js): all share the root cause of the 2026-07-08 incident class — they infer success from “a credential-shaped string was captured / no error element appeared,” none re-authenticate with the new credential before it overwrites the old one. The correct fix is a post-rotation functional check (e.g. call the target API with the new credential and assert 2xx before writing). That is a non-trivial change per script and can only be validated by running against the live UI/API, so it is flagged for human review, not blind-patched here — patching browser selectors without a live run risks introducing new silent failures. The model to copy is ovh-api-credentials.js (signed GET /me verify-before-write, revoke-after-verify).
• ovh-api-credentials.sh GH-Secret gap: adding gh_secret_name/gh_secrets would start pushing OVH_INFRA_CLIENT_ID/SECRET to GitHub Actions secrets. No workflow consumes them today, so creating those secrets may be undesirable — left for a human decision rather than silently adding a GH secret.
• sops-sync-receiver.py residual: the startswith("{") reject does not cover a JSON array-shaped value ([...]). No producer emits arrays, so this is theoretical; widen the guard to {/[ if strict-input hardening is wanted.

No script was silently skipped

Every rotator in scripts/rotate/ and scripts/sops-sync-receiver.py is accounted for above. None had a confirmed end-to-end COMPLETED run in the rotation logs except the two mongodb scripts (post-#3411) and the Mezmo/GH_TOKEN SOPS writes (via sops-set.ps1, unrelated). Every unverified rotator is marked NEEDS-LIVE-TEST or FLAG-FOR-HUMAN — per the incident lesson, no silent gaps.