p24-infra Docs

w3-w4-rotation-orchestration

19 min read

Playbook: W3 + W4 Rotation Orchestration

Related: issue #3451, #3256 (orchestrator split), #3449 (W3 secrets table cleanup), #3467 (Redis shared-instance doc/code contradiction investigation), #3542 (W4 readiness pass — mirror of W3 work), #3466 (W3 Wasabi IAM re-verification), #3550 (W4 Wasabi IAM key found stale/orphaned), #3545 (incident — WASABI_ADMIN keys accidentally exposed during #3542 investigation, rotation pending explicit human authorization), docs/playbooks/mezmo-key-rotation.md, docs/sops-templates/pinbox24-w3.keys, docs/sops-templates/pinbox24-w4.keys

This playbook was originally W3-only (#3451) and was generalized to cover W4 in #3542, once W4’s own Wasabi/Redis/integration/Mezmo-logging state had been independently re-verified rather than assumed from the W3 pattern.

⚠️ Known issue — Phase 4 (Redis) is UNSAFE / non-functional for W3 (#3467)

Do not trust Phase 4 (scripts/sops-reset-redis.ps1, V32_REDIS_PASSWORD) until this is fixed.

Investigation on 2026-07-09 (#3467) found:

  • W3’s real production Redis (v32-prod’s live REDIS_HOST/REDIS_PORT, matching V32_REDIS_HOST/V32_REDIS_PORT in pinbox24-w3.env.sops) is an OVH-managed Private Database (Valkey/Redis 7.2, service kr40258-001, host kr40258-001.dbaas.ovh.net:35689) — not a local Docker container.
  • There is no redis-v32 container on bms-1 (confirmed via full docker ps -a — only redis-v42 exists, correctly used by W4/v42-prod).
  • scripts/sops-reset-redis.ps1’s W3 branch runs docker exec -i redis-v32 redis-cli ... on bms-1, targeting that nonexistent container. It will always fail (No such container) and the script’s own error handling means SOPS is never updated — so it fails closed (no corruption), but it also can never actually rotate the real W3 Redis credential. Running sops-reset-pinbox24.ps1 -W3Only today silently no-ops on the Redis phase while giving the impression it succeeded.
  • It is confirmed NOT shared with n8n-bms4 — n8n-bms4 runs its own dedicated local Redis container (bms-4-redis-1 on bms-4, internal host redis), entirely separate. The stale doc claim that rotation required “bms-4 redis-cli + update n8n-bms4 stack simultaneously” has been removed from pinbox24-w3.keys.
  • The correct, already-existing rotation mechanism for the real instance is scripts/ovh_redis_rotate.py (OVH API changePassword against kr40258-001), used today for manual W3/W4 rotation per docs/playbooks/v42-prod-credential-rotation.md §Part B — but it is not wired into sops-reset-pinbox24.ps1/sops-reset-redis.ps1 and does not update SOPS automatically.

Until a follow-up issue fixes this: treat V32_REDIS_PASSWORD as not covered by the orchestrator’s autonomous rotation, despite being listed as Tier 2 below. Use -SkipRedis/-W3Only -SkipRedis or verify manually that the Redis phase no-op’d rather than silently trusting a “success” line for Phase 4.

Update (2026-07-11, 3714): the fix for this is in progress — docs/pinbox24/w3-redis-local-migration.md covers Phase 1 (investigation: W3’s Redis usage is a 5-minute-TTL cache only, no sessions — cold-start looks acceptable) and Phase 2 (a redis-v32 local container mirroring redis-v42 already exists on bms-1, structurally verified). Cutover (Phase 3 — pointing live V32_REDIS_HOST/PORT/PASSWORD at the new local container) is NOT done and NOT authorized yet — this section’s warning still applies until that separate, explicitly-authorized session happens. Also note: V32_REDIS_PASSWORD’s current live value was exposed during the Phase 1 investigation and needs rotating by a separately-authorized session — see #3714 — independent of the migration itself.

⚠️ Known issues — W4 (#3542, re-verified 2026-07-09)

Unlike W3, W4’s Redis container situation is correct: redis-v42 exists on bms-1, V42_REDIS_HOST equals the container name and V42_REDIS_PORT matches the container’s live configured port (verified read-only via docker ps/docker exec redis-cli CONFIG GET port — no values printed, booleans only). Phase 4 is structurally sound for W4. However, two other issues were found:

  • Wasabi IAM key is stale/orphaned (#3550). V42_s3Bucket_api_accessKeyId in secrets/pinbox24-w4.env.sops does not match any currently active Wasabi IAM access key across all 16 IAM users on the account (verified via list_users + get_access_key_last_used, read-only). This is a different failure mode than W3’s #3466 (wrong username hardcoded) — for W4 the target username pinbox24-bms1-s3 in scripts/sops-reset-wasabi.ps1 is confirmed correct, but the value currently in SOPS for that credential is stale. Phase 1 will still work correctly for a future rotation (it creates a new key for pinbox24-bms1-s3 and deletes that user’s current key, independent of whether SOPS’s old value was accurate) — but until #3550 is resolved, do not assume V42_s3Bucket_api_accessKeyId in SOPS reflects what v42-prod is actually running with.
  • Redis has no password currently set. V42_REDIS_PASSWORD exists as a key in secrets/pinbox24-w4.env.sops but its value is an empty string — confirmed via a boolean/length-only check (Get-Kv returns "", not $null; length 0), consistent with redis-v42 answering CONFIG GET port over redis-cli without an AUTH step during the Phase 4 re-verification above. scripts/sops-reset-redis.ps1’s W4 branch already handles this correctly — it no-ops with "V42_REDIS_PASSWORD not found in W4 SOPS - skipping W4 Redis" rather than attempting to rotate a nonexistent baseline (confirmed via -DryRun -W4Only, 2026-07-09) — but this means Phase 4 currently does nothing for W4, same practical effect as W3’s #3467 issue though for a structurally different reason (no container problem, just no password to rotate). Setting an initial requirepass is a live-write action out of scope for a read-only investigation — not filed as a separate bug since it isn’t “broken” so much as “not yet configured,” and the no-op is safe (fails closed, same as the empty-value guard everywhere else in this codebase).

Twilio/OneSignal/Jabber integration status (#3542, mirror of W3’s #3449): read-only investigation against the live v42-prod container (env-var name presence, compiled dist/ source grep for filenames only, node_modules package presence, log activity in the container’s short uptime window) found:

  • Twilio — actively used. SDK package present in node_modules; referenced in dist/apps/auth/auth.controller.js (auth flow), dist/globalHelpers/notifications/sendSMS.helper.js, and a dedicated sms_one_way process task. Zero log lines matched in the container’s uptime window (container had only been up a few hours at investigation time) — that absence is inconclusive, not evidence of disuse, given the source-level evidence.
  • OneSignal — appears dead. Env vars are present in the container, but zero matches for “onesignal” anywhere in the compiled dist/ tree and no OneSignal SDK package in node_modules. Strong candidate for the same “owner-confirmed inactive, remove from template” treatment already applied to PayU/Przelewy24 in pinbox24-w4.keys (2026-07-05) — needs an explicit owner confirmation before removal, not removed by this investigation. Kept as Tier 3 manual pending that confirmation.
  • Jabber/XMPP — present in code, activity unconfirmed. A dedicated messaging_jabber process task exists in the workflow task-type registry (dist/apps/processes/processInstances/ processTaskHelpers/tasks/messaging_jabber/), alongside sibling task types like discord_message and context_html_report — the code path is real, but whether any currently configured W4 workflow actually invokes the jabber task type was not determinable from a short-uptime log window. Kept as Tier 3 manual, unresolved — same “inconclusive” status #3449 reached for some of W3’s integrations.

None of the above were fixed live — all three are read-only findings per #3542’s explicit no-live-credential-rotation constraint. #3550 was filed as a follow-up bug for the Wasabi finding; the Redis empty-password and integration findings are documented here and in the #3542 issue comments rather than filed as separate bugs (neither requires an urgent live write to resolve).

Fixed — Phase 3 (MongoDB) was missing V32_PINBOX_PROD_DB (#3564)

A read-only source investigation of the deployed W3 app (#3564) found a fourth Mongo-URL variable, V32_PINBOX_PROD_DB, used by config/env.config.js’s hardcoded "production" fallback block for the mongojs-based (config/mongo_db.js) connection that serves /api/i18n/langs. It shares the same w3_app@w3_db MongoDB user as V32_MONGODB_URL/V32_PMONGODB_URL/V32_DB_URI, but was absent from scripts/sops-reset-mongodb.ps1’s W3 $uriKeyDefs list — every rotation left it on the old password while the other three keys rotated together, silently breaking any route that goes through the mongojs path (Mongoose is a separate, independently-cached connection with no retry — see #3564 for the full connection-handling writeup). This was confirmed as the root cause of the 2026-07-09 W3 “langs not loading” incident.

Fix: V32_PINBOX_PROD_DB is now included in Phase 3’s w3_app@w3_db rotation group (scripts/sops-reset-mongodb.ps1, same grouping-by-user@db logic as the other three keys — no change to that logic itself, just one more key in the list it looks at) and added to docs/sops-templates/pinbox24-w3.keys. The commit that manually re-synced the three drifted keys (94a787ac) did not touch this key — this fix prevents the same gap from reproducing the incident on the next rotation.

What this covers

scripts/sops-reset-pinbox24.ps1 is the master orchestrator for rotating Pinbox24 W3 (pinbox24-w3.env.sops) and W4 (pinbox24-w4.env.sops) Tier 1/2 credentials — pass -W3Only or -W4Only to scope a run to one app, or omit both to run both apps in the same pass. It runs four shared phase scripts in sequence, each internally branching on W3 vs W4 based on which scope flag(s) are set:

PhaseScriptRotates (W3)Rotates (W4)
1scripts/sops-reset-wasabi.ps1V32_s3Bucket_api_accessKeyId / secretAccessKey (Wasabi IAM)V42_s3Bucket_api_accessKeyId / secretAccessKey — ⚠️ current SOPS value stale, see #3550 above
2scripts/sops-reset-random-keys.ps1V32_JWT_TOKEN_SECRET, V32_monitoringTokenV42_JWT_TOKEN_SECRET, V42_tokenGenSecret, V42_tokenGenIV, V42_INVOICE_AUTH_TOKEN, V42_monitoringToken, V42_CONVERT_API (ConvertAPI JWT self-sign)
3scripts/sops-reset-mongodb.ps1V32_MONGODB_URL, V32_PMONGODB_URL, V32_DB_URI, V32_PINBOX_PROD_DB (MongoDB rs0 app-user password)V42_NEW_MONGODB_URI, V42_v3MongoUrl (MongoDB rs0 app-user password)
4scripts/sops-reset-redis.ps1V32_REDIS_PASSWORD — ⚠️ UNSAFE/non-functional for W3, see §Known issue above (#3467)V42_REDIS_PASSWORD — ⚠️ no-ops (empty baseline value), see §Known issues — W4 above

Phase 5 (sops-reset-mailgun.ps1) is W4-only and is skipped automatically under -W3Only (no W3 Mailgun domain exists) — already confirmed working for W4 prior to #3542 (uses V42_MAILGUN_EU_DOMAIN static config + V42_MAILGUN_API_KEY). Phase 6 (git commit) runs for whichever files were touched.

Each phase script can also be run standalone with its own -DryRun/-W3Only/-W4Only — see the header comment in each file.

How to run

Dry run (safe — no writes, no live credential changes)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
.\scripts\sops-reset-pinbox24.ps1 -DryRun -W3Only
.\scripts\sops-reset-pinbox24.ps1 -DryRun -W4Only

Dry-run output is unchanged in substance by the Mezmo logging additions (#3451 for W3, #3542 for W4) — Send-MezmoLog calls fire at every stage boundary exactly as in a live run, but until MEZMO_INGESTION_KEY_ROTATION is minted (see Setup required below) they are silent no-ops that never touch the network. Verified 2026-07-09: both -DryRun -W3Only and -DryRun -W4Only against the real SOPS files complete cleanly with no behavior change.

Acquire the rotation lock FIRST — MANDATORY before a live run (ADR 004)

A live run drives changeUserPassword on rs0 for the W3/W4 app users — non-idempotent live server-side writes — so per ADR 004 you MUST hold the per-secret advisory lock before starting it. Two sessions running this orchestrator against the same stack at once each mint different passwords and silently diverge (live rs0 vs SOPS/bms-1) — the #5925 race. A dry run performs no live write and does not need the lock.

Acquire a lock for each app secret the run rotates, before invoking the script:

# W3 run → W3_APP_MONGODB_PASSWORD ; W4 run → W4_APP_MONGODB_PASSWORD (acquire both for a full run).
$LOCK_ID = python scripts/rotation-log-entry.py acquire --secret W3_APP_MONGODB_PASSWORD --repo p24-infra --reason "#<issue>"
if ($LASTEXITCODE -ne 0) { throw "acquire denied/unverified — STOP, reconcile, do NOT run the live rotation" }

Hold the lock for the whole live run. sops-reset-pinbox24.ps1 does NOT call rotation-log-entry.py itself — it performs neither the acquire nor the Layer-2 re-check — so you must run the Layer-2 check manually, immediately before invoking the script, for each secret the run rotates:

# Right before .\scripts\sops-reset-pinbox24.ps1 — confirm this session still holds each lock.
python scripts/rotation-log-entry.py check $LOCK_ID
if ($LASTEXITCODE -ne 0) { throw "lock lost — do NOT start the live run; reconcile first" }

After the run completes and secrets-sync.yml has distributed, release it: python scripts/rotation-log-entry.py close $LOCK_ID (or fail $LOCK_ID --error "…"). Exit 3 = denied (another session holds it → reconcile, do not overwrite blind); exit 4 = DB unreachable → fail-closed, do not proceed. CLI reference: secret-manager-rotation-log.md.

Live run

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
.\scripts\sops-reset-pinbox24.ps1 -W3Only
.\scripts\sops-reset-pinbox24.ps1 -W4Only

Requires secrets/role-secret-manager.env.sops (Wasabi admin, MongoDB rs0 admin, VPS SSH key) to be decryptable. After a successful run: gh pr create + merge → secrets-sync.yml deploys the new pinbox24-w3.env.sops/pinbox24-w4.env.sops to bms-1 and force-recreates the v32-prod/v42-prod containers. Verify with https://api.w3.pinbox24.com/api/i18n/langs / https://api.w4.pinbox24.com/api/i18n/langs.

⚠️ Recreate (not docker restart) EVERY consumer of the rotated MongoDB credential. docker restart reuses the environment baked into a container at creation time and does not re-read the on-disk env file — a restarted container keeps the OLD credential even after the new env file lands on bms-1. The full consumer set is wider than the two backend containers named above:

  • W3 (w3_app): v32-prod
  • W4 (w4_app): v42-prod, mailgun-v42-prod, and s3-v2-v42-prod

mailgun-v42-prod in particular is deployed on a separate compose file (/root/mailgun-prod) and is easy to forget — that omission caused the 2026-07-10 silent mailgun outage (mailgun-mongodb-stale-credential-hang.md): the w4_app password was corrected but mailgun-v42-prod was never recreated, so every inbound webhook hung on a dead MongoDB connection and ~450 emails produced zero records with no error. The per-container recreate + sha256 verification steps are in w4-mongodb-credential-rotation.md / w3-mongodb-credential-rotation.md. If secrets-sync.yml handles the recreate, confirm its restart step uses --force-recreate for all of these containers, not just v32-prod/v42-prod.

Note (W4 only): docs/sops-templates/pinbox24-w4.keys records that no secrets-sync.yml job exists yet for W4 — deployment is currently manual (scp from dev or bms-4 SSH). Confirm the manual deploy step actually happened after a W4 SOPS-only PR merges; do not assume it auto-syncs the way W3 does.

Useful combinations:

# Skip MongoDB or Redis phases individually (e.g. re-running after a partial failure)
.\scripts\sops-reset-pinbox24.ps1 -W3Only -SkipMongoDB
.\scripts\sops-reset-pinbox24.ps1 -W4Only -SkipRedis
 
# Run a single phase standalone
.\scripts\sops-reset-wasabi.ps1 -W3Only -DryRun
.\scripts\sops-reset-wasabi.ps1 -W4Only -DryRun

Mezmo stage-level logging (issue #3451 for W3, #3542 for W4)

Send-MezmoLog (in scripts/lib/sops-common.psm1) ships one structured log line per stage boundary to Mezmo, so a failed rotation can be pinpointed by stage without watching the terminal live or SSHing into servers after the fact. Wired into the 4 active phase scripts plus the orchestrator itself, for both W3 and W4 code paths (W4 wiring landed in #3542, mirroring #3461’s W3 pattern exactly — same stage names, same call sites, App parameter is "sops-reset-w3" or "sops-reset-w4" depending on which app the call site belongs to), at:

  • run-start / run-complete (each phase script + the orchestrator)
  • <phase>-generate — per-key value generation (random keys, Wasabi IAM create)
  • <phase>-sops-write — per-key SOPS write success/failure
  • mongodb-rotate / mongodb-verify — mongosh password change + post-rotation auth re-check
  • redis-rotate — Redis CONFIG SET requirepass result
  • phase-complete — end of each phase script
  • run-exception — any exception caught by the orchestrator, before it rethrows and aborts

Never a secret value — every -Message/-Meta call site passes only key NAMES (e.g. V32_JWT_TOKEN_SECRET), stage names, container names, booleans (dryRun, success), and counts. Send-MezmoLog itself does not scrub input — this is enforced by what every call site in this repo is written to pass, not by the function. Do not add a call site that interpolates a rotated value, a MongoDB URI, a Wasabi secret key, etc.

What a log line looks like

Each call ships as one line in the Mezmo lines array:

{
  "line": "SOPS write succeeded",
  "app": "sops-reset-w3",
  "level": "info",
  "meta": { "key": "V32_JWT_TOKEN_SECRET", "dryRun": false, "stage": "random-key-sops-write" }
}

Shipped to https://logs.mezmo.com/logs/ingest?hostname=windows-dev&tags=sops-rotation — every call from this repo’s rotation scripts carries hostname=windows-dev and tags=sops-rotation, and app=sops-reset-w3 or app=sops-reset-w4 depending on which app’s code path emitted the line.

Finding a failed rotation by stage in the Mezmo console

Once live logging is flowing (see Setup required below):

  1. Open https://app.mezmo.com → Log Analysis.
  2. Filter by app:sops-reset-w3 or app:sops-reset-w4 to isolate one app’s orchestrator output.
  3. Filter by level:error to jump straight to failures, or level:warn for degraded-but-continued paths (e.g. MongoDB auth verify failed but SOPS was still updated — see mongodb-verify).
  4. Use the meta.stage field to see exactly which stage failed (e.g. mongodb-sops-write, wasabi-key-generate, run-exception) — cross-reference against the stage table above.
  5. Tag filter tags:sops-rotation plus a time-range around the run (the orchestrator logs run-start/run-complete timestamps you can bracket).
  6. Via API instead of the UI: use the Mezmo Log Analysis API (docs.mezmo.com/2.8/log-analysis-api) with the same app/level/meta.stage filters — the service key for API access is P24_INFRA_MEZMO_SERVICE_KEY in monitoring.env.sops (see docs/playbooks/mezmo-key-rotation.md).

Current W3 secrets table

Source of truth: docs/sops-templates/pinbox24-w3.keys (cleaned up in #3449 / PR #3450). Format: KEY_NAME|tier|provider|rotation_script|notes. Tier 1 = autonomous, Tier 2 = semi-auto (requires live SSH/mongosh/redis-cli verification), Tier 3 = human-only, Tier 0 = not a secret.

KeyTierRotation
V32_JWT_TOKEN_SECRET1sops-reset-random-keys.ps1
V32_monitoringToken1sops-reset-random-keys.ps1
V32_s3Bucket_api_accessKeyId / secretAccessKey1sops-reset-wasabi.ps1
V32_MONGODB_URL / V32_PMONGODB_URL / V32_DB_URI / V32_PINBOX_PROD_DB2sops-reset-mongodb.ps1
V32_REDIS_PASSWORD2sops-reset-redis.ps1 — ⚠️ non-functional for W3, see §Known issue (#3467)
V32_LOGDNA_KEY1NOT this orchestrator — see out-of-scope below

Out of scope for this orchestrator

  • Mailgun (V32_MAILGUN_USER_NAME, V32_MAILGUN_API_KEY) — #3449 (resolved via PR #3547, landed on main concurrently with #3542) confirmed V32_MAILGUN_API_KEY has a live consumer in the s3-v32-prod container family (not dead), but its key-type/domain-scope is still unconfirmed — kept Tier 3 manual, not promoted to automated rotation.
  • Twilio / OneSignal / Jabber (W3) — same pr #3547 resolution: all three are wired with real code + SDK (not dead scaffolding), but live-traffic usage was inconclusive in the short log window available at investigation time. Kept Tier 3 manual pending a longer-window or provider-dashboard follow-up. See docs/sops-templates/pinbox24-w3.keys for the full per-key investigation notes. (W4’s equivalent investigation — #3542 — reached a more conclusive split: Twilio active, OneSignal apparently dead, Jabber inconclusive — see the W4 table below; W3 and W4 are separate codebases/deployments and their integration usage does not necessarily match.)
  • V32_LOGDNA_KEY — this is the per-server Mezmo ingestion key for W3’s own log shipping (bms-1’s logdna-agent, per docs/playbooks/mezmo-key-rotation.md’s per-server-key pattern). It rotates via the secret-manager role’s Mezmo API flow described in that playbook, not via sops-reset-pinbox24.ps1. Do not confuse it with MEZMO_INGESTION_KEY_ROTATION below — that is a different, not-yet-created key used by this orchestrator to log its own rotation runs, while V32_LOGDNA_KEY is what ships W3’s application logs generally.

Current W4 secrets table

Source of truth: docs/sops-templates/pinbox24-w4.keys, cross-checked against the live secrets/pinbox24-w4.env.sops key list (names only) on 2026-07-09 as part of #3542. Same tier convention as the W3 table above.

KeyTierRotation
V42_JWT_TOKEN_SECRET1sops-reset-random-keys.ps1
V42_tokenGenSecret / V42_tokenGenIV1sops-reset-random-keys.ps1
V42_INVOICE_AUTH_TOKEN1sops-reset-random-keys.ps1
V42_monitoringToken1sops-reset-random-keys.ps1
V42_CONVERT_API1sops-reset-random-keys.ps1 (ConvertAPI JWT self-sign, 90-day expiry) — note: docs/sops-templates/pinbox24-w4.keys marks this REMOVED 2026-07-05 — not in running env, but the rotation script actively writes it and -DryRun -W4Only confirms it still runs; the template annotation is stale and should be corrected in a template-cleanup follow-up, not treated as authoritative
V42_s3Bucket_api_accessKeyId / secretAccessKey1sops-reset-wasabi.ps1 — ⚠️ **current SOPS value stale/orphaned, see 3550
V42_NEW_MONGODB_URI / V42_v3MongoUrl2sops-reset-mongodb.ps1
V42_REDIS_PASSWORD2sops-reset-redis.ps1 — ⚠️ no-ops, empty baseline value, see §Known issues — W4 above
V42_MAILGUN_API_KEY2sops-reset-mailgun.ps1 (W4-only phase) — confirmed working prior to #3542
V42_twilioAccountSid / AuthToken / FromNumber3manual — actively used (see integration findings above), keep manual pending an SMS-provider-specific automation decision
V42_onesignal_USER_AUTH_KEY / APP_AUTH_KEY / APP_ID3manual — appears dead, candidate for owner-confirmed removal like W4’s PayU/Przelewy24 cleanup (2026-07-05)
V42_jabber_JID / PASSWORD / HOST3manual — present in code (messaging_jabber task type), activity inconclusive
V42_PM2_PUBLIC_KEY / SECRET_KEY3manual — PM2 Plus monitoring, rotates via app.pm2.io

Stale/legacy keys still present in the live SOPS file

Twelve V42_-prefixed keys are still physically present in secrets/pinbox24-w4.env.sops despite docs/sops-templates/pinbox24-w4.keys marking them REMOVED — not in running env on 2026-07-05/07: V42_CRON_NODE_ENV, V42_CURRENT_SERVER_URL, V42_FRONTEND_URL, V42_G_MAP_API, V42_MAILGUN_BASE_URL, V42_MAILGUN_EU_DOMAIN, V42_MAILGUN_EU_HOST, V42_MAILGUN_SUFIX, V42_MAILGUN_US_DOMAIN, V42_RABBIT_MQ_URL, V42_Test, and V42_CONVERT_API (which, as noted above, is actually not dead — the template annotation for that one specifically is wrong). The other eleven were confirmed not referenced by any of the four rotation scripts and are believed genuinely unused, matching the template’s own “not in running env” audit — this table doesn’t re-verify each of them individually beyond confirming the rotation scripts never touch them. Recommended follow-up (not done here — out of scope for a read-only/logging-only issue): a template + SOPS cleanup pass removing these twelve stale keys from both files, correcting the V42_CONVERT_API annotation, and re-running the diff to confirm SOPS and the template agree.

secrets/pinbox24-w4.env.sops also contains a set of unprefixed duplicate keys with the same values as their V42_-prefixed counterparts (DB_URI, JWT_TOKEN_SECRET, MAILGUN_PASSWORD, MAILGUN_USER_NAME, PM2_PUBLIC_KEY, PM2_SECRET_KEY, s3Bucket_api_accessKeyId, s3Bucket_api_secretAccessKey, s3Bucket_BUCKET_NAME, s3Bucket_endpoint) — these are not documented in the template at all. They match the naming convention docker-deploy-prod.sh maps V42_* SOPS keys to inside the container (per the equivalent comment in pinbox24-w3.keys), so they are likely a historical snapshot of deploy-resolved names rather than a second independent credential set — confirmed for the Wasabi pair specifically (s3Bucket_api_accessKeyId holds the identical, equally-stale value as V42_s3Bucket_api_accessKeyId). Not verified for the other eight; flagged here for a future cleanup pass rather than assumed safe to delete.

Out of scope for this orchestrator (W4)

Same rationale as the W3 table above — Twilio, OneSignal, and Jabber/XMPP keys remain Tier 3 manual regardless of the #3542 usage-verification findings (Twilio active / OneSignal dead / Jabber inconclusive) — a live SMS/push/XMPP provider is still Tier 3 human-only by policy even when known to be actively used; usage verification informs documentation and cleanup decisions, not automation eligibility.

Setup required before logs actually reach Mezmo

Current state (as of #3542): Send-MezmoLog calls are safe no-ops for both W3 and W4. No new Mezmo credential was created as part of implementing #3451 or #3542 — creating a brand-new ingestion key requires a separate, explicit human-authorized secret-manager operation, which is deliberately not part of either change (see the respective PRs for rationale: the auto-mode classifier requires explicit named consent before minting a new secret-store entry, and that consent hasn’t been given yet).

Until that follow-up runs:

  • $env:MEZMO_INGESTION_KEY_ROTATION is unset in every real environment.
  • Send-MezmoLog detects this and returns immediately — no HTTP call is attempted, so there is no network dependency, no latency, and no failure mode introduced by either issue.
  • The rotation scripts (sops-reset-wasabi.ps1, sops-reset-random-keys.ps1, sops-reset-mongodb.ps1, sops-reset-redis.ps1, sops-reset-pinbox24.ps1) work exactly as they did before these issues — same output, same behavior, same exit codes — just without the new observability, for both W3 and W4 code paths.

To enable live logging, a human or secret-manager session must:

  1. Follow the per-server ingestion key pattern in docs/playbooks/mezmo-key-rotation.md §B2 to create a new dedicated ingestion key via the Mezmo API (POST https://api.mezmo.com/v1/config/keys?type=ingestion, authenticated with P24_INFRA_MEZMO_SERVICE_KEY). This is a new logical key, not tied to a physical server — name it p24-rotation-scripts (distinct from any p24-<hostname> per-server key and distinct from V32_LOGDNA_KEY).
  2. Store the value as MEZMO_INGESTION_KEY_ROTATION in secrets/monitoring.env.sops (where the other Mezmo keys already live), following the safe SOPS-write pattern in docs/playbooks/mezmo-key-rotation.md (Windows: scripts/sops-set.ps1, never >/Out-File).
  3. Register the new key in docs/sops-templates/monitoring.keys and add a row to docs/playbooks/mezmo-key-rotation.md’s per-server-key table (as a logical rotation-scripts entry — it has no server to ship a systemd/Docker agent to; it’s consumed directly by Invoke-RestMethod from Send-MezmoLog, not by logdna-agent).
  4. On a Windows dev machine with the key present in monitoring.env.sops (decrypted into $env:MEZMO_INGESTION_KEY_ROTATION before running a rotation script), Send-MezmoLog calls start reaching Mezmo automatically — no code change needed, since the function already checks for the env var on every call.
  5. Once minted, both W3 and W4 code paths start logging immediately (no further code change needed — #3542 already wired app=sops-reset-w4 call sites; they’ve been silent no-ops purely because the env var isn’t set yet, same as W3’s call sites since #3461). A further follow-up issue can extend the same pattern to et-op/et-lager if those gain their own SOPS rotation scripts later.

No live rotation and no new Mezmo key were run/created as part of implementing #3451 or #3542 — only -DryRun -W3Only (#3451) and -DryRun -W4Only (#3542) were exercised (see the respective PRs for verification output).

Graph View

Backlinks