p24-infra Docs

w4-v42-jwt-exposure-verify-first

6 min read

W4 V42_* JWT set credential exposure — verify-first investigation (#3216 / #4339)

Status: investigation complete — verify-first no-op (no rotation, no force-recreate) Owner: sys-admin / infra-task worker on bms-4 (bms-1 SSH + SOPS read); SOPS writes delegate to secret-manager. Applies to: W4 (v42-prod + microservices) on bms-1 (94.23.26.113). Batch item 4/4 — the last and highest-blast-radius (JWT auth) item of the W3/W4 verify-first rotation batch. Runs after redis (#3956), LogDNA/Mailgun (#4333), and pm2 (#3905/#4337, W3).

This documents the method (digest-compare the W4 V42_* “JWT set” without ever printing a secret value) and the 2026-07-19 finding for v42-prod, which confirmed the JWT was already rotated on 2026-07-08 and that no key in the exposed set needs re-rotation. Method is identical to the W3 pm2 playbook (w3-pm2-jlist-exposure-verify-first.md).

1. What “exposed” was here (#3216)

On 2026-07-08 an SSH command ran cat /root/builds/.../s3-environment.env | cut -c1-50 to inspect S3 key structure. cut -c1-50 is insufficient masking — a short key name (JWT_TOKEN_SECRET= is 17 chars) leaves 33 chars of value visible at the cut boundary. That leaked a chat-transcript snapshot of several V42_* values. The exposure vector was a one-time chat leak, not an ongoing egress. V42_JWT_TOKEN_SECRET was rotated in the same session (Phase 2 re-run -W4Only). This investigation verifies, 11 days later, whether the rotated value stuck and whether any other key in the exposed set is genuinely unrotated.

Exposed set from #3216 (JWT + related): V42_JWT_TOKEN_SECRET (near-full, HIGH), V42_s3Bucket_api_secretAccessKey (partial 21/40, MEDIUM), V42_MAILGUN_PASSWORD (partial, LOW), PM2_SECRET_KEY (full, LOW), V42_DB_URI / NEW_MONGODB_URI admin-prefix (partial, MEDIUM — tracked as #3215).

2. Verify-first method (never prints a value)

All commands emit key names + sha256[:12] digests only. The raw values never cross the SSH boundary — docker exec … env and cat backend-environment.env are piped into a python3 running on bms-1; only the digest dict is returned. The SOPS side is digested on the SOPS host (bms-4) with the same script, so digests are directly comparable.

The shared digest script (/tmp/wf4339_digest.py) reads dotenv lines from stdin, normalises each value (rstrip('\r\n') + strip one matching surrounding quote pair), and prints {KEY: sha256hex[:12]} (or EMPTY). Run it against three sources per key:

  1. SOPS source-of-truth (bms-4): sops -d --input-type dotenv --output-type dotenv secrets/pinbox24-w4.env.sops | python3 wf4339_digest.py <KEYS>
  2. Deployed env file (bms-1): cat /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env | python3 …
  3. Live container process env (bms-1): docker exec v42-prod env | python3 …

Prefix mapping. SOPS holds each secret twice: the V42_-prefixed form (the source-of-truth the hardened deploy script docker-deploy-prod.sh reads, #4289) and a legacy unprefixed duplicate. The deploy renders the container’s unprefixed env var from the V42_-prefixed SOPS key. So compare container JWT_TOKEN_SECRET ↔ SOPS V42_JWT_TOKEN_SECRET, container s3Bucket_api_secretAccessKey ↔ SOPS V42_s3Bucket_api_secretAccessKey, etc.

Egress check: docker exec v42-prod pm2 jlist | python3 -c '<check any pm2_env.km_link is True>'. If km_link=True, PM2 Plus (app.pm2.io) is linked and process metadata streams off-box.

A key that is SOPS-wired + digest-matches (V42_ source) + not additionally leaked is a healthy no-op — do not rotate it (same rule as #4169 stale-deploy clobber, #4333 LogDNA no-op, #4337 W3 pm2 no-op).

3. Finding — 2026-07-19 (v42-prod)

All 10 keys of the #3216 exposed set: live container deployed env-file SOPS V42_* source-of-truth. Digests (sha256[:12], safe to record — not values):

Container key (unprefixed)live == fileSOPS V42_*match
JWT_TOKEN_SECRET05e656229897V42_JWT_TOKEN_SECRET 05e656229897✅
tokenGenSecret40c945e32c87V42_tokenGenSecret 40c945e32c87✅
tokenGenIVbb144d9443beV42_tokenGenIV bb144d9443be✅
s3Bucket_api_secretAccessKeycc95b7e1fef3V42_s3Bucket_api_secretAccessKey cc95b7e1fef3✅
s3Bucket_api_accessKeyId0259d9cea084V42_s3Bucket_api_accessKeyId 0259d9cea084✅
PM2_SECRET_KEY85bff01cda77V42_PM2_SECRET_KEY 85bff01cda77✅
MAILGUN_API_KEYf6169bcf017bV42_MAILGUN_API_KEY f6169bcf017b✅
pinbox24PublicSecretAccessKeyb754272566f3V42_pinbox24PublicSecretAccessKey b754272566f3✅
NEW_MONGODB_URIaacd12049d53V42_NEW_MONGODB_URI aacd12049d53✅
monitoringTokene7e139f34e16V42_monitoringToken e7e139f34e16✅
  • JWT (highest blast radius): the value live in v42-prod SOPS `V42_JWT_TOKEN_SECRET` the value rotated on 2026-07-08 (#3216). The rotation completed and stuck. No stale-clobber, no drift. Re-rotating would invalidate all active W4 sessions for zero security gain → do not re-rotate.
  • Deployed env-file == live process env for every key ⇒ container is correctly wired; no “edited-file-but-container-holds-old-value” drift (the #3712 docker restart hazard did not occur here).
  • MAILGUN_PASSWORD / mailgunPassword are NOT present in the v42-prod process env at all — the backend uses MAILGUN_API_KEY; SMTP password lives with the separate mailgun-prod microservice (pinbox24-backends.env.sops). The unprefixed MAILGUN_PASSWORD in pinbox24-w4.env.sops is legacy/unused for this container. Mailgun already verified valid in #4169 / #4333.
  • Endpoints healthy: https://api.w4.pinbox24.com/api → 200, https://mailgun-api.w4.pinbox24.com/ → 200; v42-prod up 11h, mailgun-v42-prod up 5h; 0 error signals (WRONGPASS/ECONNREFUSED/MongoServerError/uncaughtException) in the last 30 min. A broken/mismatched JWT secret would produce auth failures — none observed.

Verdict: 0 genuinely-compromised, unrotated keys → verify-first no-op. No SOPS rotation. No force-recreate.

4. One egress difference from W3 — PM2 Plus IS linked

Unlike W3 (km_link=False, #4337), v42-prod has PM2 Plus linked (km_link=True) — process metadata streams to app.pm2.io. This is a known, accepted monitoring dependency: do NOT remove PM2 Plus until it is replicated in Prometheus (#2142), and rotating the PM2 link keys (PM2_SECRET_KEY / PM2_PUBLIC_KEY) has its own tracker (#2537). It is not part of the JWT-set scope and does not change the no-op verdict — but it is the one standing exposure surface for W4 and should be closed out via #2537 when PM2 Plus is retired.

5. Non-security drift (housekeeping, same class as W3 stale V32_REDIS_PASSWORD)

pinbox24-w4.env.sops carries legacy unprefixed duplicates of most V42_* keys. The deploy renders only from the V42_* prefixed source (proven by monitoringToken: live/file e7e139f34e16 == SOPS V42_monitoringToken, while the unprefixed SOPS monitoringToken 5082c3976489 has drifted). Harmless — the unprefixed dupes are not what the container reads — but a secret-manager housekeeping reconcile (remove the unprefixed legacy duplicates, or re-align them) prevents future confusion. Same finding class as #4337’s stale V32_REDIS_PASSWORD. Not a breach; not auto-dispatched.

  1. secret-manager (low): reconcile/remove the legacy unprefixed duplicate keys in pinbox24-w4.env.sops (JWT_TOKEN_SECRET, PM2_SECRET_KEY, s3Bucket_api_secretAccessKey, monitoringToken [drifted], mailgunPassword, …) — keep only the V42_* source-of-truth set.
  2. #2537 (deferred): rotate PM2 link keys when PM2 Plus is retired (blocked on Prometheus replication #2142). Until then km_link=True is accepted.
  3. #3215 / MongoDB track: V42_NEW_MONGODB_URI admin-password exposure is a separate rs0 rotation (w4-mongodb-credential-rotation.md) — out of JWT-set scope, wired+matching here.
  4. #3216: the exposed-set premise is now resolved (JWT rotated + stuck; rest wired+matching). Leave close to a human — no auto-close (no merged-PR association; #4251 guard).

Graph View

Backlinks