p24-infra Docs

w3-pm2-jlist-exposure-verify-first

5 min read

W3 pm2 jlist credential exposure — verify-first investigation (#3905 / #4336)

Status: investigation complete — verify-first no-op (no rotation, no force-recreate) Owner: sys-admin / infra-task worker on bms-4 (bms-1 SSH); SOPS writes delegate to secret-manager. Applies to: W3 (v32-prod-*) containers on bms-1 (94.23.26.113). Same method applies to the rest of the W3/W4 rotation batch — the W4 V42_* JWT set (#3216) was verified next in w4-v42-jwt-exposure-verify-first.md (#4339, also a no-op).

This documents both the method (how to verify-first a “creds exposed via pm2 jlist” report without ever printing a secret value) and the 2026-07-19 finding for v32-prod-reso, which corrected the issue’s premise.

1. What “exposed via pm2 jlist” actually is here

v32-prod-reso is a Docker container (compose project p24-v-32, service reso, dir /home/gitlab-runner/builds/<hash>/0/pinbox24/p24-v-3.2), not a host pm2 process. Inside it:

docker-compose (env_file:) → container Config.Env (41 vars incl. secrets)
  → CMD: pm2-runtime start ecosystem.config.js
    → pm2 records the inherited process environment in pm2_env
      → `docker exec v32-prod-reso pm2 jlist` dumps it

So the credentials appear in pm2 jlist because pm2-runtime records the inherited process environment — the standard way pm2 works. Reaching it requires root / docker access on bms-1 (the top trust boundary; equivalent to /proc/<pid>/environ or docker inspect).

2. Verify-first method (never prints a value)

All commands emit key names + sha256[:12] digests only. Never cat/echo/docker inspect raw env. Raw JSON is piped into a python3 running on bms-1 (or on the SOPS host); only names/digests cross the SSH boundary.

  1. Enumerate container cred key names: docker exec v32-prod-reso pm2 jlist | python3 -c '<extract [..] json, print pm2_env keys>' (pm2 prepends a non-JSON line — slice from first [ to last ] before json.loads.)
  2. Digest each container cred: hashlib.sha256(value)[:12] inside the same remote python.
  3. Digest the SOPS source of truth on the SOPS host: sops -d --input-type dotenv --output-type dotenv secrets/pinbox24-w3.env.sops | python3 … The container strips the V32_ prefix, so container DB_URI ↔ SOPS V32_DB_URI, etc.
  4. Compare digests. Match ⇒ container runs the SOPS-managed value (healthy, wired). Mismatch ⇒ drift — investigate whether live or SOPS is authoritative before any write.
  5. Check external egress: is PM2 Plus/keymetrics linked? (km_link set + an interactor/agent process running). If km_link=False and no agent process, the env is not streaming off-box.

A key that is SOPS-wired + digest-matches + not externally leaked is a healthy no-op — do not rotate it (same rule proven by #4169 stale-deploy clobber and #4333 LogDNA no-op).

3. Finding — 2026-07-19 (v32-prod-reso)

  • ~22 credential env vars exposed via pm2 jlist (Mongo/DB URIs, JWT_TOKEN_SECRET, LOGDNA_KEY, s3Bucket_api_*, onesignal_*, twilio*, jabber_*, monitoringToken, MONGODB_W3_APP_PASSWORD, REDIS_*, …).
  • 20/22 digests exactly match secrets/pinbox24-w3.env.sops (V32_*). Container is correctly SOPS-wired.
  • S3_ACCESS_KEY_ID / S3_SECRET_ACCESS_KEY are container-only aliases whose digests equal the SOPS-managed s3Bucket_api_accessKeyId / s3Bucket_api_secretAccessKey (same value, second name).
  • One drift: REDIS_PASSWORD — container value = local-redis password (matches REDIS_LOCAL_PASSWORD); SOPS V32_REDIS_PASSWORD still holds the decommissioned OVH DBaaS Redis (kr40258-001, #3712) value. Stale SOPS, not used live (W3 is on local redis-v32). Harmless — a secret-manager housekeeping reconcile, not a breach.
  • No external egress: PM2 Plus not linked (km_link=False, no interactor/keymetrics agent).
  • Endpoints healthy: v32-prod (api), v32-prod-socket (socket), v32-prod-reso (reso) all return HTTP 200 (local probe on bms-1).

Verdict: 0 genuinely-compromised keys → verify-first no-op. No rotation. No force-recreate.

4. The issue’s proposed fix does not apply

The tracking issue proposed “adjust the pm2 ecosystem config so jlist no longer dumps secrets (env sourced from file, not inline args).” Empirically false premise:

  • /app/ecosystem.config.js is 14 lines with no env: block and no secret literals (only name/interpreter/script/log-path refs to non-secret process.env vars).
  • docker-compose.yml sources the reso env via env_file: (5 directives) — no inline literals.

There is no inline env to remove. Secrets are in pm2 jlist purely via process-env inheritance. No pm2-ecosystem or compose edit removes them while the Node app reads process.env.

5. The only real fix — and why it’s a human/dev-coder decision

To keep secrets out of pm2 jlist / /proc/<pid>/environ, the W3 backend would have to read secrets from a mounted file at runtime instead of process.env — a code change in GitLab pinbox24/p24-v-3.2, shipped as push + MR (human merge) and a coordinated redeploy of a 2022-era prod app. Marginal security benefit (the file is still root-readable; exposure is already bounded by the root/docker trust boundary and confirmed non-egressing), high blast radius. Do not auto-apply — surface as a human risk-acceptance decision.

  1. secret-manager (low): reconcile/remove stale SOPS V32_REDIS_PASSWORD (points at decommissioned OVH DBaaS kr40258-001; live uses local redis-v32). Tie to #3712.
  2. human / dev-coder (optional hardening): decide whether the §5 secrets-from-file app change is worth the redeploy risk, or accept the exposure as defense-in-depth given no external egress.
  3. Related known log-leak class (separate): Mongoose logs full Mongo URI on startup → #2397.

Graph View

Backlinks