p24-infra Docs

bms1-pm2-log-credential-exposure-and-rotation-gap

6 min read

bms-1 pm2 log — cleartext MongoDB credential exposure + log-rotation gap (#5439)

Status: remediated (server-side) — 2026-08-04. Rotation of the one live exposed credential delegated to secret-manager (#5450). Owner: sys-admin / infra-task worker on bms-1 (SSH + SOPS read for verify-first); SOPS writes/rotation delegate to secret-manager. Applies to: all /var/log/<container>/pm2/*.log files on bms-1 (94.23.26.113).

This documents both the method (how to verify-first a “cleartext Mongo URI in a pm2 log” report without ever printing a secret) and the 2026-08-04 finding + fix for the never-rotated s3-v2-v42-prod out-log and its many siblings.

1. The two problems (they compound)

  1. Credential-in-log: the Pinbox24 backends (W3/W4 forks) log their full Mongo connection URI — including embedded credentials — to the pm2 out-log at container startup (Mongoose/mongojs startup logging, the #2397 class). Every URI in mongodb://user:pass@host/db form is a cleartext secret.
  2. Rotation gap: many pm2 log dirs on bms-1 were never rotated, so those startup lines accumulated for months. s3-v2-v42-prod’s out-log covered 2026-02-26 → 2026-08-04 (32 MB, 511 840 lines) with the Feb-26 startup URI still sitting at the top.

The rotation gap turned a transient startup line into a 5-month-durable at-rest secret, and the Mezmo agent (/etc/logdna.conf, logdir includes /var/log) shipped those lines off-box.

2. Root cause of the rotation gap

logrotate configs existed but the glob missed the pm2/ subdirectory:

ConfigGlobResult
/etc/logrotate.d/s3-v2-v42-prod (broken)/var/log/s3-v2-v42-prod/*.loglogrotate -d → “does not exist — skipping” (logs live in …/pm2/)
/etc/logrotate.d/pm2-v42-prod (worked)/var/log/v42-prod/pm2/*.logrotated fine
/etc/logrotate.d/v32-prod (worked)/var/log/v32-prod/*.log /var/log/v32-prod/**/*.log** matched the subdir

On top of that, ~15 pm2 log dirs (incl. v32-prod-socket 180 MB, v32-prod-reso 180 MB, mailgun-v42-prod 15 MB) had no config at all — the same gap class the issue asked to sweep for.

3. Verify-first method (never prints a value)

All commands emit key names + sha256[:12] digests + LIVE/DEAD verdicts only — never cat/echo of a URI, never the password. Extraction and digesting run in python3 on bms-1; auth probes pass the value only via an env var into mongosh --nodb --eval (never on argv → never in ps).

  1. Enumerate + digest every mongodb://u:p@… URI across /var/log/*/pm2/*.log* (incl. .gz): print user, authSource, sha256[:12] of the password, and count — no values. (See the scan_all_pm2.py shape in the #5439 transcript.)
  2. Liveness probe each distinct URI against the live cluster — verbatim URI (preserves the URL-encoded password) into mongosh --nodb --quiet --eval reading process.env.PU; classify LIVE / DEAD_AUTH / UNKNOWN:<reason>. Use --nodb or mongosh auto-connects to localhost first. For ssl=true URIs add tlsAllowInvalidCertificates=true before concluding.
  3. Cross-check against SOPS: digest the candidate SOPS key the same way (sops -d … | grep '^KEY=' | cut -d= -f2- | sha256sum | cut -c1-12) and compare. A log value that is LIVE and byte-matches the current SOPS value is an active production-credential exposure.
  4. A value that is DEAD_AUTH was rotated out already — re-rotating it closes no live exposure (same no-op outcome class as w4-s3v2-mailgun-password-verify-first.md); purge from disk only.

--nodb is mandatory: without it mongosh connects to 127.0.0.1:27017 on startup and you get a misleading ECONNREFUSED before your connect(uri) ever runs.

4. Finding — 2026-08-04

28 distinct credential URIs across the pm2 logs. Liveness (verify-first):

CredentialWhereVerdictAction
rs0 admin (root, authSource=admin)s3-v2-v42-prod, s3-v42-prod, s3-v42-stage out-logs (Feb-26 startup)DEAD (superseded by 2026-06-27/07-01/07-09 admin rotations)purge only — no rotation
w3_app (readWrite@w3_db) — 15 distinct valuesold-s3, s3-v32-prod, s3-v32-stage14 DEAD, 1 LIVELIVE one = active exposure → rotate
— the LIVE w3_app/var/log/old-s3/pm2/…out.log (×72)LIVE, and byte-matches current SOPS W3_APP_MONGODB_PASSWORD / mongodb_w3_app_passwordrotate → #5450 (secret-manager)
w3_read_v42v42-prod err .5/.6/.7.gzDEADpurge only
pinbox_production@eat-hn*.artnet.pl, eat1@eat-hn*.nephax.nets3-v32-prod, s3-v42-prod, v42-stageexternal decommissioned vendor clusters (not our rs0)flag only — not rotatable by us

Only the live w3_app credential required rotation. The issue’s premise (an exposed rs0 admin URI) was correct about the log line but the specific admin value was already dead; the genuinely live exposure was the w3_app app-user credential the sweep surfaced in a sibling (old-s3) log.

5. The fix (server-side, bms-1)

5a. Consolidated logrotate config (fixes the gap for every container at once)

Replaced the 3 ad-hoc configs (s3-v2-v42-prod broken, pm2-v42-prod, v32-prod) with one glob covering every pm2 dir. copytruncate is required — pm2 holds the log fd open, so rename-based rotation would strand writes on the old inode.

# /etc/logrotate.d/pm2-containers   (bms-1)
/var/log/*/pm2/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    maxsize 100M
}

Validate before trusting it:

logrotate -d /etc/logrotate.conf 2>&1 | grep -iE 'duplicate|error:'   # expect none
logrotate -d /etc/logrotate.d/pm2-containers 2>&1 | grep considering    # expect all pm2 logs listed

Removing the 3 old configs avoids logrotate “duplicate log entry” errors (a file matched by two stanzas). Old configs backed up to /root/logrotate-backup-5439/ on bms-1.

Not yet Ansible-managed. bms-1 logrotate is hand-maintained on the host (no role applies it). This config is version-controlled here as the source of truth; re-apply it by hand (or fold it into an Ansible role) if bms-1 is rebuilt.

5b. Purge the exposed cleartext

truncate -s 0 <file> each credential-bearing log (safe for running pm2 — same semantics as copytruncate), then logrotate -f /etc/logrotate.d/pm2-containers to baseline. Also delete rotated .gz copies that pre-date the fix and still hold URIs. Confirm with a final recursive scan:

# expect TOTAL_FILES_WITH_CRED_URIS=0
python3 -c "import re,glob,gzip,os;p=re.compile(r'mongodb(?:\+srv)?://[^:\s/]+:[^@\s]+@');\
h=0
for x in glob.glob('/var/log/**/*',recursive=True):
 if os.path.isfile(x):
  try:
   o=gzip.open if x.endswith('.gz') else open
   c=sum(1 for l in o(x,'rt',errors='replace') if p.search(l))
   if c: h+=1;print('REMAINING',x,c)
  except: pass
print('TOTAL_FILES_WITH_CRED_URIS=',h)"

5c. Rotate the live credential (delegated)

On-disk purge does not invalidate a leaked value (it was also shipped to Mezmo). The live w3_app credential must be rotated → dispatched as job_type=secret-manager (#5450), playbook w3-mongodb-credential-rotation.md. Distribution must include the GitLab CI/CD variable for the W3 pipeline, or the next pipeline redeploy reintroduces the old value.

6. Prevention / follow-ups

  1. Durable app-side fix (dev-coder, GitLab pinbox24/*): stop logging the full Mongo URI at startup (redact credentials in the connect log line). Until then every restart re-writes the secret; the rotation config bounds retention, not creation. Related: #2397.
  2. Mezmo blast radius: /var/log is in the Mezmo logdir, so exposed URIs were shipped off-box. Values found live must be rotated regardless of on-disk purge. A Feb-26 line is well beyond typical Mezmo retention, but confirm/purge in Mezmo if a value was live recently.
  3. Ansible: consider a logrotate role for bms-1 so /etc/logrotate.d/pm2-containers is managed as code and survives a rebuild.

Graph View