p24-infra Docs

pinbox24-s3-mongo-uri-log-redaction

5 min read

Playbook: Redact MongoDB URI from s3-v32-prod startup log (bms-1)

Applies to: bms-1 (94.23.26.113) · Pinbox24 w3 S3 microservice container s3-v32-prod Issue: #2397 · Root incident: #2350 Severity: P1 credential exposure (recurs on every container restart) Related: pinbox24-s3-wasabi-bms1.md · static-api-key-incident-rotation.md

⚠️ bms-1 is a P0 EOL server (Ubuntu 20.04, v3.x sunset in progress — see docs/priorities.md / #745). Proceed with caution. Prefer the persistent (volume-mount) redaction so the fix survives container recreation — a bare docker cp is lost on recreate, exactly when the leak would otherwise recur.

1. Problem

s3-v32-prod (Node.js microservice, PM2-managed) logs the full Mongoose connection URI on startup, including the DB password:

info: Mongoose connected to mongoDB server: mongodb://admin:<password>@.../...

The URI is emitted from a Mongoose connection-event handler in the compiled bundle under /app/build/src/api/storage/. Because it prints on every connection, the credential is re-exposed in PM2 logs (docker exec s3-v32-prod pm2 logs, readable by anyone with docker access on bms-1) every time the container restarts — so SOPS/DB-password rotation (#2350) buys nothing: the new password leaks again at the next restart.

The application source is not in this repo. s3-v32-prod runs from an ECR image (563740926945.dkr.ecr.eu-central-1.amazonaws.com/...) built from GitLab pinbox24/p24-v-3.2. The durable upstream fix belongs in that GitLab repo (§4); the procedure below is the in-place mitigation that runs on bms-1 and survives container recreation via the same persistent-patches/ volume-mount already used for local.js and uploadAwsS3.helper.js (see pinbox24-s3-wasabi-bms1.md §8, Open Issues 2, FIXED 2026-06-29).

2. Who can run this

Requires root SSH to bms-1 (root@94.23.26.113, key /root/.ssh/id_ed25519). The autonomous bms-4 infra-task worker runs as claude-runner/claude-runner-2 without that key and cannot perform this operation — it is human-action / root-session only.

W3 working directory on bms-1: /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/

Secret-safety (ABSOLUTE): never print the URI value. Do not run docker logs s3-v32-prod or pm2 logs unfiltered — both echo the password. Verify only with the credential-stripping grep in §3 Step 6. If a secret value appears in output, stop and follow static-api-key-incident-rotation.md.

3. In-place persistent remediation (bms-1)

Step 1 — Locate the log statement (read-only, safe)

The compiled source references a URI variable, not the literal password, so reading the source is safe:

# find the file + line that logs the connection URI
docker exec s3-v32-prod sh -lc \
  "grep -rIn -e 'Mongoose connected' -e 'mongoDB server' /app/build/src/api/storage/"

Note the file path (e.g. /app/build/src/api/storage/<file>.js) and the exact log line. It is typically a Mongoose event handler of the form:

mongoose.connection.on('connected', () => {
  logger.info(`Mongoose connected to mongoDB server: ${uri}`);   // <-- leaks credentials
});

Step 2 — Copy the file out

mkdir -p /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/persistent-patches
docker cp s3-v32-prod:/app/build/src/api/storage/<file>.js \
  /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/persistent-patches/<file>.js   # PLAYBOOK: pinbox24-s3-mongo-uri-log-redaction.md

Step 3 — Redact the URI in the copied file

Edit the copied file so the handler logs only the host/database portion, never the user:pass@ credentials. Two equivalent approaches — pick whichever matches the code:

a) Log a static, credential-free message (simplest):

mongoose.connection.on('connected', () => {
  logger.info('Mongoose connected to mongoDB server (credentials redacted)');
});

b) Strip credentials from the URI before logging (keeps host/db for diagnostics):

mongoose.connection.on('connected', () => {
  const safeUri = String(uri).replace(/\/\/[^@/]*@/, '//<redacted>@');
  logger.info(`Mongoose connected to mongoDB server: ${safeUri}`);
});

Edit with vi on the server (do not paste the URI value anywhere). Change only the log line — do not alter connection logic.

Step 4 — Mount the patched file as a persistent volume

Add a volume mount so the redaction survives container recreation, mirroring the existing persistent-patches/ entries in the w3 compose file (local.js, uploadAwsS3.helper.js):

cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/
# In docker-compose.yml, under the s3 service `volumes:` block, add (adjust <file>.js):
#   - ./persistent-patches/<file>.js:/app/build/src/api/storage/<file>.js:ro

Keep the existing local.js / uploadAwsS3.helper.js mounts intact.

Step 5 — Apply the fix

Two options — Step 5a stops the leak immediately without a recreate; Step 5b makes the volume mount active. Do both so it is fixed now and durable.

# 5a — apply to the running process now (stops the current leak; non-persistent by itself)
docker cp /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/persistent-patches/<file>.js \
  s3-v32-prod:/app/build/src/api/storage/<file>.js   # PLAYBOOK: pinbox24-s3-mongo-uri-log-redaction.md
docker exec s3-v32-prod pm2 reload all
 
# 5b — recreate so the :ro volume mount is bound (env vars must be set first, per rotation playbook)
cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
export CONTAINER_NAME=s3-v32-prod
export IMAGE_NAME=$(docker inspect s3-v32-prod --format "{{.Config.Image}}")
docker-compose up -d --force-recreate s3   # PLAYBOOK: pinbox24-s3-mongo-uri-log-redaction.md

After any recreate, re-verify local.js is still mounted (uploads depend on it) — see pinbox24-s3-wasabi-bms1.md §6 diagnosis.

Step 6 — Verify (credential-safe)

Confirm the new connection log line is redacted without printing the password. This grep keeps only lines that would be a leak; a clean fix prints nothing:

# Expect: NO output (any output = still leaking → fix not applied)
docker exec s3-v32-prod pm2 logs --nostream --lines 200 2>&1 \
  | grep -E 'mongodb://[^@]*:[^@]*@' || echo "OK — no credential URI in recent logs"
 
# Confirm the service is up
docker exec s3-v32-prod pm2 list

4. Durable upstream fix (GitLab — out of repo scope)

The ECR image is rebuilt from GitLab pinbox24/p24-v-3.2. To make the redaction permanent across image rebuilds, apply the Step 3 change to the connection handler in that repo’s source (src/api/storage/…) and rebuild/redeploy the image. Until then, the persistent-patches/ volume mount (§3 Step 4) is the load-bearing mitigation and must be preserved on any w3 redeploy. Track under the bms-1 v3.x sunset plan (#745).

5. Rollback

# Remove the volume mount line from docker-compose.yml, then recreate:
cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
export CONTAINER_NAME=s3-v32-prod
export IMAGE_NAME=$(docker inspect s3-v32-prod --format "{{.Config.Image}}")
docker-compose up -d --force-recreate s3   # PLAYBOOK: pinbox24-s3-mongo-uri-log-redaction.md

The image’s original (leaking) file returns — only do this if the redaction breaks startup.

Graph View

Backlinks