p24-infra Docs

pinbox24-w3-operations

5 min read

Playbook: Pinbox24 W3 Operations

Last updated: 2026-07-06 (session 53) Applies to: bms-1 (94.23.26.113) — Pinbox24 W3 production (w3.pinbox24.com) Services: v32-prod (Node.js backend), s3-v32-prod (file microservice)

Architecture

W3 is the v3.2 Pinbox24 workspace. It runs on bms-1 as two Docker containers managed by docker-compose.

Browser → nginx-proxy (test-net) → v32-prod (port 3000)
                                         ↓ proxies uploads
                               s3-v32-prod (port 3000, internal)
                                         ↓
                               Wasabi S3 (eu-central-1)
                                         ↓
                               MongoDB rs0 (bms-2 PRIMARY + bms-3)
                                         ↑
                               v32-prod also connects directly

File locations on bms-1

PathPurpose
/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/Build directory (docker-compose root)
docker-compose.ymlService definitions
backend-environment.envv32-prod env vars — managed by SOPS, do NOT overwrite manually
s3-environment.envs3-v32-prod env vars — managed by SOPS, do NOT overwrite manually
persistent-patches/Host-mounted JS patches that survive container recreate
/opt/p24-infra/bms-1/pinbox24-w3.envDecrypted SOPS file deployed by CI (reference only)

Services

v32-prod (backend)

ItemValue
Containerv32-prod
Image563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod
Port3000 (internal)
Networktest-net (external bridge)
Env filebackend-environment.env — source of truth: secrets/pinbox24-w3.env.sops keys: V32_MONGODB_URL, V32_PMONGODB_URL
Restart policyno — BUG: containers do not start after server reboot (see Fix #1)

V8 heap baseline (issue #4035, measured 2026-07-12): v32-prod runs babel-node (interpreter ./node_modules/.bin/babel-node, Node 16.20.2) which keeps the whole Babel transpile pipeline + transformed sources resident in old-space, so its steady-state heapUsed/heapTotal ratio is ~0.92 (observed 0.918–0.921; Used Heap ~114 of a 124 MiB heapTotal, flat; container RSS ~200–215 MiB; 0 PM2 restarts over 13h). This is HEALTHY, not a leak — a real leak shows Used Heap / heapTotal climbing over time ending in a PM2/OOM restart (restart counter rising). Because of this high-but-flat baseline, the shared Pinbox24HeapUsageHigh alert (>0.85) was recalibrated to >0.93 for v32-prod only (monitoring/prometheus/rules/pinbox24.yml); Pinbox24HeapUsageCritical (>0.95) still backstops a genuine runaway. On a v32 heap alert, first confirm the trend is growing (docker exec v32-prod pm2 show 0 | grep -iE "Heap|restart|uptime" twice, minutes apart) before treating it as a leak — a flat ~0.92 with 0 restarts is normal.

Persistent volume mounts (survive force-recreate):

Host pathContainer pathPurpose
persistent-patches/filesUpload.js/app/app-backend/controllers/filesUpload.jsFix B1: regId/recId fallback from query params (2026-07-06)
persistent-patches/crudService.js/app/app-backend/helperFunction/crudService.jsFix B2: callback() on files without bucket in async.forEach (2026-07-06)
app-backend/helperFunction/processHelper/tasksService/tasks/excel_import.js/app/app-backend/helperFunction/processHelper/tasksService/tasks/excel_import.jsStock git-tracked file in p24-v-3.2, NOT a p24-infra patch. Mounted from the build-dir source tree, byte-identical to the image copy (mount = redundant no-op). Restored by the GitLab checkout; no p24-infra source of truth needed (#4708, verified 2026-07-30).

s3-v32-prod (file microservice)

ItemValue
Containers3-v32-prod
Image563740926945.dkr.ecr.eu-central-1.amazonaws.com/old-s3
Port3000 (internal, not exposed to internet)
Networktest-net (internal only, behind v32-prod proxy)
Env files3-environment.env — DB_URI from secrets/pinbox24-w3.env.sops key V32_DB_URI (add pending)
Restart policyno — BUG: containers do not start after server reboot (see Fix #1)

Persistent volume mounts:

Host pathContainer pathPurpose
persistent-patches/local.js/app/config/local.jsOverrides: MongoDB URL from DB_URI, Wasabi S3 from s3Bucket_* env vars
persistent-patches/controller.js/app/build/src/api/storage/controller.jsUpload handler — saves file metadata to MongoDB after Wasabi upload

Network

test-net is an external bridge network managed by nginx-proxy (jwilder/nginx-proxy container). It is NOT created by this docker-compose.yml — it must exist before docker-compose up.

# Verify:
docker network ls | grep test-net
# Must show: test-net bridge local

Secrets / Credentials

All secrets live in secrets/pinbox24-w3.env.sops (SOPS+age).

SOPS keyEnv var in containerServiceNotes
V32_MONGODB_URLMONGODB_URLv32-prodmongojs connection (crudService)
V32_PMONGODB_URLPMONGODB_URLv32-prodmongoose connection (i18n, models)
V32_DB_URIDB_URIs3-v32-prodmongoose for file metadata — to be added (#3060)
V32_s3Bucket_endpoints3Bucket_endpoints3-v32-prodWasabi endpoint
V32_s3Bucket_BUCKET_NAMEs3Bucket_BUCKET_NAMEs3-v32-prodBucket name
V32_s3Bucket_api_accessKeyIds3Bucket_api_accessKeyIdbothWasabi IAM access key
V32_s3Bucket_api_secretAccessKeys3Bucket_api_secretAccessKeybothWasabi IAM secret

Secrets distribution:

  • secrets-sync.yml deploys SOPS to /opt/p24-infra/bms-1/pinbox24-w3.env — does NOT update env files in build dir
  • env files in build dir must be updated manually or via a dedicated deploy script that reads from SOPS

Redeployment survival analysis

ScenarioSurvives?Notes
docker-compose restart✅ partialContainers restart but do NOT reload env_file
docker-compose up -d --force-recreate✅ YESReloads env_file, mounts stay; safest operation
docker update --restart unless-stopped✅ partialSets restart policy but not in compose
Server reboot❌ NONo restart policy — containers dead after reboot (Fix #1 needed)
docker-deploy-prod.sh run❌ BREAKSOverwrites env files with hardcoded stale creds — NEVER run (issue #3063)
secrets-sync.yml run✅ safeDeploys to /opt/p24-infra/bms-1/pinbox24-w3.env, does NOT touch build dir
Fresh ECR pull⚠️ riskyECR token expires — use local image cache or refresh token first

Known issues (open)

#IssueRisk
#3060MongoDB admin password exposed in PM2 log; V32_PMONGODB_URL uses admin instead of w3_app — caused INCIDENT-2026-07-08-002 (login broken) when admin password was rotated. Fix STAGED (dedicated w3_app user created, password in SOPS as W3_APP_MONGODB_PASSWORD) — deploy to bms-1 scheduled after 17:00 2026-07-08. See docs/pinbox24/incident-2026-07-08-w3-login-broken-admin-password-desync.md.High — rotation staged, deploy pending
#3063docker-deploy-prod.sh has hardcoded credentials — verify if activeMedium — verify then rotate
—No restart: unless-stopped in docker-compose.ymlHigh — containers die on server reboot

Fix #1 — Add restart policy (IMMEDIATE)

Until docker-compose.yml is updated, apply via docker update:

# SSH to bms-1
docker update --restart unless-stopped v32-prod
docker update --restart unless-stopped s3-v32-prod
 
# Verify
docker inspect v32-prod --format "{{.HostConfig.RestartPolicy.Name}}"
docker inspect s3-v32-prod --format "{{.HostConfig.RestartPolicy.Name}}"
# Expected: unless-stopped

Permanent fix: add restart: unless-stopped to both services in docker-compose.yml.

Correct procedure: update credentials after SOPS rotation

  1. Decrypt from SOPS on Windows dev: 
    $env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Extract V32_MONGODB_URL and V32_PMONGODB_URL silently
  2. URL-encode passwords (MongoDB URIs contain special chars): 
    from urllib.parse import quote
encoded = quote(password, safe='')
  3. Update backend-environment.env on bms-1 (Python stdin pipe via SSH)
  4. Update s3-environment.env on bms-1 (DB_URI line only)
  5. Force-recreate backend: 
    cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
docker-compose up -d --force-recreate backend
  6. Force-recreate s3: 
    docker-compose up -d --force-recreate s3
  7. Verify:
    • v32-prod: /var/log/v32-prod/pm2/pm2_v32-prod_production_out.log — no authentication fail
    • s3-v32-prod: /var/log/s3-v32-prod/pm2/pm2_s3-v32-prod_production_out.log — Mongoose default connection open

Reference: docs/playbooks/w3-mongodb-credential-rotation.md

Correct procedure: force-recreate after image update

  1. Authenticate with ECR (if pulling new image): 
    aws ecr get-login-password --region eu-central-1 | \
  docker login --username AWS --password-stdin 563740926945.dkr.ecr.eu-central-1.amazonaws.com
docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod
docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/old-s3
  2. Force-recreate (env files already correct): 
    cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
docker-compose up -d --force-recreate
  3. Verify network: docker network inspect test-net
  4. Verify persistent-patches mounted: docker exec v32-prod ls /app/app-backend/helperFunction/crudService.js

DO NOT

  • Run docker-deploy-prod.sh — it overwrites env files with hardcoded stale credentials (#3063)
  • Use docker-compose restart and expect env_file changes to take effect
  • Mount patches into s3-v32-prod as v32-prod — filesUpload.js belongs in v32-prod only
  • Set DB_URI in s3-environment.env to admin credentials — use w3_app (#3060)
  • Commit env files (backend-environment.env, s3-environment.env) to git — they are plaintext secrets

Health checks

# API response
curl -I https://w3.pinbox24.com/api/health 2>/dev/null | head -1
 
# v32-prod PM2
docker exec v32-prod pm2 list
 
# s3-v32-prod PM2
docker exec s3-v32-prod pm2 list
 
# Mongoose connection (no auth errors)
tail -20 /var/log/s3-v32-prod/pm2/pm2_s3-v32-prod_production_out.log | grep -i "mongoose\|error"
 
# Both containers running
docker ps --filter name=v32-prod --filter name=s3-v32-prod --format "{{.Names}} {{.Status}}"

Graph View

Backlinks