Pinbox24 W3/W4 — Architecture Specification

Audited: 2026-07-06 (bms-1 SSH audit) W4 persistent-patch state reconfirmed: 2026-08-09 (#5992 — live mount ground-truth on bms-1; the §3.1 / §3.x patch tables below reflect this, not the 2026-07-06 snapshot) Source: /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ (W4 — confirmed p24-back-ts), /root/builds/pn3C9eHo/0/pinbox24/p24-v-3.2/ (W3 — path as read at 2026-07-06 audit; superseded, see build-dir note below) Server: bms-1 (94.23.26.113) — Ubuntu 20.04 EOL, OVH Kimsufi

⚠️ W3 build-dir correction (#4985). The authoritative, GitLab-runner-owned W3 compose dir is /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/ (per secrets-sync.yml and docs/bms-1/gitlab-ci-w3.ymlBUILDER_DIR). The root-owned /root/builds/pn3C9eHo/... recorded above is not the prod serving directory — but an on-server audit (2026-08-01) proved it is not abandoned either: it is an active W3-staging mount source and must not be deleted wholesale (see docs/w3-w4-stack-operations.md §3). Confirm the live dir before use: docker inspect v32-prod --format '{{index .Config.Labels "com.docker.compose.project.working_dir"}}'.


1. Overview

W3 (v3.2)W4 (v4.2)
CodebaseNode.js legacy + JavaScriptTypeScript 4.x
Backend version~3.2 (legacy)4.2.17
Backend containerv32-prodv42-prod (image: v42-prod:merged-20260706-0512)
File microservices3-v32-prod (ECR: old-s3)s3-v42-prod (ECR: v4-s3)
Build dir/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/ (authoritative for prod; /root/builds/pn3C9eHo/... is not the prod serving copy but is an active W3-staging mount source — do NOT delete — #4985)/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/
SOPS filesecrets/pinbox24-w3.env.sopssecrets/pinbox24-w4.env.sops
SOPS deployed to/opt/p24-infra/bms-1/pinbox24-w3.env/opt/p24-infra/bms-1/pinbox24-w4.env
Networktest-net (external bridge)test-net (external bridge)
Restart policyunless-stopped (via docker update)unknown — needs verification
Persistent patches4 filesv42-prod: 5 live mounts (uploadAwsS3.helper.js, socket.middleware.js, app.js, postbookReport.helper.js, ecosystem.config.js) + credentials.json; GUS WSDL mount removed. mailgun-v42-prod: integration.helper.js (git-tracked #5992). See §3.1 / §3.x (reconfirmed 2026-08-09)
Microservice count29

2. W3 — Microservices Detail

2.1 v32-prod (main backend)

PropertyValue
ECR image563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod
RuntimeNode.js, PM2 cluster (2 instances)
FrameworkExpress.js (JavaScript)
Port3000 (internal, behind nginx-proxy)
Log dir/var/log/v32-prod/pm2/

External connections:

ServiceProtocolEnv keyNotes
MongoDB rs0mongoosePMONGODB_URLw3_app user, authSource=w3_db; Path B/api/i18n/langs
MongoDB rs0mongojsMONGODB_URLcrudService CRUD (~70 controllers); Path A — falls back to PINBOX_PROD_DB in the production block
Wasabi S3HTTP/AWS SDKs3Bucket_*file storage
s3-v32-prodHTTPs3ApiUrlproxied file uploads

Connection-layer deep dive: the two Mongo paths above are independent, cached once at startup (rotation needs a container restart), and fail silently/independently. Root cause of the “langs not loading” incident and the V32_PINBOX_PROD_DB rotation gap are documented in W3 connection handling & rotation (#3564).

Persistent state:

  • PM2 cluster — persistent processes required
  • No WebSocket (not found in W3 source)
  • No message queue (W3 only)

Persistent patches (volume mounts):

FilePatches
filesUpload.jsFix B1: regId/recId fallback from query params
crudService.jsFix B2: async.forEach callback on files without bucket
excel_import.jsNot a patch — stock git-tracked file in p24-v-3.2, mounted from the build-dir source tree (not persistent-patches/); byte-identical to the image copy, so the mount is a redundant no-op. Restored by the GitLab checkout, needs no p24-infra source of truth (#4708).

2.2 s3-v32-prod (file microservice)

PropertyValue
ECR image563740926945.dkr.ecr.eu-central-1.amazonaws.com/old-s3
RuntimeNode.js/TypeScript, PM2 (old-s3 project)
Port3000 (internal only, not exposed to nginx-proxy)
Log dir/var/log/s3-v32-prod/pm2/

External connections:

ServiceProtocolEnv keyNotes
MongoDB rs0mongooseDB_URIw3_app user, saves file metadata
Wasabi S3AWS SDKs3Bucket_*stores actual files

Persistent patches:

FilePatches
local.jsOverrides: MongoDB URL from DB_URI, Wasabi from s3Bucket_* env
controller.jsUpload handler — saves file metadata after Wasabi upload

3. W4 — Microservices Detail

Full container inventory (bms-1, as of 2026-07-06)

ContainerImagePurpose
v42-prod563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod:merged-20260706-0512Main backend (TypeScript)
s3-v42-prodECR: v4-s3File upload microservice
redis-v42redis:7-alpineLocal Redis — RabbitMQ dedup + Google Sheet sync
wkhtml-v42-prodECR: wkhtmltopdf imagePDF generation server (IS running)
mailgun-prodlocal/ECREmail relay proxy
s3-v2-v42-prodECR: s3-v2Secondary S3 microservice
git-deploy-v42-prodlocalGit-based deploy trigger
pdf-gen-v42-prodlocalPDF generation helper
v42-notify-prodECRNotification microservice

Build dir: /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/

3.1 v42-prod (main backend)

PropertyValue
ECR image563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod
RuntimeNode.js 18+, TypeScript compiled, PM2 cluster (2 instances)
FrameworkExpress.js
Port3000 (internal, behind nginx-proxy)
Log dir/var/log/v42-prod/pm2/

External connections:

ServiceProtocolEnv keyNotes
MongoDB rs0 (primary)mongoose keepAliveNEW_MONGODB_URImain app data, w4_app user
MongoDB rs0 (v3 compat)mongojsv3MongoUrlbackward-compat reads from W3 DB
RabbitMQAMQPRABBIT_MQ_URLpersistent consumer started at boot (+2s delay)
Redis (local)TCPREDIS_HOST, REDIS_PORT, REDIS_PASSWORDredis-v42 container on bms-1 (NOT OVH Redis — replaced due to instability)
Wasabi S3AWS SDKs3Bucket_*, pinbox24Public*files + public assets
s3-v42-prodHTTPs3ApiUrl, s3ApiUrlV2proxied file uploads
Google SheetsHTTPS OAuth2GOOGLE_APPLICATION_CREDENTIALSfile path to JSON credentials
Mailgun EUHTTPSMAILGUN_API_KEY, MAILGUN_EU_*transactional email
Jabber/XMPPXMPPjabber_HOST, jabber_JID, jabber_PASSWORDnotifications
PayUHTTPSPAYU_CLIENT_ID, PAYU_CLIENT_SECRET, PAYU_MD5_SECOND_KEYpayments
Przelewy24HTTPSprzelewy24Auth, przelewy24MerchantIdpayments
TwilioHTTPStwilioAccountSid, twilioAuthTokenSMS
OneSignalHTTPSonesignal_APP_ID, onesignal_APP_AUTH_KEYpush notifications
GUS REGONSOAP/WSDL(WSDL resolves from node_modules/@pobidowski/gus-api-regon/dist/wsdl/ — the old persistent-patches/gus-api-regon-wsdl/ mount was removed, it never took effect; see §3.1 patch table + v42-prod-gus-wsdl-crash-loop.md)Polish tax registry
ConvertAPIHTTPSCONVERT_APIdocument conversion
wkhtml-v42-prodHTTPWK_HTML_TO_PDF_SERVER_URLPDF generation (running — wkhtml-v42-prod container)
PM2 PlusHTTPSPM2_PUBLIC_KEY, PM2_SECRET_KEYmonitoring

Persistent state (critical — NOT serverless-compatible):

ComponentDetails
Socket.IOinitSocket(SocketIO(mainServer, options)) — WebSocket + polling at startup
RabbitMQ consumerstartRabbitMqConsumer() — persistent AMQP connection, heartbeat 60s
Redis connectionredis-v42 (local bms-1 container) — RabbitMQ dedup; was OVH Redis, replaced for stability
Dynamic cron jobscheckAndStartCronJobs() — reads from OfficeCronModel (MongoDB), creates CronJob instances in memory
Static CronJobnew CronJob('0 0 6 * * *', ...) — hardcoded daily cron
PM2 cluster2 instances, pm_id used for single-instance cron guard

Persistent patches (volume mounts) — live state reconfirmed on bms-1 2026-08-09 (#5992):

Git source: infra-src/pinbox24/w4/persistent-patches/ → shipped by secrets-sync.yml (sync-pinbox24-w4 → “Deploy W4 persistent-patches to bms-1”) to /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/persistent-patches/.

Mount source (bms-1)Container pathPurposeStill needed?
persistent-patches/uploadAwsS3.helper.js/app/dist/globalHelpers/uploadAwsS3.helper.jsDe-hardcodes Wasabi S3 creds to process.env.s3Bucket_* and adds the multi-region _bucketEndpointMap for s3getSignedUrlYES. GitLab development src carries the de-hardcode fix (commit 63686ca5), but (a) the deployed ECR image’s baked dist/ still ships the old hardcoded creds and lacks the fix (verified #5992 — image built 2026-08-06 does not reflect merged source), and (b) upstream src has no _bucketEndpointMap (US-region signed URLs would break). Removing the mount would revert the fix and re-expose hardcoded credentials in running code.
persistent-patches/socket.middleware.js/app/dist/middleware/socket.middleware.jsWraps Socket.IO onAuth’s verifyAuthToken in try/catch (#3966) so an expired/invalid JWT rejects cleanly via next(err) instead of throwing uncaughtYES. GitLab development src has the equivalent fix (commit 1fba09e5), but the deployed image’s baked dist/ still lacks it (verified #5992). Removing the mount reverts to the uncaught-throw / reconnect-storm behaviour.
persistent-patches/app.js/app/dist/app.jsRemoves dead GUS-API test/debug code that ran on every startup (#3407 follow-up, v42-prod-gus-wsdl-crash-loop.md)YES
persistent-patches/postbookReport.helper.js/app/dist/apps/postbook/postbookReport.helper.jsPostbook report fixYES
ecosystem.config.js/app/ecosystem.config.jsPM2 cluster configYES
credentials.json/app/credentials.jsonGoogle Service Account creds — SOPS-managed (V42_GOOGLE_CREDENTIALS_JSON_B64, #3737), bind-mounted over the baked-in copyYES
gus-api-regon-wsdl//app/dist/wsdlREMOVED — never took effect (real require() resolves from node_modules/@pobidowski/gus-api-regon/dist/wsdl/); dropped alongside the app.js GUS-code disable— (removed)

Why the patches can’t be retired yet (#5992). The de-hardcode / try-catch fixes ARE merged into p24-back-ts development, but the deployed v42-prod:latest ECR image’s compiled dist/ does not reflect them — its baked dist/globalHelpers/uploadAwsS3.helper.js still contains hardcoded Wasabi credentials and its socket.middleware.js still lacks the try/catch. Retiring either mount is blocked until an image genuinely built from current development is deployed and its baked dist/ verified fix-complete. Tracked as a follow-up (image-build / at-rest-credential remediation).

3.x mailgun-v42-prod (email relay proxy)

PropertyValue
Containermailgun-v42-prod (compose dir /root/mailgun-prod/, image mailgun-v42-prod:latest, p24-ms-mailgun repo)
Endpointmailgun-api.w4.pinbox24.com

Persistent patch (volume mount) — git-tracked as of #5992 (2026-08-09):

Mount source (bms-1)Container pathPurposeGit source / deploy
/root/mailgun-prod/patches/integration.helper.js/app/helper/integration.helper.js”PATCH v3” — uploads mail attachments via s3-v2 first and unwraps the { success, result } envelope before creating the register recordinfra-src/pinbox24/w4/mailgun-patches/integration.helper.js, shipped by secrets-sync.yml (sync-pinbox24-backends → “Deploy mailgun-v42-prod persistent patch to bms-1”). Before #5992 this file lived only on the server with no git source and no deploy step.

3.2 s3-v42-prod (file microservice)

PropertyValue
ECR image563740926945.dkr.ecr.eu-central-1.amazonaws.com/v4-s3
RuntimeNode.js/TypeScript, PM2
Port3000 (internal only)

External connections:

ServiceProtocolEnv keyNotes
MongoDB rs0mongooseDB_URI (from s3-environment.env)saves file metadata
Wasabi S3AWS SDKs3Bucket_*stores actual files

3.3 redis-v42 (local Redis)

PropertyValue
Imageredis:7-alpine
ScopeW4-only — not shared with W3 or any other stack
PurposeRabbitMQ message deduplication + Google Sheet sync state
Why localReplaced OVH/bms-4 Redis (was unstable); bms-1-local eliminates network dependency
Restart policyverify — should be unless-stopped

Note: If bms-1 is restarted and redis-v42 starts after v42-prod, the RabbitMQ consumer will fail to connect to Redis. Both must start before processing begins. Ensure restart policy is set on both.


4. Shared Infrastructure

Both W3 and W4 share:

ResourceDetail
Networktest-net external Docker bridge (nginx-proxy controlled)
MongoDB rs0PRIMARY: bms-2 (145.239.133.104), secondary: bms-3 (51.68.155.224)
Wasabi S3pinbox24 bucket, eu-central-1 region
bms-1 servernginx-proxy (HTTP entry point), 24+ total containers

W4-exclusive infrastructure on bms-1:

ResourceDetail
redis-v42redis:7-alpine — local Redis for W4 only; replaces unstable OVH Redis
wkhtml-v42-prodwkhtmltopdf container for PDF generation
mailgun-prodMailgun relay proxy
v42-notify-prodPush notification microservice

5. Legacy Environments on bms-1

Beyond W3/W4 production, bms-1 runs additional environments from an old private registry (private-registry.dev.pinbox24.com):

ContainerAgePurpose
v32-prod-socket5 yearsW3 socket/realtime testing env
s3-v32-prod-socket5 yearsFile service for socket env
cron-v32-prod-socket5 yearsCron for socket env
v32-prod-reso5 yearsW3 RESO protocol env
s3-v32-prod-reso5 yearsFile service for reso env
cron-v32-prod-reso5 yearsCron for reso env
s3-v32-prod-renamed4 yearsUnknown
cron-v32-prod4 yearsAdditional cron instance
v32-prod-socket (v5)5 yearsSecond socket env

Status: Not audited for credential exposure. All likely have hardcoded credentials from the old private registry era (pre-SOPS, pre-ECR). See pinbox24-w3-w4-security-policy.md.


6. Vercel Feasibility Assessment

W3 — NOT feasible

BlockerSeverity
PM2 cluster modeCritical
Persistent patches (4 files, hide code bugs)Critical
No serverless MongoDB connectionHigh
File uploads (size + timeout limits)High

Verdict: Would require full rewrite of upload flow + fixing all patched bugs in source. Estimated 4-8 weeks.

W4 — NOT feasible

BlockerSeverity
Socket.IO (WebSocket + polling)Critical — Vercel Serverless has no persistent connections
RabbitMQ consumer (persistent AMQP)Critical — can’t run a persistent consumer
Redis — local container (redis-v42)Critical — replaced OVH Redis for stability; Vercel migration would require external Redis provider (Upstash/Redis Cloud) + connection changes
Dynamic cron jobs from MongoDBCritical — in-memory CronJob instances require persistent process
PM2 cluster (pm_id guards)Critical
7 additional W4 microservices on bms-1Critical — wkhtml, mailgun-prod, s3-v2, git-deploy, pdf-gen, v42-notify all need separate hosting
GOOGLE_APPLICATION_CREDENTIALS as file pathHigh — needs refactor to env var
Persistent patches (WSDL + Wasabi fix)High
Jabber/XMPP persistent connectionHigh

Verdict: W4 has MORE blockers than W3. Socket.IO + RabbitMQ + Redis (now intentionally local for stability) + dynamic crons + 7 co-located microservices are fundamental architecture choices. Moving Redis from local to an external provider would be a regression in stability. Estimated 8-12 weeks of architectural refactoring to make serverless-compatible.

Summary

W3 → bms-1 (Docker) — Vercel migration: not feasible (4-8 weeks rewrite)
W4 → bms-1 (Docker) — Vercel migration: not feasible (8-12 weeks rewrite)

Both stay on bms-1. CI/CD via extended secrets-sync.yml.

7. Security Status

ItemW3W4
MongoDB app user (not admin)✅ w3_app (2026-07-06)needs audit
Persistent patches in repo❌ only on disk❌ only on disk
docker-deploy-prod.sh with hardcoded creds❌ issue #3063.bak exists too
PM2 logs MongoDB URI❌ issue #3066❌ likely same pattern
Restart policy in docker-compose.yml❌ needs PR❌ needs PR
CI/CD auto-deploy on credential rotation❌ in progress❌ in progress

8. Env Keys by Service

W3 — secrets/pinbox24-w3.env.sops

SOPS keyContainerEnv var
V32_MONGODB_URLv32-prodMONGODB_URL
V32_PMONGODB_URLv32-prodPMONGODB_URL
V32_PINBOX_PROD_DBv32-prodPINBOX_PROD_DBenv.config.js production-block fallback for the mongojs path (/api/i18n/langs); shares the w3_app@w3_db credential and must stay in sync with V32_MONGODB_URL. See W3 connection handling & rotation.
V32_DB_URIs3-v32-prodDB_URI
V32_s3ApiUrlv32-prods3ApiUrl
V32_s3Bucket_endpointboths3Bucket_endpoint
V32_s3Bucket_BUCKET_NAMEboths3Bucket_BUCKET_NAME
V32_s3Bucket_api_accessKeyIdboths3Bucket_api_accessKeyId
V32_s3Bucket_api_secretAccessKeyboths3Bucket_api_secretAccessKey

W4 — secrets/pinbox24-w4.env.sops

Keys that should be in SOPS (currently in backend-environment.env directly on bms-1):

Env varContainerSensitivity
NEW_MONGODB_URIv42-prodCritical
v3MongoUrlv42-prodCritical
RABBIT_MQ_URLv42-prodCritical
REDIS_HOSTv42-prodMedium (points to redis-v42 container on test-net)
REDIS_PORTv42-prodLow
REDIS_PASSWORDv42-prodHigh
JWT_TOKEN_SECRETv42-prodCritical
MAILGUN_API_KEYv42-prodHigh
PAYU_CLIENT_SECRET, PAYU_MD5_SECOND_KEYv42-prodCritical
przelewy24Authv42-prodCritical
twilioAuthTokenv42-prodHigh
pinbox24PublicSecretAccessKey, pinbox24PublicOfficeSecretAccessKeyv42-prodHigh
s3Bucket_api_secretAccessKeys3-v42-prodHigh
tokenGenSecret, tokenGenIVv42-prodCritical
PM2_SECRET_KEYv42-prodMedium

9. Open Issues

#IssuePriority
#3060W3: MongoDB admin creds in PM2 log → rotation complete 2026-07-06Done
#3063W3/W4: docker-deploy-prod.sh hardcoded credsHigh
#3066W3/W4: PM2 logs full MongoDB URI — needs source fix in v32-prod and v42-prodHigh
#3564W3: sops-reset-mongodb.ps1 W3 group omits V32_PINBOX_PROD_DB → each rotation re-breaks /api/i18n/langs; add it to $uriKeyDefs (secret-manager scope). See connection handling & rotationHigh
W4: CI/CD (secrets-sync → force-recreate) not implementedHigh
W4: All credentials in env file, not in SOPSHigh
W4: No restart: unless-stopped in docker-compose.ymlMedium
W4: redis-v42 restart order — must start before v42-prod on server rebootMedium
Legacy bms-1 containers (socket, reso, cron) — credential auditMedium
W4: v42-prod actual container name differs from build dir name p24-back-ts — document in deploy scriptsLow