Pinbox24 S3/Wasabi Configuration Ă˘â‚¬â€ť bms-1 (w3 and w4)

Applies to: bms-1 (94.23.26.113) Ă‚Â· Pinbox24 production (w3 = v3.2, w4 = v4.2) Last updated: 2026-06-29 Incident ref: 2026-06-29 Ă˘â‚¬â€ť w3 upload broken (DO Spaces hardcoded endpoint)

1. Overview

Pinbox24 uses a single Wasabi bucket (pinbox24, region eu-central-1) for all file storage. 159,000+ objects going back to 2021 Ă˘â‚¬â€ť treat as production-critical data.

DigitalOcean Spaces was legacy config only Ă˘â‚¬â€ť files never actually went there. DO Spaces credentials (CY5VG3XCNNB4N3BZSLOE prefix) appeared in hardcoded configs but are invalid; investigate whether that key is still active (see Open Issues below).

2. Wasabi Bucket

Parameter	Value
Bucket	`pinbox24`
Region	`eu-central-1`
Endpoint	`s3.eu-central-1.wasabisys.com`
IAM user	`p24-access`
Key limit	2 active keys max (Wasabi IAM constraint)

3. Architecture

3.1 w3 (v3.2)

Component	Detail
Backend container	`v32-prod`
Backend image	`563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod`
Working dir	`/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/`
Backend env file	`backend-environment.env`
S3 microservice container	`s3-v32-prod`
S3 microservice image	ECR (same account)
S3 microservice env file	`s3-environment.env`

The S3 microservice (s3-v32-prod) is a node-config application. It reads /app/config/default.js (hardcoded Ă˘â‚¬â€ť see Warning below) and /app/config/local.js (override via env vars). Without local.js, all uploads go to the hardcoded DigitalOcean Spaces endpoint Ă˘â‚¬â€ť which is wrong.

WARNING Ă˘â‚¬â€ť local.js is NOT persistent. It is written directly into the running container via docker cp. It is lost on container recreation. A volume mount or image rebuild is required to make it permanent (see Open Issues).

3.2 w4 (v4.2)

Component	Detail
Backend container	`v42-prod`
Backend image	`563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod`
Working dir	`/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/`
Backend env file	`backend-environment.env`
S3 microservice (shared)	`s3-v32-prod` (same container as w3)
S3-v2 microservice	`s3-v2-v42-prod` (TypeScript, v4-specific features)
S3-v2 working dir	`/root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2/`
S3-v2 env file	`s3-v2-environment.env`

WARNING Ă˘â‚¬â€ť v42-prod has HARDCODED Wasabi credentials in compiled TypeScript: /app/dist/globalHelpers/uploadAwsS3.helper.js

This file also contains an AWS key (AKIAYGQMT4PQ3ZLOWHRR prefix) Ă˘â‚¬â€ť purpose unknown, needs investigation. The hardcoded Wasabi section causes getSignedUrl to return 403 Forbidden after any key rotation until manually patched. This patch is also NOT persistent across container recreation (see Open Issues).

4. Environment Variable Key Names

Env file	Access key var name	Secret key var name
`backend-environment.env` (v32 and v42)	`s3Bucket_api_accessKeyId`	`s3Bucket_api_secretAccessKey`
`s3-environment.env` (v32 and v42)	`s3Bucket_api_accessKeyId`	`s3Bucket_api_secretAccessKey`
`s3-v2-environment.env`	`pinbox24PublicAccessKeyId`	`pinbox24PublicSecretAccessKey`
`uploadAwsS3.helper.js` (hardcoded in v42)	hardcoded string (patch required)	hardcoded string (patch required)

Additional vars present in env files: s3Bucket_endpoint, s3Bucket_BUCKET_NAME.

5. Rotation Procedure

When the Wasabi key for p24-access needs to be rotated:

Step 1 Ă˘â‚¬â€ť Rotate in Wasabi IAM

Wasabi console Ă˘â€ â€™ IAM Ă˘â€ â€™ Users Ă˘â€ â€™ p24-access Ă˘â€ â€™ Security Credentials
Delete old key Ă˘â€ â€™ Create new key Ă˘â€ â€™ Save (access key ID prefix + secret)

Wasabi enforces a 2-key maximum. You must delete the old key before creating a new one (or you already have 2 keys and must delete the inactive one).

Step 2 Ă˘â‚¬â€ť Update env files on bms-1

SSH to root@94.23.26.113. Update key names s3Bucket_api_accessKeyId / s3Bucket_api_secretAccessKey in all five files:

# w3 backend
vi /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env
 
# w3 S3 microservice
vi /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/s3-environment.env
 
# w4 backend
vi /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env
 
# w4 S3 microservice (shared with w3, but separate env file for w4 working dir)
vi /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/s3-environment.env
 
# w4 S3-v2 (uses pinbox24Public* key names)
vi /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2/s3-v2-environment.env

Step 3 Ă˘â‚¬â€ť Recreate containers

Use the IMAGE_NAME/CONTAINER_NAME pattern Ă˘â‚¬â€ť env vars must be set before docker-compose:

# w3 backend
cd /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2
export CONTAINER_NAME=v32-prod
export IMAGE_NAME=$(docker inspect v32-prod --format "{{.Config.Image}}")
docker-compose up -d --force-recreate backend
 
# w3 S3 microservice
export CONTAINER_NAME=s3-v32-prod
export IMAGE_NAME=$(docker inspect s3-v32-prod --format "{{.Config.Image}}")
docker-compose up -d --force-recreate s3
 
# w4 backend
cd /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts
export CONTAINER_NAME=v42-prod
export IMAGE_NAME=$(docker inspect v42-prod --format "{{.Config.Image}}")
docker-compose up -d --force-recreate backend
 
# w4 S3-v2
cd /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2
export CONTAINER_NAME=s3-v2-v42-prod
export IMAGE_NAME=$(docker inspect s3-v2-v42-prod --format "{{.Config.Image}}")
docker-compose up -d --force-recreate s3-v2

Step 4 Ă˘â‚¬â€ť Restore local.js in s3-v32-prod (ALWAYS required after recreate)

The container loses /app/config/local.js on recreation. Write it back:

# Write local.js to a temp file on bms-1 (use actual values from env file)
cat > /tmp/local.js << 'EOF'
module.exports = {
  s3Bucket: {
    endpoint: process.env.s3Bucket_endpoint,
    accessKeyId: process.env.s3Bucket_api_accessKeyId,
    secretAccessKey: process.env.s3Bucket_api_secretAccessKey,
    bucket: process.env.s3Bucket_BUCKET_NAME,
  }
};
EOF
 
docker cp /tmp/local.js s3-v32-prod:/app/config/local.js
docker exec s3-v32-prod pm2 reload all
rm /tmp/local.js

Verify the process restarted:

docker exec s3-v32-prod pm2 list

Step 5 Ă˘â‚¬â€ť Patch uploadAwsS3.helper.js in v42-prod (ALWAYS required after recreate)

This file has hardcoded Wasabi credentials. After recreating v42-prod, patch it:

# Copy out the file
docker cp v42-prod:/app/dist/globalHelpers/uploadAwsS3.helper.js /tmp/uploadAwsS3.helper.js
 
# Edit: replace the hardcoded accessKeyId and secretAccessKey strings with new values
# Key names: look for the old access key ID prefix and replace both strings
vi /tmp/uploadAwsS3.helper.js
 
# Copy back and reload
docker cp /tmp/uploadAwsS3.helper.js v42-prod:/app/dist/globalHelpers/uploadAwsS3.helper.js
docker exec v42-prod pm2 reload all
rm /tmp/uploadAwsS3.helper.js

Step 6 Ă˘â‚¬â€ť Verify

# w3 upload test Ă˘â‚¬â€ť from inside v32-prod or via API
# Expected response: {"success": true, "location": "https://pinbox24.s3.eu-central-1.wasabisys.com/..."}
curl -s -X POST https://<w3-domain>/api/files/<officeId>/upload -F file=@/tmp/test.txt
 
# w4 signed URL test
# Expected response: {"signedUrl": "https://..."}
curl -s https://<w4-domain>/api/offices/files/<fileId>/getSignedUrl

6. Diagnosis Procedure

Symptom: w3 uploads return no response / timeout

Check s3-v32-prod logs: docker logs s3-v32-prod --tail 50
Check if /app/config/local.js exists: docker exec s3-v32-prod cat /app/config/local.js
If missing or showing DO Spaces endpoint Ă˘â€ â€™ run Step 4 above (restore local.js)
Confirm endpoint in use: docker exec s3-v32-prod env | grep s3Bucket_endpoint Expected: s3.eu-central-1.wasabisys.com

Symptom: w4 getSignedUrl returns 403 Forbidden

Check v42-prod logs: docker logs v42-prod --tail 50
The cause is almost certainly the hardcoded credentials in uploadAwsS3.helper.js
Run Step 5 above to patch the file
Verify the access key ID prefix in the patched file matches the current active Wasabi key

Symptom: s3-v2-v42-prod fails (w4 TypeScript S3 service)

Check logs: docker logs s3-v2-v42-prod --tail 50
Check env: docker exec s3-v2-v42-prod env | grep -i pinbox24
Verify pinbox24PublicBucketName Ă˘â‚¬â€ť currently set to test-replicated-to-us-bucket which may be incorrect (see Open Issues)

7. node-config Architecture (s3-v32-prod)

The S3 microservice uses the npm config package. Resolution order (later overrides earlier):

/app/config/default.js Ă˘â‚¬â€ť hardcoded, contains ams3.digitaloceanspaces.com (WRONG Ă˘â‚¬â€ť legacy)
/app/config/local.js Ă˘â‚¬â€ť override written by ops, reads from process.env.s3Bucket_*

The service does NOT use custom-environment-variables.json, so raw env vars are ignored without local.js. This is the root cause of the 2026-06-29 incident.

Long-term fix required: rebuild the image with local.js baked in, or mount it via Docker volume.

8. Open Issues

#	Issue	Severity	Status
1	`local.js` in s3-v32-prod is not persistent Ă˘â‚¬â€ť lost on container recreate	HIGH	FIXED 2026-06-29 — volume mount via `persistent-patches/local.js` in compose + containers recreated
2	`uploadAwsS3.helper.js` patch in v42-prod is not persistent	HIGH	FIXED 2026-06-29 — volume mount via `persistent-patches/uploadAwsS3.helper.js` in `docker-compose.yml` + containers recreated
3	`s3-v2-v42-prod` uses bucket `test-replicated-to-us-bucket` Ă˘â‚¬â€ť is this intentional?	MEDIUM	open — tracked in #2028; bucket confirmed on Wasabi eu-central-1
4	AWS key `AKIAYGQMT4PQ3ZLOWHRR` found in `uploadAwsS3.helper.js` `awsConf` Ă˘â‚¬â€ť purpose unknown	MEDIUM	open — dead code; #2047 (human: AWS IAM console revoke)
5	DO Spaces key prefix `CY5VG3XCNNB4N3BZSLOE` appeared in hardcoded config Ă˘â‚¬â€ť verify if still active	LOW	open — #2046 (human: DO console revoke)
6	`s3-v32-prod` logs the full MongoDB URI (with password) on startup — re-exposes the DB credential in PM2 logs on every restart, defeating rotation	HIGH (P1)	open — #2397 · fix procedure: `pinbox24-s3-mongo-uri-log-redaction.md` (persistent `persistent-patches/` volume-mount, same pattern as 2). Root SSH to bms-1 required — human-action.
7	W3 uploads broken after DB_URI migrated to rs0 — Mongoose 4.x (old-s3 image) uses OP_QUERY-wrapped inserts, incompatible with MongoDB 7.0.25	HIGH	FIXED 2026-07-01 — `persistent-patches/controller.js` bypasses MongoDB save: returns 200 + Wasabi file data on Mongoose error; `filesUpload.js` mirror insert writes metadata to w3_db on rs0 via mongojs. Volume mount added to docker-compose.yml s3 service.

9. Credential Incident Log

Date	Incident	Status
2026-06-29	Old Wasabi key (`VCUC7X6A1G...` prefix) appeared in chat output during `docker inspect` and `cat` of `uploadAwsS3.helper.js`	Key was already deleted before exposure; no action required
2026-06-29	`pinbox24PublicSecretAccessKey` value appeared in chat from `docker inspect s3-v2-v42-prod`	Same old/deleted key; no action required

Rule: if a secret value appears in chat output, stop immediately, reference key NAME only, and follow docs/playbooks/static-api-key-incident-rotation.md.

docs/playbooks/static-api-key-incident-rotation.md Ă˘â‚¬â€ť credential rotation procedure
docs/playbooks/sops-windows-crlf.md Ă˘â‚¬â€ť safe Windows SOPS writes
secrets/bms-servers.env.sops Ă˘â‚¬â€ť BMS root passwords (not Wasabi keys)
secrets/monitoring.env.sops Ă˘â‚¬â€ť contains WASABI_ADMIN_ACCESS_KEY for IAM self-management

Wasabi IAM is managed via boto3 against iam.wasabisys.com using the admin key in secrets/administration.env.sops (key name: WASABI_ADMIN_ACCESS_KEY + WASABI_ADMIN_SECRET_KEY).

Audit Log — Log to infra_operations

After completing the key rotation (Step 6 verification passed), log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="pinbox24-s3-wasabi",
    result="success",  # "success" | "failed" | "skipped"
    detail="Wasabi p24-access key rotated — env files updated on bms-1, containers recreated, local.js and uploadAwsS3.helper.js restored",
    env="bms-1",
    gh_issue=2730,  # replace with current working issue number if one exists
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'pinbox24-s3-wasabi', 'success', 'Wasabi p24-access key rotated — env files updated on bms-1, containers recreated', 'bms-1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

11. Post-Mortem: Inline Preview 403 (2026-07-03/04)

Root Causes

RC-1: IAM s3v2-prod-bms1 brak dostepu do 3 bucketow plikow — policy pokrywala tylko stary bucket pinbox24; s3-v2 uzywa 3 bucketow (60% plikow niedostepnych).
RC-2: Hardcoded usuniety klucz Wasabi w uploadAwsS3.helper.js — ccessKeyId: “VCUC7X6A1GINU54MDWON” (usuniety) + hardcoded endpoint us-east nie obslugiwany.
RC-3: IAM pinbox24-bms1-s3 brak dostepu do 3 bucketow — getSignedUrl path tez bez uprawnien.
RC-4: Wasabi EU IAM oddzielny od US IAM — zmiany przez iam.wasabisys.com nie propaguja do eu-central-1; wymagany iam.eu-central-1.wasabisys.com.

Fixes Applied (2026-07-04)

Fix	Wynik
Policy s3v2-prod-bms1 — wszystkie 3 buckety	OK
Policy pinbox24-bms1-s3 — US + EU IAM endpoint	OK
Patch uploadAwsS3.helper.js — env vars + _bucketEndpointMap	OK
Persistent patch — volume mount w docker-compose.yml	OK
docker-deploy-prod.sh — stare/usuniete klucze zastapione	OK

Patch Survival Matrix

Zdarzenie	Przezywa?
PM2 reload/restart	TAK (plik na dysku)
docker restart v42-prod	TAK (volume mount)
docker-compose up -d	TAK (zmodyfikowany docker-compose.yml)
docker-deploy-prod.sh run	TAK (skrypt naprawiony)
Swiezy git clone nadpisujacy build dir	RYZYKO — git-deploy-v42-prod to tylko Discord webhook, nie klonuje autonomicznie

Wasabi EU IAM Discovery (CRITICAL)

Wasabi ma oddzielne IAM subsystemy:

US (us-east-1, us-east-2): iam.wasabisys.com
EU (eu-central-1): iam.eu-central-1.wasabisys.com

Policy update przez US endpoint zwraca HTTP 200 ale NIE propaguje do EU bucketow.

Persistent Patch Location

Host: /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/persistent-patches/uploadAwsS3.helper.js Mount: /app/dist/globalHelpers/uploadAwsS3.helper.js:ro w kontenerze v42-prod

12. Multi-Bucket Architecture (s3-v2-v42-prod)

s3-v2 uploaduje kazdy plik do 3 bucketow rownolegle (promiseAny — odpowiedz po pierwszym sukcesie; kazdy udany upload aktualizuje MongoDB storageInfo niezaleznie).

Buckety

Bucket	Region	Datacenter
test-replicated-to-us-bucket	eu-central-1	Frankfurt, DE
p24-was-us-east-1	us-east-1	Ashburn, VA
test-us-bucket-for-replication-testing	us-east-2	Columbus, OH

Nazwy z prefixem test-* sa mylace — to produkcyjne buckety od ok. 2022.

Dystrybucja plikow w4 (stan 2026-07-04)

Metryka	Wartosc
Lacznie unikalnych plikow	214,727
Pelna redundancja (3 kopie)	127,374 (59%)
2 kopie	41,623 (19%)
1 kopia (zagrozenie)	45,730 (21%)
Job naprawczy	GH issue #2709

IAM Users (po naprawie 2026-07-04)

IAM User	Uzywa	Policy
s3v2-prod-bms1	s3-v2-v42-prod (getBase64File)	s3v2-prod-bucket-access — 3 buckety plikow
pinbox24-bms1-s3	v42-prod (getSignedUrl)	pinbox24-bucket-access — pinbox24 + 3 buckety plikow

p24-infra Docs

Explorer

pinbox24-s3-wasabi-bms1

Pinbox24 S3/Wasabi Configuration Ă˘â‚¬â€ť bms-1 (w3 and w4)

1. Overview

2. Wasabi Bucket

3. Architecture

3.1 w3 (v3.2)

3.2 w4 (v4.2)

4. Environment Variable Key Names

5. Rotation Procedure

Step 1 Ă˘â‚¬â€ť Rotate in Wasabi IAM

Step 2 Ă˘â‚¬â€ť Update env files on bms-1

Step 3 Ă˘â‚¬â€ť Recreate containers

Step 4 Ă˘â‚¬â€ť Restore local.js in s3-v32-prod (ALWAYS required after recreate)

Step 5 Ă˘â‚¬â€ť Patch uploadAwsS3.helper.js in v42-prod (ALWAYS required after recreate)

Step 6 Ă˘â‚¬â€ť Verify

6. Diagnosis Procedure

Symptom: w3 uploads return no response / timeout

Symptom: w4 getSignedUrl returns 403 Forbidden

Symptom: s3-v2-v42-prod fails (w4 TypeScript S3 service)

7. node-config Architecture (s3-v32-prod)

8. Open Issues

9. Credential Incident Log

Audit Log — Log to infra_operations

11. Post-Mortem: Inline Preview 403 (2026-07-03/04)

Root Causes

Fixes Applied (2026-07-04)

Patch Survival Matrix

Wasabi EU IAM Discovery (CRITICAL)

Persistent Patch Location

12. Multi-Bucket Architecture (s3-v2-v42-prod)

Buckety

Dystrybucja plikow w4 (stan 2026-07-04)

IAM Users (po naprawie 2026-07-04)

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

pinbox24-s3-wasabi-bms1

Pinbox24 S3/Wasabi Configuration Ă˘â‚¬â€ť bms-1 (w3 and w4)

1. Overview

2. Wasabi Bucket

3. Architecture

3.1 w3 (v3.2)

3.2 w4 (v4.2)

4. Environment Variable Key Names

5. Rotation Procedure

Step 1 Ă˘â‚¬â€ť Rotate in Wasabi IAM

Step 2 Ă˘â‚¬â€ť Update env files on bms-1

Step 3 Ă˘â‚¬â€ť Recreate containers

Step 4 Ă˘â‚¬â€ť Restore local.js in s3-v32-prod (ALWAYS required after recreate)

Step 5 Ă˘â‚¬â€ť Patch uploadAwsS3.helper.js in v42-prod (ALWAYS required after recreate)

Step 6 Ă˘â‚¬â€ť Verify

6. Diagnosis Procedure

Symptom: w3 uploads return no response / timeout

Symptom: w4 getSignedUrl returns 403 Forbidden

Symptom: s3-v2-v42-prod fails (w4 TypeScript S3 service)

7. node-config Architecture (s3-v32-prod)

8. Open Issues

9. Credential Incident Log

10. Related Files

Audit Log — Log to infra_operations

11. Post-Mortem: Inline Preview 403 (2026-07-03/04)

Root Causes

Fixes Applied (2026-07-04)

Patch Survival Matrix

Wasabi EU IAM Discovery (CRITICAL)

Persistent Patch Location

12. Multi-Bucket Architecture (s3-v2-v42-prod)

Buckety

Dystrybucja plikow w4 (stan 2026-07-04)

IAM Users (po naprawie 2026-07-04)

Graph View

Table of Contents

Backlinks