p24-infra Docs

pinbox24-do-spaces-investigation-bms1

7 min read

Playbook: Pinbox24 DigitalOcean Spaces / S3 history & credential investigation (bms-1)

Created: 2026-06-29 Origin: #2028 — investigation opened from the w3/w4 upload incident (2026-06-29) Companion (the FIX side): pinbox24-s3-wasabi-bms1.md (#2027) — config-persistence: how the Wasabi overrides survive container recreate. Related: pinbox24-w3-w4-outage-diagnosis.md, bms-1-v3x-sunset-migration-plan.md, cloud-services-operations.md §Wasabi S3

Scope & boundaries. This is the investigation/audit half of the 2026-06-29 S3 incident (#2028). Its companion #2027 (pinbox24-s3-wasabi-bms1.md) documents how to keep the Wasabi fix from reverting; this doc tracks what was historically connected and why credentials were hardcoded. The bms-1 application source (uploadAwsS3.helper.js, the S3-proxy default.js) lives in the separate Pinbox24 production codebase, not in p24-infra, so the live/runtime answers need a server-operation session on bms-1 + external console access. This doc consolidates the in-repo facts and tracks the open items — it does not change any runtime config.

1. Discoveries (from the 2026-06-29 incident)

  1. s3-v32-prod (v3.x S3-proxy, default.js) had a hardcoded default endpoint: 'ams3.digitaloceanspaces.com' — almost certainly legacy config from before the DigitalOcean Spaces → Wasabi migration. (#2027 overrides this via /app/config/local.js.)
  2. v42-prod (uploadAwsS3.helper.js) had hardcoded Wasabi credentials baked into the compiled helper — same values as the env file, but source-embedded rather than env-read.
  3. s3-v2-v42-prod referenced pinbox24PublicBucketName=test-replicated-to-us-bucket — an unknown bucket name, not the expected pinbox24.
  4. A DigitalOcean Spaces access-key id (prefix CY5VG3X…) appeared in code — activity status unknown.
  5. An AWS access-key id with prefix AKIA… (AKIAYGQ…) appeared in uploadAwsS3.helper.js awsConf. The AKIA prefix denotes a real AWS IAM key (not Wasabi) — purpose unknown.

Credential handling. Key IDs are shown above in truncated form only. Never paste a secret access-key VALUE into this doc, a commit, or chat. The full IDs are already on issue #2028; if any secret VALUE turns up in code or command output, stop and follow static-api-key-incident-rotation.md.

2. Storage architecture (known)

GenerationContainer(s)PathBackend (known)
v3.x (legacy)s3-v32-prod, -renamed, -socket, -reso (image old-s3:latest, ECR)App → internal S3-proxy microservice → object storeProxy default.js defaulted to DigitalOcean Spaces (ams3.digitaloceanspaces.com); corrected to Wasabi via the #2027 local.js override. Internal-only services on prod-v-3-net.
v4.x (current)v42-prod (uploadAwsS3.helper.js)App → Wasabi S3 directlyWasabi bucket pinbox24. Confirmed 159,108 objects present (≥ 2021).

The s3-v32-prod* proxies are catalogued as v3.x “supporting microservices” in bms-1-v3x-sunset-migration-plan.md §2.2 and are slated for removal in the v3.x sunset (only after all w3 clients migrate — v3.x still has live users, that plan §3).

Wasabi account context: the infra-side Wasabi account is 100000049371, but the buckets in cloud-services-operations.md §Wasabi S3 (ecotrans-monitoring, ecotrans-monitoring-test, p24-infra) are the infra/monitoring buckets. The Pinbox24 app bucket pinbox24 is a separate, app-owned bucket — not currently catalogued there, and (per #2027) the app’s Wasabi credentials are not yet in secrets/*.env.sops.

3. Open investigation tracker

Items require production access to bms-1 + external console access (DigitalOcean / AWS / Wasabi). Human-gated — run from a server-operation session with the relevant playbook annotation, never unattended by an automated worker. Each maps to a #2028 checklist item.

#QuestionHow to answerOwnerStatus
1Did a pinbox24 bucket ever exist on DigitalOcean Spaces, and does it still hold files?DO Spaces console / s3cmd --host=ams3.digitaloceanspaces.com ls with the DO keyhuman (server-op)open
2Are all historical (2021+) files on Wasabi pinbox24?Partially confirmed: 159,108 objects on Wasabi. Cross-check vs. any DO Spaces residue from #1humanpartially confirmed
3What is test-replicated-to-us-bucket (s3-v2-v42-prod pinbox24PublicBucketName)?Inspect s3-v2-v42-prod env on bms-1; check Wasabi/AWS for that bucket + any replication rule—answered 2026-06-29 — see §3.2; bucket EXISTS on Wasabi eu-central-1 (agent verified)
4Is DigitalOcean Spaces key CY5VG3X… still active?DO API/console: list keys; test a scoped ls. If active → rotate (static-api-key-incident-rotation.md) and de-hardcode—answered 2026-06-29 — see §3.2; #2046 (DO console)
5What is AWS key AKIAYGQ… (awsConf) for? AKIA = real AWS IAMAWS IAM console: identify user/role + policies; determine if still used (ECR vs. S3)—answered 2026-06-29 — see §3.2; #2047 (AWS console)
6When did the DO Spaces → Wasabi migration happen, and was data migration complete?Correlate S3-proxy default.js history, bucket creation dates, and the 159,108-object counthumanopen
7Region discrepancy: which Wasabi region is authoritative for pinbox24?#2027 names the app upload endpoint s3.eu-central-2.wasabisys.com; #2028’s report says objects are on eu-central-1. Confirm the live bucket region + endpoint on bms-1 and reconcile both playbooks—answered 2026-06-29 — see §3.2

3.2 Investigation findings — 2026-06-29 server-op session

Item 3 — test-replicated-to-us-bucket is a Wasabi eu-central-1 bucket (not AWS S3)

s3-v2-v42-prod container env (bms-1, inspected 2026-06-29):

pinbox24PublicBucketName=test-replicated-to-us-bucket
pinbox24PublicEndpoint=s3.eu-central-1.wasabisys.com
pinbox24PublicOfficeBucketName=test-replicated-to-us-bucket
pinbox24PublicOfficeEndpoint=s3.eu-central-1.wasabisys.com

The uploadToAWSS3 export and appendToLogFile in persistent-patches/uploadAwsS3.helper.js both use s3.eu-central-1.wasabisys.com as endpoint with test-replicated-to-us-bucket as the bucket. This is not AWS S3 — it is a Wasabi eu-central-1 bucket of that name.

head-bucket against both test-replicated-to-us-bucket and pinbox24 (Wasabi eu-central-1) returned 403 (not 404) with the v42-prod env credentials — consistent with stale app credentials (the same stale-key issue from #2027); bucket likely exists. Confirm and list via Wasabi admin key.

Also resolves item 7: all pinbox24 app containers use eu-central-1 (s3.eu-central-1.wasabisys.com). The eu-central-2 references elsewhere in p24-infra docs (bms1-dr-plan.md, bms1-ubuntu-2204-upgrade.md) are for the infra/monitoring Wasabi bucket (p24-infra) — a completely separate account. Playbook #2027 was already correct on this point.

Two Wasabi eu-central-1 buckets confirmed in use (2026-06-29 full container scan):

BucketUsed by containers
pinbox24s3-v32-prod, s3-v32-prod-renamed, s3-v32-prod-socket, v42-prod
test-replicated-to-us-buckets3-v2-v42-prod only

Item 4 — DO Spaces key CY5VG3XCNNB4N3BZSLOE — inactive in production

Grep across all bms-1 build dirs and running container envs (2026-06-29):

  • Found in stage.env (v3.2 staging build) as s3Bucket_api_accessKeyId: CY5VG3XCNNB4N3BZSLOE
  • Found COMMENTED OUT in docker-deploy-prod.sh and docker-deploy-stage.sh
  • NOT present in any running container env (scanned all containers)

This is a legacy DO Spaces key from before the Wasabi migration, still referenced in staging config but not active in production. It may still be a live key on the DigitalOcean side.

Action required (human, DO console): #2046 — revoke CY5VG3XCNNB4N3BZSLOE. Zero production impact.

Item 5 — AWS IAM key AKIAYGQMT4PQ3ZLOWHRR — dead code, effectively unused

Location: persistent-patches/uploadAwsS3.helper.js line 35 in the awsConf block.

The || process.env.ACCESS_KEY_ID fallback is unreachable (non-empty string literal is always truthy). Every S3 instance that uses the global awsConf also explicitly sets accessKeyId from env vars, overriding the AKIA key ID. The key is never transmitted to any endpoint — dead code.

The secretAccessKey in awsConf is set from the Wasabi secret env var — wrong pairing for an AWS IAM key (confirms abandoned mid-migration). appendToLogFile inherits the Wasabi secret via global config but overrides accessKeyId with the Wasabi key from env.

Action required (human, AWS console): #2047 — deactivate AKIAYGQMT4PQ3ZLOWHRR (24h hold, then delete). Zero production impact.

3.1 Remediation (regardless of investigation outcome)

  • Hardcoded credentials are a standing security finding. Even where hardcoded values match the env file, source-embedded credentials must be removed in favour of process.env.* reads, then rotated. This is the durable Option B in #2027 — owned by the Pinbox24 app repo, not p24-infra. Track there and link back here.
  • Any key confirmed active in 5 → rotate immediately via the static-key incident playbook; update the app secrets store + live container env; never re-hardcode.

4. Safe inspection commands (read-only, human-run on bms-1)

Never print a secret value. Check key NAMES / object listings only.

ssh root@94.23.26.113   # PLAYBOOK: pinbox24-do-spaces-investigation-bms1.md
 
# Which S3 endpoint does the v3.x proxy actually resolve to? (names only, redact values)
docker exec s3-v32-prod env | grep -iE 'endpoint|bucket|region' | sed 's/=.*/=<redacted>/'
 
# v42-prod storage config — names only, never the secret value
docker exec v42-prod pm2 env 0 2>/dev/null | grep -iE 'bucket|endpoint|region|s3' | sed 's/value: .*/value: REDACTED/'
 
# SAFE: --format scoped to avoid Env section exposure (issue #2970)
# Confirm the unknown bucket name on the v2/v42 proxy (key name only — value redacted)
docker exec s3-v2-v42-prod printenv | grep -iE '^PublicBucketName=' | sed 's/=.*/=<redacted>/'
# NOTE: do NOT use 'docker inspect --format {{range .Config.Env}}' — it dumps ALL env vars
# before the grep filter runs (entire Env section passes through the shell pipeline buffer)

Do NOT run docker exec <c> env or pm2 env 0 without the grep+redact filter — they print credentials in plaintext (see the warning in pinbox24-w3-w4-outage-diagnosis.md §“pm2 env 0 leaks credentials”).

5. Disposition

  • In repo (#2028, this playbook): architecture + facts consolidated; investigation tracked.
  • Companion #2027 (pinbox24-s3-wasabi-bms1.md): the Wasabi config persistence/fix.
  • Human-action / server-op session: items 1–7 above + rotation of any active key.
  • Pinbox24 app repo: remove hardcoded credentials from uploadAwsS3.helper.js and the S3-proxy default.js defaults; switch to env reads (Option B in #2027).

Update the tracker as each item resolves; close #2028 only when all seven are answered (or explicitly deferred with a reason).

Graph View

Backlinks