p24-infra Docs

w3-w4-stack-operations

21 min read

Pinbox24 W3/W4 Stack — Ownership, Permissions & Operations

Status: design / plan (issue #4162) Applies to: Pinbox24 W3 (v32-prod, s3-v32-prod) + W4 (v42-prod + 8 microservices) on bms-1, MongoDB rs0 on bms-2 (PRIMARY) / bms-3 / bms-4 (arbiter), plus pinbox24/p24-back-ts and pinbox24/p24-v-3.2 on GitLab. Owner (recommended): sys-admin role, executed via the existing infra-task worker on bms-4. Delegates: secret-manager (all SOPS writes), human (cross-provider IAM — e.g. Wasabi/OVH account-level actions). GitLab merge on pinbox24/* is fully autonomous as of 2026-08-02 (see §2) — no longer a human delegate.

This is the top-level ownership + operations index for the Pinbox24 stack. It replaces “search 30+ playbooks” as the entry point for any W3/W4 task. Everything in this doc links out to the authoritative playbook for each subtopic — do not duplicate credential-handling, migration procedures, or incident recipes here.

1. Ownership recommendation

Extend the existing sys-admin role with a formally-declared “Pinbox24 W3/W4 stack” specialisation. Dispatch continues via the infra-task worker on bms-4 (/opt/p24-infra/scripts/spawn-infra-task-worker.sh).

Do NOT create a new pinbox-stack-admin role. Rationale:

  • The infra-task worker on bms-4 already IS sys-admin in worker form — it holds the SSH keys to bms-1/2/3, has Docker access, and has executed every W3/W4 operation the fleet has actually run in the last quarter (#4150, #4159, W3 rollback 2026-07-10, W4 pm2 crash, MongoDB app-user rotations). See docs/playbooks/infra-task-executor.md.
  • Role boundaries stay honest: secret-manager OWNS all SOPS writes for W3/W4 (pinbox24-w3.env.sops, pinbox24-w4.env.sops, pinbox24-backends.env.sops, bms-servers.env.sops); sys-admin READS them for verification (already permitted by CLAUDE.md §Role Enforcement).
  • Creating a new role costs a new role file + /role-* skill + boundary-matrix row with zero capability gain over “sys-admin acting on the Pinbox24 subset.”
  • GitLab MR merge on pinbox24/* (p24-back-ts, p24-v-3.2) is fully autonomous, unconditional — see §2. Architect decision 2026-08-02, superseding the 2026-08-01 3-condition carve-out (which itself superseded the original “human-only” recommendation). The worker may open, review, and merge its own MRs without a prior human approval, pipeline-green check, or schema-diff restriction. Upstream (pinbox24/*) is still Pinbox24-team-owned by a separate business, not p24-infra-owned code — this is a deliberate, explicit risk acceptance by the repo owner given GITLAB_ADMIN_PAT’s pre-existing group-level Owner access made the prior gate a policy choice, not a technical one.

2. Permission matrix — sys-admin (pinbox-stack subset)

CapabilityGrantScopeSource of grantNotes
SSH root to bms-1 (94.23.26.113)✅Docker/compose ops for V32/V42 containersboth bms-4 Claude-account keys — claude-runner and claude-runner-2 id_ed25519 — in /root/.ssh/authorized_keys on bms-1 (see note ‡)Already deployed — verify with ssh -o StrictHostKeyChecking=yes root@94.23.26.113 hostname
SSH root to bms-2 (145.239.133.104)✅MongoDB rs0 PRIMARY — read admin DB, run rs.status(), restart mongod containersame keys (both accounts, note ‡)Preferred entry point for rs0 admin ops (PRIMARY)
SSH root to bms-3 (51.68.155.224)✅MongoDB rs0 SECONDARY (data-bearing) — verify replication lag, catch-up on rebuildssame keys (both accounts, note ‡)See docs/playbooks/mongodb-admin-password-recovery.md
SSH root to bms-4 (54.36.123.110)✅Arbiter maintenance + n8n workflow reconciliation for W4 direct-API ingestionsame keys (both accounts, note ‡)This is where the worker RUNS
Read secrets/pinbox24-w3.env.sops✅ (verify only)Rotation verification, incident diagnosis — never writeage recipient in the SOPS blocksops -d ... | grep -q "^KEY=" ; echo exists pattern — never print values (CLAUDE.md §Safe secret handling)
Read secrets/pinbox24-w4.env.sops✅ (verify only)Sameage recipientSame
Read secrets/pinbox24-backends.env.sops✅ (verify only)mailgun-prod container troubleshootingage recipientSame
Read secrets/bms-servers.env.sops✅ (verify only)rs0 app-user + admin creds for connectivity checks (w3_app, w4_app, admin)age recipientSame
WRITE any SOPS file❌——Always delegate to secret-manager — spawn agent per CLAUDE.md §Delegation pattern. Trying to write from sys-admin is an out-of-role action.
Docker on bms-1 (docker ps, docker exec, docker-compose up -d --force-recreate)✅V32/V42 containers listed in pinbox24-w3-w4-architecture-spec.md §3root via SSHEvery mutating ssh + docker command must carry # PLAYBOOK: <name>.md annotation (hooks enforce — see .claude/hooks/pre-bash-safety.sh)
MongoDB rs0 — connect as w3_app / w4_app (readWrite on w3_db / w4_db)✅Data fixes on pinbox.profiles, office.users, regRecords, etc.password in bms-servers.env.sops → mongodb_w3_app_password / mongodb_w4_app_passwordNever open port 27017 to the internet — bms-2/bms-3 ufw allowlist stays enforced
MongoDB rs0 — connect as admin✅ (rare)rs.reconfig, user create/rotatepassword in administration.env.sops → MONGODB_ROOT_PASSWORDPrefer w3_app / w4_app for data ops; admin only for cluster-topology changes
Redis (redis-v42 on bms-1) — connect for W4 ingestion dedup + set-password rate limits✅Debug W4 ingestion locks / RabbitMQ dedup keyspassword in pinbox24-w4.env.sops → REDIS_PASSWORDLocal container, test-net bridge — no external exposure
Wasabi S3 (pinbox24 bucket, eu-central-2) — object-level access via app IAM key✅ (read + list, avoid delete)Verify uploads, diagnose file-not-found reportskeys in pinbox24-w3.env.sops / pinbox24-w4.env.sops → s3Bucket_api_accessKeyId / pinbox24PublicAccessKeyIdBucket-admin (delete/policy) requires WASABI_ADMIN from administration.env.sops — delegate to secret-manager if a full-key rotation is needed. ⚠️ See the warning below the table — W4’s actual file storage is NOT this pinbox24 bucket.
GitLab pinbox24/p24-back-ts — clone / branch / push / open MR✅Ship code-change diffs for W4 (p24-back-ts is the W4 TS source)GITLAB_ADMIN_PAT in secrets/pinbox24-gitlab.env.sops (bms-4-decryptable narrow copy, #4796)Access restored — re-verified 2026-07-30 (#3775 closed): GET /projects/pinbox24%2Fp24-back-ts returns 200, 85 project memberships, full clone succeeds. Credential-access gap fixed 2026-08-01 (#4796): the GitLab-side grant was fine, but the bms-4 worker had no local age key to decrypt the PAT at all (administration.env.sops is single-recipient, developer-only) — blocked #4792 twice before the narrow file was added.
GitLab pinbox24/p24-v-3.2 — clone / branch / push / open MR✅Ship code-change diffs for W3 (p24-v-3.2 is the W3 legacy JS source)same PAT, same fileSame — access restored (#3775 closed); same #4796 credential-access fix applies
GitLab pinbox24/* — merge MRs✅ (unconditional)Merge freely — no approval-count check, no pipeline-status check, no schema-diff restriction. The worker may author, review, and merge its own MRs.GITLAB_ADMIN_PAT, same as abovePolicy changed 2026-08-02 (architect decision): the 2026-08-01 3-condition carve-out (real human approved_by entry + green/not-required pipeline + non-schema diff) is superseded — repo owner explicitly requested full autonomy, no conditions, after being shown the carve-out’s mechanics and choosing the broadest option. Upstream (pinbox24/*) remains Pinbox24-team-owned code, not p24-infra’s own — this is a deliberate risk acceptance given GITLAB_ADMIN_PAT already had unrestricted group-level Owner access underneath the policy layer regardless. Still exercise normal engineering judgment (read the diff, check CI if present) even though no gate enforces it — “unconditional” means no automated block, not “skip review effort.” History for context: original state was “human-only, no technical gap, standing risk-acceptance”; first revision (2026-08-01) added the 3-condition carve-out after approvals_left==0 was found to be a meaningless signal on these zero-approval-rule projects; this revision (2026-08-02) removes all three conditions. ⚠️ Coupled merge⇒deploy exception (#5762): merge autonomy is unconditional for the review/merge decision, but it does not authorize an unscheduled production deploy. Where a target branch’s .gitlab-ci.yml couples merge and deploy (a deploy-stage job with only:[<branch>] + when: on_success or no when: key), merging deploys to prod immediately — see the mandatory pre-merge check in the callout below the table.
GitLab runner on bms-1 — re-register / rotate token⚠️Playbook-driven onlydocs/playbooks/gitlab-runner-bms1-reregister.mdNot down — #2690 was closed 2026-08-01 as a false alarm (runner bms-1-autodeploy verified online: true, actively contacted, most recent pipeline !815 succeeded through the deploy stage). Re-run this playbook only if the runner is confirmed offline again.
Vercel — deploy env changes❌Not applicable to Pinbox24 (bms-1 Docker, not Vercel)——

Anything not listed → delegate. Ambiguity → treat as out-of-role.

⚠️ Merge autonomy ≠ deploy autonomy — mandatory pre-merge coupled-deploy check (#5762). The “merge MRs — ✅ (unconditional)” grant above covers the code-review/merge decision only. It does not grant authority to trigger an unscheduled production deploy. For most Pinbox24 repos/branches a GitLab merge and the prod deploy are separate steps (the deploy job is when: manual or explicitly scheduled), so “merge is safe, deploy is a later step” holds. This assumption does not generalize. At least one repo/branch collapses the two: merging into pinbox24/pinbox24-ms-s3-v2’s master branch auto-recreated the live s3-v2-v42-prod container on bms-1 within ~30 s, mid-week, because its .gitlab-ci.yml defines a deploy-stage job prod-back-end-deploy with only:[master] + when: on_success (the #5762 incident, during #2048).

Before merging any pinbox24/* MR, you MUST check the target branch’s own .gitlab-ci.yml for a coupled auto-deploy trigger — a deploy-stage job scoped to the target branch (only:[<branch>]) with when: on_success (or no when: key, which defaults to on_success). If one exists, the merge will deploy to production immediately, with no reliable window to intercept it after the fact (the deploy job goes pending→success in as little as ~30 s). In that case, either merge only inside a coordinated deploy window (e.g. #4481) or record an explicit “deploy risk accepted” note in the task’s safety assessment — never merge on autopilot assuming merge and deploy are decoupled. Full procedure, the exact API check, and the incident timeline: docs/playbooks/pinbox24-s3v2-master-branch-autodeploy.md. This is a timing/coordination obligation layered on top of the unconditional merge-review autonomy — it does not reintroduce the retired 2026-08-01 approval/pipeline/schema conditions.

‡ Both bms-4 Claude accounts are authorized on bms-1/2/3 (#5691). Under ADR-002’s multi-account topology, bms-4 runs two Claude Linux accounts — claude-runner (→radieu) and claude-runner-2 (→ecotrans) — and infra-task jobs load-balance across both. Each must be able to SSH root→bms-1/2/3 for W3/W4 ops. The claude-runner key was deployed out-of-band before the ansible key-management role existed; claude-runner-2 was never propagated, so any infra-task dispatched under it failed with Permission denied (publickey) (#5666, surfaced by the #5663 P1 diagnostic). Both keys are now the declared, managed trusted set: claude-runner-2’s public key is in root_trusted_authorized_keys (ansible/inventory/group_vars/all.yml), deployed by the additive-only ssh-authorized-keys role (ansible/playbooks/ssh-authorized-keys.yml). Managing both declaratively is the anti-drift fix — re-running the play is idempotent and can never remove a key or cause a lockout. To bring claude-runner’s primary key fully under the same managed list (it is present on the hosts but not yet in group_vars), capture its id_ed25519.pub and add it the same way.

⚠️ W4’s real file storage is 3 misleadingly-named “test-*” buckets — DO NOT treat them as disposable. test-replicated-to-us-bucket (eu-central-1), test-us-bucket-for-replication-testing (us-east-2), and p24-was-us-east-1 (us-east-1) look like throwaway test/migration buckets by name, but a live MongoDB check (2026-08-04) confirmed they hold ~215,000 files — the vast majority of W4’s actual production file data. p24-back-ts’s uploadMultiFiles() (src/globalHelpers/uploadAwsS3.helper.ts) hardcodes test-replicated-to-us-bucket as its primary target; pinbox24-w4.env.sops’s pinbox24PublicBucketName is currently, actively set to that same bucket — this is not stale config, it’s what’s live today. Never include these 3 buckets in a Wasabi cost-cleanup / lifecycle-policy / “delete unused test buckets” pass without first re-checking w4_db.files.storageInfo distribution. A rename to correctly-named production buckets is tracked as future work in p24-infra#2709 — deliberately deferred as of 2026-08-05 (architect decision) in favor of first shipping #2709’s missing-copy repair job; this warning exists so the naming hazard isn’t rediscovered from scratch or, worse, acted on destructively before the rename lands. See also wasabi-key-rotation.md for the Wasabi credential-side context.

3. Where the code lives

StackGitLab repoDefault branchDeployed imageDeploy target on bms-1
W3 (v3.2)gitlab.com/pinbox24/p24-v-3.2developmentECR 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prodcontainer v32-prod, compose dir /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/ (GitLab-runner-owned, authoritative — confirm live, see note below)
W3 s3 microservicegitlab.com/pinbox24/p24-ms-s3transport (separate repo, not a p24-v-3.2 subtree — corrected #3066, was previously documented incorrectly here)masterECR .../old-s3container s3-v32-prod, same compose dir. No .gitlab-ci.yml in this repo — old-s3 has its own manual build chain, out of scope of the p24-v-3.2 W3 pipeline (docs/playbooks/w3-gitlab-ci-image-pipeline.md §Out of scope); shipping a source fix requires a manual image rebuild + ECR push + container recreate on bms-1, not just a merge.
W4 (v4.2)gitlab.com/pinbox24/p24-back-tsdevelopmentECR .../v42-prodcontainer v42-prod, compose dir /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/
W4 microservices (s3-v42-prod, s3-v2-v42-prod, wkhtml-v42-prod, mailgun-prod, v42-notify-prod, git-deploy-v42-prod, pdf-gen-v42-prod, redis-v42)mixed (same GitLab repo for TS microservices; redis:7-alpine upstream for redis-v42)—ECR (mostly) + upstream Alpine for Redisco-located on bms-1, test-net bridge

⚠️ Build-dir path can drift — confirm live before use (#4985). The W3/W4 compose dirs are owned by the GitLab-CI runner and their build-hash segment (eZQeLfuJe, 7N4sbbrB) changes if the runner is re-registered. secrets-sync.yml (sync-pinbox24-w3) and the GitLab CI config (docs/bms-1/gitlab-ci-w3.yml → BUILDER_DIR) are the source of truth. A root-owned /root/builds/pn3C9eHo/0/pinbox24/p24-v-3.2/ still exists on bms-1 and is NOT the prod serving copy — editing it has zero effect on production v32-prod/s3-v32-prod (#4925 wasted a rotation on it). Always ground-truth first: ssh root@94.23.26.113 "docker inspect v32-prod --format '{{index .Config.Labels \"com.docker.compose.project.working_dir\"}}'".

🚫 But it is NOT safe to delete — it is an ACTIVE W3-STAGING dependency (#4985, confirmed on-server 2026-08-01). pn3C9eHo was originally reported as “stale/abandoned, safe to remove”; ground-truth proved that wrong. /opt/p24-infra/bms-1/docker-compose-w3-stage.yml bind-mounts 5 files out of this directory (persistent-patches/{filesUpload,crudService,local,controller}.js + app-backend/helperFunction/processHelper/tasksService/tasks/excel_import.js) into the running s3-v32-stage (and W3-stage backend) containers; /root/recreate_v32_3714.sh / recreate_redis_v32_3714.sh reference it too. rm -rf here breaks W3 staging (Docker recreates the deleted mount sources as empty dirs → staging services crash). 3 of the 5 mounted patches (crudService.js, filesUpload.js, excel_import.js) differ from the authoritative eZQeLfuJe copies — staging runs different patch code than prod, so repointing is a review decision, not a mechanical swap. Removal is blocked until W3-staging is first migrated off this path — see the #4985 remediation plan on the issue. Before any deletion, ground-truth mounts too, not just the serving working_dir: for c in $(docker ps -aq); do docker inspect "$c" --format '{{range .Mounts}}{{.Source}}{{"\n"}}{{end}}' | grep -q pn3C9eHo && docker inspect "$c" --format '{{.Name}}'; done.

Container inventory + persistent-patch table: docs/pinbox24-w3-w4-architecture-spec.md §3. GitLab repo location: docs/pinbox24/gitlab-repo-locations.md. PAT access regression (#3775) resolved and closed 2026-07-30.

Persistent-patches gotcha

Both stacks ship real production fixes as host-mounted JS files (survive force-recreate, do NOT survive image rebuild + docker-compose down -v). Any GitLab MR that lands the fix upstream must be paired with removing the mount from docker-compose.yml on bms-1 or the mount silently overrides the newly-shipped fix. Full mount inventory: docs/pinbox24-w3-w4-architecture-spec.md §2.1 (W3) + §3.1 (W4).

⚠️ “Fix merged upstream” is NOT sufficient to drop a mount — verify the DEPLOYED IMAGE’s baked dist/, not just GitLab development (#5992). The de-hardcode (63686ca5) and socket-auth try/catch (1fba09e5) fixes for uploadAwsS3.helper.js / socket.middleware.js are both merged into p24-back-ts development, yet the live v42-prod:latest ECR image (built 2026-08-06) still bakes the old dist/ — hardcoded Wasabi creds and no try/catch. Dropping either mount would revert the fix (and re-expose credentials in running code). Ground-truth by extracting the baked file from the image itself (docker create <img> → docker cp <cid>:/app/dist/... -), bypassing the :ro mount — do not trust the merge alone. Both mounts are therefore still load-bearing.

W4 persistent-patch inventory (git-tracked source → deploy step, reconfirmed live 2026-08-09 #5992):

Patch fileContainerGit sourceDeployed by (secrets-sync.yml step)Status
uploadAwsS3.helper.jsv42-prodinfra-src/pinbox24/w4/persistent-patches/sync-pinbox24-w4 → Deploy W4 persistent-patchesrequired (deployed image dist stale + carries hardcoded creds)
socket.middleware.jsv42-prodinfra-src/pinbox24/w4/persistent-patches/sync-pinbox24-w4 → Deploy W4 persistent-patchesrequired (deployed image dist lacks the try/catch)
app.js, postbookReport.helper.jsv42-prodinfra-src/pinbox24/w4/persistent-patches/sync-pinbox24-w4 → Deploy W4 persistent-patchesrequired
storage.controller.js, storage.config.js, app.routing.jss3-v2-v42-prodinfra-src/pinbox24/w4/s3-v2-persistent-patches/sync-pinbox24-w4 → Deploy s3-v2-v42-prod patchesrequired
mailgunFileHandler.helper.jss3-v2-v42-prodinfra-src/pinbox24/w4/s3-v2-patches/sync-pinbox24-w4 → Deploy s3-v2-v42-prod patchesrequired
integration.helper.jsmailgun-v42-prodinfra-src/pinbox24/w4/mailgun-patches/sync-pinbox24-backends → Deploy mailgun-v42-prod persistent patchrequired — **git-tracked as of 5992 (previously server-only, no deploy step)

The gus-api-regon-wsdl/ mount was removed (never took effect). Orphaned untracked patch files (common.const.js, redis.config.js) that sat in the W4 build dir were removed from bms-1 in #5992 (unmounted, never in git; common.const.js additionally carried hardcoded token literals so it could not be committed).

s3-v32-prod recreate target — the compose container is STAGING (alias trap, #5170)

The container named s3-v32-prod is NOT the one prod resolves. Same class as the pn3C9eHo trap above: the obvious target is not the prod-serving one. A rotation that addresses the S3 service by container name refreshes staging and leaves prod on the old credential generation, silently.

Verified live on bms-1 (ns367522), 2026-08-03, read-only (#5170, split out of the 5166 exposure audit):

ContainerImageNetwork(s)s3-v32-prod alias on prod-v-3-net?Role
s3-v32-prod (compose-managed)ECR .../old-s3test-net onlyno (alias s3 + short-id)staging
s3-v32-prod-renamedprivate-registry.dev.pinbox24.com/old-s3prod-v-3-net (+others), restart_policy: noyesprod-serving S3
  • On prod-v-3-net, the DNS name s3-v32-prod resolves to s3-v32-prod-renamed (172.19.0.7), not to the compose container of that name. Confirmed from the backend: docker exec v32-prod getent hosts s3-v32-prod → 172.19.0.7 s3-v32-prod = -renamed.
  • s3-v32-prod-renamed carries no prod-v-3-net entry in infra-src/pinbox24/w3/docker-compose.yml — it is an out-of-band container holding the alias. The compose s3 service declares test-net only, so docker-compose up -d --force-recreate ... s3 (and the docker stop/rm s3-v32-prod that precedes it in sync-pinbox24-w3) operate on the staging container.
  • The sync-pinbox24-w3 post-deploy health checks inspect the same staging container — including the inter-container nc gate, which hardcodes .NetworkSettings.Networks."test-net". So a W3 credential rotation can go fully green while prod S3 runs stale creds (silent-stale-credential class, #5153).
  • Live confirmation: sync run #5198 (in progress ~23:29 UTC 2026-08-02) force-recreated the compose s3-v32-prod at ~23:31 with the new password; s3-v32-prod-renamed (up since 21:51) was untouched — the bug reproduced during an unrelated real rotation.

Recreate-target rule (until the durable fix lands): to operate on prod S3, address the alias holder on prod-v-3-net, never the container name. Ground-truth first: docker network inspect prod-v-3-net + docker inspect s3-v32-prod-renamed --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{$v.Aliases}}{{"\n"}}{{end}}'.

Durable fix (Friday-window sys-admin, coordinated — NOT a merge-only change): give the compose-managed s3 service prod-v-3-net with aliases: [s3-v32-prod] and retire the orphan -renamed (overlaps #4886; same name-vs-alias root class as #4572). This is a cutover, sequence matters: remove the alias from -renamed first, then recreate compose s3 onto prod-v-3-net. Merging the compose change alone (or recreating before removing the old alias) leaves two containers claiming s3-v32-prod on prod-v-3-net → nondeterministic Docker embedded DNS → ~50% of prod file ops hit whichever container answers → partial prod outage. Two coupled constraints for that change:

  • Keep s3-environment.env free of VIRTUAL_HOST / LETSENCRYPT_HOST — adding prod-v-3-net puts s3 in nginx-proxy reach and the deliberate allowlist omission (see the 3778 block in secrets-sync.yml) is what keeps api.w3 traffic from 404ing.
  • Repoint the sync-pinbox24-w3 health checks to resolve S3 by the prod alias (drop the hardcoded "test-net" index) in the same change, or they keep validating staging.

Finding 2 — authoritative env store (verified #5170). W3 has three live env generations plus a stale legacy copy:

secrets/pinbox24-w3.env.sops   (SOPS, canonical — AUTHORITATIVE)
  → secrets-sync.yml (sync-pinbox24-w3)
    → /opt/p24-infra/bms-1/pinbox24-w3.env   (deployed, read-only mirror)
      → upsert into /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/{backend,s3}-environment.env
        (what the containers mount via env_file)
/home/p24-server-scripts/v3/v32/backend-environment.env  (LEGACY, mtime 2026-07-11 — NOT in the chain, ignore)

The transient /opt-newer-in-wall-clock-but-older-in-content drift the audit saw at 14:44 was reconciled by the next sync: run #5198 rewrote /opt/.../pinbox24-w3.env (2026-08-02 23:31:15) and the gitlab-runner copies (23:31:20) together. Authoritative store = the SOPS file. Any future divergence of the deployed /opt file from SOPS content means a direct-to-host write bypassing secrets-sync.yml — re-run sync-pinbox24-w3 to reconcile; never hand-edit the deployed .env.

4. Deploy path

Developer / Claude worker on bms-4 (sys-admin, §2: unconditional autonomous merge)
  └─ clone p24-back-ts via GITLAB_ADMIN_PAT
     └─ push branch, open MR, merge MR
                                                        │
                                                        ▼
                                   GitLab CI (runner on bms-1)
                                                        │
                              ┌─────────────────────────┴─────────────────────────┐
                              ▼                                                   ▼
                   Build & push ECR image                    (fallback if runner is ever down)
                              │                                  Manual: docker pull + force-recreate on bms-1
                              ▼                                  (`docs/playbooks/pinbox24-bms1-manual-deploy.md`)
                     force-recreate v32-prod / v42-prod

Detail:

5. Auth / password model

Client-side MD5, verbatim string compare on the server. Reset flow mints a token, emails the raw form, stores sha256(token) at rest (target model — hardening in flight via #4159, see w3-w4-password-set-secret-manager.md).

5.1 Redis auth model — V42_REDIS_PASSWORD is intentionally EMPTY (verified #5069)

V42_REDIS_PASSWORD in secrets/pinbox24-w4.env.sops is the one line with no ENC[...] wrapper because SOPS leaves empty strings unencrypted — the value is empty, and that is currently safe by network isolation, not an accidental exposure gap. This is re-flagged periodically (originally #4981 item 3 → split to #5069); this note exists so it is not re-derived each time.

Verified live on bms-1 (ns367522), 2026-08-02 (#5069, read-only):

  • redis-v42 (redis:7-alpine) publishes no host port — docker port redis-v42 is empty and NetworkSettings.Ports."6379/tcp" is null. Port 6379 is not bound to any host interface, so the internet / host network cannot reach it (ufw is moot — there is nothing to firewall).
  • It sits only on the internal docker bridges prod-v-4-net + test-net.
  • No requirepass is configured (redis-cli ping → PONG, CONFIG GET requirepass length 0), so the empty SOPS value matches the running container. V42_REDIS_HOST=redis-v42 (docker service name), so the W4 backend reaches it over the bridge, not over TCP.
  • v42-prod connects cleanly — Up, zero redis auth/connection errors in its logs.

Divergence from W3 / defense-in-depth gap (not fixed here): redis-v32 does carry a V32_REDIS_PASSWORD (W3 historically ran an external OVH-managed Redis — see closed #2415 “hasło zgubione po rotacji”, #3481). redis-v42 shares test-net with ~21 containers, including the internet-facing nginx-proxy / nginx-proxy-letsencrypt and the entire W3 stack, so any compromised co-located container has unauthenticated access to it (lateral movement). Setting a requirepass would close that, but it is a coordinated multi-container prod deploy (set requirepass on redis-v42 + update SOPS + secrets-sync.yml to bms-1 + recreate every redis consumer atomically), which is sys-admin deploy scope, not a secret-manager value judgement. Tracked as a follow-up hardening issue — do not treat the empty value as a live incident in the meantime.

6. Import paths (RESO + W4 direct API)

  • RESO Excel import (W3): mounted at app-backend/helperFunction/processHelper/tasksService/tasks/excel_import.js (NOT a persistent-patches/ file). This is a stock, git-tracked file in pinbox24/p24-v-3.2, not a p24-infra-managed hotfix — restored by the GitLab checkout itself, not by secrets-sync.yml, and byte-identical to the copy baked into the v32-prod image (the mount is a redundant no-op). Verified #4708 (2026-07-30). So force-recreate cannot lose it: if the build-dir copy is gone, the checkout restores it; if the mount is dropped, the image copy is identical.
  • W4 direct API ingestion (n8n MUjApqruo6H88ebw on bms-4): scheduleTrigger polls at :20/:30/:40 UTC into three folders (rechnungen / gutshrifts / standard). No singleton lock — overlap on backlog runs causes duplicate re-ingest (dedup on backend prevents double-booking but balloons the manual-confirm queue). Design fix filed as #4008 (Redis per-folder lock).
  • Post-incident verify checklist: docs/playbooks/reso-import-post-incident-verify.md
  • Incident history: docs/pinbox24/incidents/ + inline in docs/pinbox24/incident-register.md

7. Monitoring hooks

  • Prometheus rules: monitoring/prometheus/rules/pinbox24.yml — Pinbox24HeapUsageHigh (per-service threshold; v32-prod recalibrated to >0.93 due to babel-node baseline — see pinbox24-w3-operations.md §V8 heap baseline) + Pinbox24HeapUsageCritical (>0.95).
  • Grafana dashboard: docs/grafana-pinbox24-ops-dashboard.md.
  • Monitoring strategy plan (blackbox + /api/health + connectivity exporters): docs/plans/plan-4132-w3-w4-monitoring-strategy.md.
  • Merged gap fixes: #4012 (W3 v32-prod app-layer parity, PR #4015) + #4014 (pm2-exporter real heap, PR #4017) + #4013 / cross-repo et-operational-platform#1207.
  • Cross-stack Stack Connectivity Health dashboard (design in flight): #4018.

8. Escalation paths

SituationEscalate toVehicle
Need to add / rotate / retrieve any W3 or W4 credentialsecret-managerdocs/playbooks/secret-manager-request.md — never inline SOPS writes
Need to merge a GitLab MR on pinbox24/*autonomous (sys-admin), unconditional§2 — no approval/pipeline/schema gate as of 2026-08-02. Log the merge (what changed, why) in the session report / issue comment regardless — no gate doesn’t mean no record. First run the mandatory coupled-deploy pre-merge check (§2 callout + docs/playbooks/pinbox24-s3v2-master-branch-autodeploy.md): a merge into a branch whose .gitlab-ci.yml has a deploy job only:[<branch>] + when: on_success deploys to prod immediately (#5762).
GITLAB_ADMIN_PAT returns 404 for pinbox24 grouphuman (GitLab UI, owner access)Historical: #3775 (closed 2026-07-30, access restored). If it recurs, file a new issue — don’t reopen #3775.
GitLab runner on bms-1 offlinehuman (GitLab UI runner regen + on-server registration)#2690 — docs/playbooks/gitlab-runner-bms1-reregister.md
MongoDB rs0 topology change (member add/remove, reconfig)Playbook-driven sys-admindocs/playbooks/mongodb-admin-password-recovery.md + rs.reconfig helpers
Suspected security incident (creds leaked in logs, unexpected user, unknown container)sys-securitySpawn per CLAUDE.md §Delegation pattern; incident report BEFORE workaround (repo radieu/p24-infra label security)
Feature-code change in p24-back-ts / p24-v-3.2 beyond a targeted patchdev-coder, autonomous mergeDesign the diff, open MR, merge per §2 (unconditional) — larger/riskier diffs still warrant more careful self-review before merging, just not a mandatory external gate

9. Out of scope for this doc (already covered elsewhere)

  • SOPS file naming, age recipient list, canary + Windows-vs-Linux write pattern → CLAUDE.md §Secrets + docs/playbooks/secret-manager.md
  • Server IP inventory + VPN status → CLAUDE.md §Server IPs + docs/playbooks/server-ssh-connectivity.md
  • Node/PM2 log redaction fix (MongoDB URI leak) → #2397 + docs/playbooks/pinbox24-s3-mongo-uri-log-redaction.md
  • .env.prod committed with hardcoded prod creds in p24-back-ts (W4) → #4992 + docs/playbooks/pinbox24-w3-w4-security-policy.md §3; rotation folded into #2763; history scrub is human-gated (external prod repo — coordinate before force-push, per §2)
  • Legacy bms-1 containers (v32-prod-socket, -reso, etc.) audit → docs/playbooks/pinbox24-w3-w4-security-policy.md §4
  • Artnet decommission dependency (W3 -socket/-reso env still on artnet) → docs/artnet-resignation-letter.md

10. Follow-up (out of scope for this design PR)

Three separate issues, one per delegate:

  1. secret-manager (verify-only) — confirm the age recipient list on pinbox24-w3.env.sops, pinbox24-w4.env.sops, pinbox24-backends.env.sops, bms-servers.env.sops includes the sys-admin/infra-task worker’s key. No writes; issue is verify + report.
  2. sys-admin (verify-only) — confirm claude-runner id_ed25519 is in /root/.ssh/authorized_keys on bms-1 / bms-2 / bms-3. Already true in current fleet; issue is verify-only.
  3. human — GitLab merge-rights decision on pinbox24/p24-back-ts and pinbox24/p24-v-3.2 — fully resolved 2026-08-02: (a) restore pinbox24 group membership for GITLAB_ADMIN_PAT — done, #3775 closed 2026-07-30; (b) decide whether to grant Maintainer/merge rights or keep human-merge — GITLAB_ADMIN_PAT already carried group-level Owner access (GET /projects/:id → permissions.group_access.access_level: 50), so there was never a technical rights gap, only a policy choice; (c) how far to extend the policy — 2026-08-01 first landed a 3-condition carve-out, then 2026-08-02 the repo owner explicitly chose full unconditional autonomy after reviewing the carve-out’s mechanics (see §2). No open follow-up remains on this item.

Graph View

Backlinks