p24-infra Docs

pinbox24-w3-w4-health-verification

12 min read

Pinbox24 W3/W4 Health Verification — THE checklist to run after any deploy/rotation/incident

Created: 2026-08-03 (issue #5180) — consolidates verification steps that were previously re-derived ad hoc in every incident’s issue comments (#5093, #5107, 5166 each improvised their own one-off checklist). Status: operational checklist — scannable, meant to be run during an incident, not read start-to-end. Owner: sys-admin (infra-task worker) for live checks · secret-manager for anything touching a credential value. See docs/w3-w4-stack-operations.md for the full role/permission matrix.

Automated counterpart (#5184): the authenticated login + PDF-gen checks in §2, a W3 RESO-upload/S3-pull-back/delete infra-connectivity check (w3_reso_upload_pullback, against a real RESO EUROPA record, cleaned up after every run), and three read-only W4 checks — w4_postbook_list (postbook register listing), w4_postbook_download (download an existing postbook-type file), w4_inline_preview (getSignedUrl + fetch, asserts Content-Disposition: inline), and a direct-Wasabi wasabi_sandbox_roundtrip (put/list/get/delete against the isolated pinbox24-e2e-sandbox bucket with the scoped W3_E2E_WASABI_* key from #5184 Phase A2 / PR #5196 — verifies that credential is live and touches zero production data) — are scripted in .github/workflows/w3-w4-e2e-verification.yml (gh workflow run w3-w4-e2e-verification.yml), backed by scripts/tests/test-w3-w4-e2e.py. The W3 check proves upload → S3 write → S3 read works — it does not exercise the excel_import workflow/task itself (registry-config-specific, verified separately via §Step 3 below). The three W4 checks are read-only against pre-existing Eco-Trans records (found live 2026-08-09, never guessed — see the script’s docstring) — no cleanup needed, but also no coverage of a disposable W4 upload/delete round trip. Still not automated: RESO import processing itself, inline-PDF display in ecotrans (et-operational-platform, cross-repo — investigated 2026-08-11, routed to et-operational-platform#1677; it is an ecotrans-owned route needing a Supabase zarzad JWT this suite does not hold, so it is not scriptable here), and a W4-side disposable upload/pull-back check (no confirmed disposable W4 record yet).

Gotcha found live (2026-08-09): a Wasabi SigV4 presigned URL signs the HTTP method — calling getSignedUrl and then issuing a HEAD against the returned URL gets a 403 from Wasabi even though the URL is valid. Always GET it (or match whatever method the URL was actually signed for).

This is not a diagnosis or fix playbook. It only tells you whether W3/W4 are healthy right now and gives you a fast, safe way to confirm it. If a check below fails, jump straight to §4 When this isn’t enough.

0. Start here — zero-token scripted check (2026-08-09)

Before doing anything manual (SSH, docker ps, pm2 list, rs.status()), run:

.\scripts\w3-w4-quick-status.ps1

This reads three signals that are already scripted and running on their own schedule — it does not SSH anywhere and costs no AI tokens beyond the few seconds it takes to read the output:

  1. Latest health-check.yml run (cron: every 2h) — w3 / w4 / pdfgen HTTP step outcomes.
  2. Latest w3-w4-e2e-verification.yml run (cron: daily 05:30 UTC) — per-check PASS/FAIL parsed from the log (w3_login, w4_login, pdfgen, w3_reso_upload_pullback).
  3. Alertmanager firing alerts filtered to W3/W4-relevant alertnames (Pinbox24*, V32Prod*, V42Prod*, Mongo*, Pm2*, MailgunPipeline*) — credentials are PROMETHEUS_BASIC_AUTH_USER/PROMETHEUS_BASIC_AUTH_PASSWORD in secrets/monitoring.env.sops (NOT GRAFANA_ADMIN_* — see #2191; several docs had this wrong until 2026-08-09).

Pass -TriggerFresh to fire an on-demand run of both workflows instead of trusting the latest scheduled one (~1-2 min wall clock, still zero AI tokens — these are plain scripted checks, same ones CI already runs unattended).

Why this covers more than it looks like: container crash-loops (PM2 restart count), Mongoose disconnects, V8 heap pressure, and MongoDB rs0 replication lag/no-PRIMARY are already continuously monitored by Prometheus (monitoring/prometheus/rules/pinbox24.yml, monitoring/prometheus/rules/mongodb-replset.yml) — re-deriving them by hand over SSH every time someone asks “is W3/W4 healthy” duplicates monitoring that’s already running. Only reach for the manual §2 checklist below when this script reports RED, or when you need the JWT/SOPS-digest cross-check in §2.3 after a credential rotation (that one genuinely can’t be scripted — it’s a point-in-time proof that a specific rotation was picked up, not a standing health signal).

Upload / PDF-generation coverage specifically (the parts people usually ask about by name):

  • PDF generation — exercised for real by the pdfgen check in both health-check.yml and w3-w4-e2e-verification.yml (real POST /api/v1/pdf-gen, asserts %PDF magic bytes).
  • Upload → storage round-trip — exercised by w3_reso_upload_pullback (W3 only): uploads a disposable fixture to a real RESO EUROPA record, reads it back via the app’s own storage-proxy route, asserts byte-for-byte match, deletes it. Proves the S3/Wasabi write+read path works.
  • NOT covered yet (see scripts/tests/test-w3-w4-e2e.py module docstring for the full list): a W4-side upload/pull-back check (no confirmed disposable W4 record identified), the excel_import workflow/task itself (only the storage path is exercised, not RESO import processing), and inline-PDF preview/display in ecotrans (et-operational-platform — cross-repo, investigated 2026-08-11 and routed to et-operational-platform#1677: the flow is the ecotrans-owned GET /api/pinbox/preview-url route + iframe, authenticated by a Supabase zarzad/platform_admin JWT this suite does not hold, so it is not scriptable from here).

1. What “healthy” means

1.1 Container map (current — see note on staleness below)

DomainBackend containerNetworkDB connection
w3.pinbox24.com / api.w3.pinbox24.comv32-prod on bms-1test-net (+ prod-v-3-net)connects as w3_app (readWrite on w3_db) to MongoDB rs0 (bms-2 PRIMARY + bms-3), not admin — see docs/w3-w4-stack-operations.md §2
s3.w3.pinbox24.com (internal, via v32-prod proxy)s3-v32-prodtest-net (internal only)DB_URI → same w3_app model
w4.pinbox24.com / api.w4.pinbox24.comv42-prod on bms-1test-netconnects as w4_app (readWrite on w4_db) to the same rs0 cluster
pdf-gen-api.w4.pinbox24.compdf-gen-v42-prodtest-netstateless, no DB, no credential env vars
Notificationsv42-notify-prodprod-v-4-net, test-net—

MongoDB rs0 topology: bms-2 (145.239.133.104) PRIMARY, bms-3 (51.68.155.224) SECONDARY, bms-4 (54.36.123.110) ARBITER.

⚠️ Do not trust the DB-connection column of the older pinbox24-w3-w4-outage-diagnosis.md container-map table — it still lists W3 connecting to an artnet.pl-hosted rs0 with an admin credential. That model is superseded: since PR 6c3c5b03, both v32-prod and v42-prod connect as their own scoped w3_app/w4_app MongoDB users on the bms-2/bms-3 rs0 cluster, never as admin (confirmed in docs/secrets-rotation-log.md’s 2026-08-02 MongoDB admin rotation row: “the recreate every bms-1 Pinbox24 Mongo consumer step was correctly skipped as no-longer-applicable… v32-prod connects as w3_app and v42-prod connects as w4_app, never as admin”). The artnet.pl reference in the outage-diagnosis doc is historical structure only — use the table above.

1.2 Expected HTTP responses (unauthenticated, safe to call anytime)

EndpointExpectedMeaning
https://api.w3.pinbox24.com/api/i18n/langs2xxv32-prod backend answering
https://api.w4.pinbox24.com/api/i18n/langs2xxv42-prod backend answering
https://w3.pinbox24.com2xxfrontend served
https://w4.pinbox24.com2xxfrontend served
https://pdf-gen-api.w4.pinbox24.com/api/v1/pdf-gen (POST, sample payload)200 + PDF magic bytes %PDFChromium/Puppeteer pipeline functional (see docs/playbooks/pdf-gen-v42-prod-missing-chromium-libs.md if this fails)
502 on any of the abovenginx-proxy can’t reach the container (network split)→ outage-diagnosis Step 2
504 on any of the abovenginx-proxy reached the container but it hung→ outage-diagnosis Step 3

These are exactly the endpoints .github/workflows/health-check.yml (health-http job, steps w3 and w4) polls every 2 hours — running the same curl locally reproduces CI’s signal immediately instead of waiting for the next scheduled run.

1.3 DB connectivity expectations

  • Both v32-prod and v42-prod should show Mongoose connected (and, for W3, mongojs connected via crudService.js) in PM2 logs shortly after a (re)start, with no Mongoose disconnected loop.
  • rs.status().members on any rs0 node should report bms-2 PRIMARY/1, bms-3 SECONDARY/1, bms-4 ARBITER/1.
  • A working DB connection does not guarantee the app-layer secret is correct — see §2.3 for the JWT-level spot-check, since a stale-but-connectable Mongo credential and a stale JWT secret fail differently (connection succeeds, auth silently mismatches).

2. Quick verification checklist

Run this after any W3/W4 deploy, credential rotation, or incident remediation — before declaring it done. Every step here is read-only / non-destructive and never prints a secret value.

2.1 HTTP up-checks (30 seconds, no SSH needed)

# Both endpoints — 2xx is the only acceptable range
(Invoke-WebRequest "https://api.w3.pinbox24.com/api/i18n/langs" -TimeoutSec 10 -UseBasicParsing).StatusCode
(Invoke-WebRequest "https://api.w4.pinbox24.com/api/i18n/langs" -TimeoutSec 10 -UseBasicParsing).StatusCode
 
# Frontends
(Invoke-WebRequest "https://w3.pinbox24.com" -TimeoutSec 10 -UseBasicParsing).StatusCode
(Invoke-WebRequest "https://w4.pinbox24.com" -TimeoutSec 10 -UseBasicParsing).StatusCode

Bash equivalent (bms-1 or any host with egress — see w4-health-check-egress.md for why this must NOT be run from a Claude Code Remote sandbox):

curl -o /dev/null -sw "%{http_code}\n" https://api.w3.pinbox24.com/api/i18n/langs
curl -o /dev/null -sw "%{http_code}\n" https://api.w4.pinbox24.com/api/i18n/langs

2.2 Container status / uptime (SSH to bms-1, # PLAYBOOK: pinbox24-w3-w4-health-verification.md)

ssh root@94.23.26.113
docker ps --filter name=v32-prod --filter name=s3-v32-prod --filter name=v42-prod --format "{{.Names}}\t{{.Status}}"
docker exec v32-prod pm2 list
docker exec v42-prod pm2 list

Look for: 0 or low, non-climbing restart counts and uptime consistent with the last deploy (a fast-climbing restart count with sub-minute uptime means the process is crash-looping — go straight to §4).

NEVER grep docker logs output for mongo/mongodb/connect/connected unfiltered (issue #5209). Every Node/Mongoose app in this stack (W3 v32-prod/s3-v32-prod, W4 v42-prod, and siblings) logs the full connection string, including the embedded password, on every connect/reconnect — e.g. Mongoose connected to mongoDB server: mongodb://w3_app:<PASSWORD>@145.239.133.104,51.68.155.224/w3_db?.... The natural command to confirm DB connectivity is exactly what catches that line:

# NEVER — prints the live password to the session transcript
docker logs s3-v32-prod --tail 30 | grep -i mongo
docker logs v32-prod --tail 30 | grep -i connected
# Bare docker logs with no filter at all is even more exposed (dumps the whole log).
docker logs v32-prod

This is exactly how the live w3_app MongoDB password leaked on 2026-08-02/03 (#5209) — the same failure class as #5165 (docker inspect .Config.Env) and #4966 (pm2 env grep), via a third leak vector: application log output. Hook-enforced: both .claude/hooks/pre-bash-safety.sh and .claude/hooks/pre-bash-safety-windows.ps1 block docker logs <container> with no grep filter at all, and docker logs | grep for mongo/mongodb/connect(ed|ing) unless the pipeline also redacts the credential or the grep is count-only. Safe alternatives:

# Redact the credential out of the URI before it can print — safe regardless of search term.
docker logs v32-prod --tail 30 2>&1 | grep -i connected | sed -E 's#://[^@]+@#://[REDACTED]@#g'
# Count-only — never emits the matched line itself, safe even against a risky term.
docker logs v32-prod --tail 30 2>&1 | grep -c connected
# Prefer a search term that cannot match the connection-string line at all.
docker logs v32-prod --tail 30 2>&1 | grep -iE 'error|fail|crash'

Best option — avoid the log entirely. Use the container status / pm2 restart-count check above (§2.2) or the JWT/auth round trip below (§2.3) to confirm the app is healthy and using live credentials, without ever touching a line that could contain a connection string.

2.3 Safe JWT/auth spot-check — never print a credential value

Use this whenever a rotation claims to be done and you need to prove the app is actually using the new secret, not just that the file/SOPS says so. This is the exact method used in #5166 (documented in docs/secrets-rotation-log.md, row “2026-08-02 21:xx UTC — COMPLETED | 5165”):

a) Functional round trip — sign+verify inside the container, both cases:

# Positive case (loaded secret) — must succeed. Negative case (wrong secret) — must fail.
docker exec v32-prod node -e '
  const jwt = require("jsonwebtoken");
  const secret = process.env.JWT_TOKEN_SECRET;
  const tok = jwt.sign({probe:1}, secret);
  console.log("sign+verify (correct secret):", !!jwt.verify(tok, secret));
  try { jwt.verify(tok, "deliberately-wrong-secret"); console.log("FAIL: wrong secret verified"); }
  catch { console.log("OK: wrong secret correctly rejected"); }
'

Run the same against v42-prod for JWT_TOKEN_SECRET / V42_JWT_TOKEN_SECRET. This proves the app’s own crypto path works with whatever is actually loaded — never assume a file edit or SOPS write alone means the running process picked it up (see w3-w4-redeploy-idempotency.md §7.1 on why docker restart does not reload env).

b) Digest cross-check — confirm live deployed-file SOPS, without ever printing a value:

# On bms-1: hash the live container env value and the on-disk env file value
docker exec v32-prod sh -c "cat /proc/1/environ | tr '\0' '\n' | grep '^JWT_TOKEN_SECRET=' | sha256sum"
grep '^JWT_TOKEN_SECRET=' backend-environment.env | sha256sum
 
# On the SOPS host (bms-4 or windows-dev): hash the SOPS source of truth the same way
sops -d --input-type dotenv --output-type dotenv secrets/pinbox24-w3.env.sops \
  | grep '^V32_JWT_TOKEN_SECRET=' | sha256sum

All three digests must match. A mismatch means the container is running a stale credential — recreate it (docker compose up -d --force-recreate), never docker restart. Full method + prefix-mapping rules: w4-v42-jwt-exposure-verify-first.md (identical method documented for the W3 pm2 case in w3-pm2-jlist-exposure-verify-first.md).

2.4 Checklist summary — copy/paste for an incident comment

[ ] api.w3.pinbox24.com/api/i18n/langs -> 2xx
[ ] api.w4.pinbox24.com/api/i18n/langs -> 2xx
[ ] w3.pinbox24.com -> 2xx        [ ] w4.pinbox24.com -> 2xx
[ ] v32-prod pm2 list -> stable, low restart count
[ ] v42-prod pm2 list -> stable, low restart count
[ ] rs0 rs.status() -> bms-2 PRIMARY / bms-3 SECONDARY / bms-4 ARBITER, all health=1
[ ] (if a secret changed) JWT sign+verify round trip -> positive passes, negative fails
[ ] (if a secret changed) live == deployed-file == SOPS digest match
[ ] w3_reso_upload_pullback -> upload/S3-write/S3-read round trip OK, fixture cleaned up
[ ] w4_postbook_list -> success:true, >=1 result
[ ] w4_postbook_download -> success:true, non-empty base64
[ ] w4_inline_preview -> getSignedUrl 200 + GET (not HEAD) on the URL -> 200, Content-Disposition: inline

The last four rows are exactly what gh workflow run w3-w4-e2e-verification.yml checks — run it instead of reproducing them by hand unless you’re debugging a specific failure.

3. When this isn’t enough — escalate

  • Something above is actually broken (502/504, PM2 crash loop, Mongoose disconnected loop) → pinbox24-w3-w4-outage-diagnosis.md — full diagnosis flowchart + fixes A–D (RabbitMQ, MongoDB password mismatch, socket/reso image drift).
  • A fix looks like it worked, then came back after the next redeploy → w3-w4-redeploy-idempotency.md — the four-layer state model for what a redeploy actually restores, and why a fix applied outside git/SOPS/ansible does not survive.
  • A W3/W4 credential itself needs rotating (not just verifying) → w3-w4-rotation-orchestration.md (rotation SOP) — this playbook only verifies, it does not rotate.
  • The GH Actions health-check itself is failing (not a live incident, the CI job) → check .github/workflows/health-check.yml steps w3/w4 outputs and w4-health-check-egress.md for the egress-policy caveat if you’re reproducing the check from an agent sandbox rather than a host with real outbound egress.

Graph View

Backlinks