incident-2026-07-08-w4-signed-url-broken

Incident: W4 Inline PDF Preview — Investigation + Deploy-Resilience Fix (2026-07-08)

Pole	Wartość
Platforma	W4 (w4.pinbox24.com, bms-1)
Symptom	Inline PDF preview via signed URL reported broken for Ecotrans (w4)
Detected	2026-07-08
Status	RESOLVED — confirmed working by user after investigation. This PR is a preventive/deploy-resilience fix, not an active hotfix.
Related	PR #3227 (Wasabi key rotation), docs/playbooks/pinbox24-s3-wasabi-bms1.md §11 (2026-07-03/04 recurrence of the same symptom class)

Root cause of the 2026-07-08 symptom — actually already resolved by the time of investigation

Initial hypothesis (disproven — kept here for the record): that the automated secrets-sync.yml job sync-pinbox24-w4 never ran (it requires the bms-4 self-hosted runner, which was offline — secrets-sync #28907834298 queued, per docs/secrets-rotation-log.md), and that the manual SSH fallback only patched s3-environment.env, leaving v42-prod’s actual env_file (backend-environment.env) with the old, deleted access key.

Direct verification on bms-1 disproved this. SHA256 hashes (never the raw values) of s3Bucket_api_accessKeyId/secretAccessKey were compared across four sources: the SOPS file, both backend-environment.env and s3-environment.env on disk, and the live v42-prod process environment (docker exec ... printenv). All four hashes matched — the manual patch during today’s rotation actually did update both env files correctly, and backend-environment.env’s mtime (01:01:10 UTC) was ~12s before v42-prod’s container StartedAt (01:01:22 UTC), confirming the container picked up the new key on recreation. User confirmed inline preview works. So: no credential mismatch was present at investigation time — today’s manual rotation workaround was already complete and correct. The category of risk below (uncommitted, sometimes-hardcoded persistent-patch file; no CI/CD deploy step for it) is real and worth fixing preventively, but it was not the active cause of any outage at the time of this PR.

Second, independent problem found while investigating (the actual ask: “why doesn’t the fix survive a rebuild”)

uploadAwsS3.helper.js is volume-mounted into v42-prod from /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/persistent-patches/uploadAwsS3.helper.js on bms-1 disk — this file was never committed to this repo. Unlike its sibling postbookReport.helper.js (which is committed under infra-src/pinbox24/w4/persistent-patches/ but was, until this fix, never actually deployed by CI/CD either — see below), it existed only as server-side state edited by hand via docker cp/vi during past incidents. Three of its four functions (awsConf, uploadToAWSS3, appendToLogFile) still had hardcoded Wasabi credentials baked in from a much older version — a second latent bug: any future rotation invalidates those hardcoded values with no visibility in source control, no diff, no review, and no way to redeploy it consistently (this is the same failure class as INCIDENT-2026-07-01-002, RC-2 in pinbox24-s3-wasabi-bms1.md).

Separately, secrets-sync.yml’s sync-pinbox24-w4 job deploys docker-compose.yml from infra-src/pinbox24/w4/ on every run, but had no step at all that ships the persistent-patches/ directory — so even the one file that was committed (postbookReport.helper.js) was not actually kept in sync with the server by CI/CD.

Fix applied (this PR)

1. infra-src/pinbox24/w4/persistent-patches/uploadAwsS3.helper.js committed to source control, with all four functions now reading Wasabi credentials from process.env.s3Bucket_api_accessKeyId / process.env.s3Bucket_api_secretAccessKey (matching s3getSignedUrl, which already did this). No hardcoded credentials remain in this file.
2. .github/workflows/secrets-sync.yml — added a “Deploy W4 persistent-patches to bms-1” step in sync-pinbox24-w4, run before container recreation, so this directory is shipped from git on every sync (manual workflow_dispatch or automatic on SOPS push) instead of relying on manual docker cp.

With this in place, ordinary Wasabi key rotations only need the SOPS + env-file update (already automated) — the JS helper file no longer needs a manual re-patch after every rotation (Step 5 in pinbox24-s3-wasabi-bms1.md is now obsolete for this file).

3. bms-4 runner single point of failure — fixed. Added a shared secrets-deploy custom label to both bms4-p24infra and ionos-2 (via the GH API, no runner restart) and repointed sync-pinbox24-w3/sync-pinbox24-w4/sync-pinbox24-backends at [self-hosted, secrets-deploy]. These jobs are SOPS-decrypt + SCP/SSH orchestration only (the actual docker-compose work runs remotely on bms-1), matching the “secret sync” workload already approved for vps-i1 in docs/playbooks/gh-runner-assignment-policy.md. A bms-4 outage no longer stalls these deploys — GitHub Actions now dispatches to whichever of the two runners is online. Details: docs/playbooks/gh-runner-assignment-policy.md §secrets-deploy shared label.

Follow-ups / related open issues

• infra-src/pinbox24/w3/ has no persistent-patches/ directory in source control at all, despite pinbox24-s3-wasabi-bms1.md describing multiple server-side patches for w3 (local.js, controller.js, filesUpload.js) — same category of risk, not addressed here.
• The other secrets-sync.yml jobs (sync-vps-i1, sync-vps-h1, sync-bms-4, sync-brandpilot, etc.) are still pinned to [self-hosted, bms4] only. If any of them are also pure SOPS/SCP/SSH (no local heavy compute), the same secrets-deploy fallback could be extended to them — not done here to keep this PR scoped to the jobs that actually caused today’s incident.