p24-infra Docs

pdf-gen-v42-prod-missing-chromium-libs

16 min read

Playbook: pdf-gen-v42-prod — Puppeteer Chromium fails to launch (missing shared libs)

Status: ✅ DURABLE FIX LANDED 2026-08-09 (#4849, closed) — this is no longer a live-patch-only recurring issue. Root cause of all 10 prior recurrences: docker-deploy-prod.sh:91 (docker-compose up -d wkhtml notify pdf-gen git-deploy redis s3-v2) reads the pdf-gen image ref from the on-disk, uncommitted docker-compose.yml on bms-1 — and pdf-gen had never been committed to git at all (git show HEAD:docker-compose.yml had 0 occurrences of “pdf-gen”), so any refresh of that working tree reverted to the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image. Fixed by: (1) live force-recreate of pdf-gen-v42-prod onto the maintained registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest image (ldd clean, live POST returns %PDF), and (2) pinbox24/p24-back-ts MR !812 (merged daa39866) committing the pdf-gen service block — pointed at the maintained image — to fix/v42-prod-compose-persistent-config (the branch bms-1 actually deploys from). Scope was limited to pdf-gen only; the other drifted services (s3-v2, redis, notify, git-deploy) are untouched and remain p24-infra#4568’s separate scope. If this regresses again it indicates a new mechanism — the previously-identified one is now closed by the git commit — re-verify with git show <ref>:docker-compose.yml | grep pdf-gen on the live deploy branch before re-applying the temporary live-patch below.

_Prior status, superseded: REGRESSED a TENTH time 2026-08-07 (#5876) — a container recreation at started=2026-08-07T15:37:48Z (still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image, restarts=0) again wiped the earlier 2026-08-07 patch, re-blanking all 15 Chromium libs (ldd → 15× not found). Caught by the 2-hourly health-check.yml pdfgen step (run 31177683829), which auto-filed #5876 (server-down / infra-check-fail / triage). A dev-issue worker (ai-dev-bms4-2) claimed it and correctly rerouted to infra-task (no in-repo code change fixes the live container). The infra-task worker on bms-4 (bms4-cw-1) re-applied the Chromium-libs live patch 2026-08-07 (direct SSH ssh -o IdentitiesOnly=yes -i ~/.ssh/id_bms1 root@94.23.26.113; ldd clean — 0 not found; headless chrome dumps DOM; live POST /api/v1/pdf-gen with Content-Type: application/json returns %PDF, 6597 bytes — the exact health-check.yml probe passes; fresh health-check.yml re-run 31214751665 triggered to confirm the alert clears). This is the ninth+ successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix (#5006 successor: repoint the live compose off the dead registry to registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest, gated on the #4568 compose-drift human decision) lands. Do not close 5006 or treat this file’s live patch as a permanent resolution.

_Prior recurrence: REGRESSED a NINTH time 2026-08-07 (#5850) — a container recreation at started=2026-08-07T07:36:37Z (still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image, restarts=0) again wiped the 2026-08-05 patch, re-blanking all 15 Chromium libs (ldd → 15× not found). Caught by the scheduled w3-w4-e2e-verification.yml runs 31076840462 (06:18Z) and 31152281678 (05:57Z): [FAIL] pdfgen: HTTP 200 but body is not a PDF (magic=b'Fail') (leading bytes of Failed to launch…), the other three checks green. Surfaced to a worker by the daily gha-failed-jobs-audit (#5850, failed-gh-actions); the sibling health-check.yml pdfgen alert is #5848. The infra-task worker on bms-4 (bms4-cw-1) re-applied the Chromium-libs live patch 2026-08-07 (ldd clean — 0 not found; headless chrome dumps DOM; fresh w3-w4-e2e-verification.yml re-run 31166110849 green — all four checks PASS, pdfgen: PDF generated OK (6298 bytes, %PDF magic)). This is the eighth+ successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix (#5006 successor: repoint the live compose off the dead registry to registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest, gated on the #4568 compose-drift human decision) lands. Do not close 5006 or treat this file’s live patch as a permanent resolution.

Env note (2026-08-07, #5850): on this run bms-4 → bms-1 SSH connected directly with ssh -o IdentitiesOnly=yes -i ~/.ssh/id_bms1 root@94.23.26.113 — the ~/.ssh/id_bms1 key was present (unlike the 2026-08-05 gap below) and direct access worked (no ProxyJump needed this time). If direct fails with a firewall/timeout, fall back to the bms-1 ProxyJump alias per the 2026-08-05 note; if id_bms1 is absent, ~/.ssh/vps_root_key also authenticates to bms-1.

Prior recurrence: REGRESSED an EIGHTH time 2026-08-05 (#5550) — a third container recreation the same day (running container started=2026-08-05T11:39:41Z, still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image, restarts=0) again wiped the earlier 2026-08-05 patch, re-blanking all 15 Chromium libs (ldd → not found). Caught by the 2-hourly health-check.yml pdfgen step (run 30999157368), which auto-filed #5550 (server-down / infra-check-fail). A dev-issue worker (bms4-cw-1) claimed it, verified the live Fail body, and correctly rerouted to infra-task (no in-repo code change fixes the live container). The infra-task worker on bms-4 (bms4-cw-1) re-applied the Chromium-libs live patch 2026-08-05 (ldd clean, all 15 libs resolved; headless chrome dumps DOM; live POST /api/v1/pdf-gen with Content-Type: application/json returns %PDF, 6597 bytes; fresh health-check.yml re-run 31042893256 triggered to confirm the alert clears). This now recurs THREE times in a single day — the recreation cadence is still accelerating, which further strengthens the case to prioritise the durable compose fix (#5006 successor: repoint the live compose off the dead registry to registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest — see “Corrected root cause” below). This is the seventh successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix lands. Do not close 5006 or treat this file’s live patch as a permanent resolution.

Env note (2026-08-05, #5550): bms-4 → bms-1 SSH is via the bms-1 alias in ~/.ssh/config (ProxyJump through vps-i1-jump, bms-1’s source-IP SSH firewall blocks bms-4 directly, #3591). The config’s IdentityFile ~/.ssh/id_bms1 was absent on the worker; the present key ~/.ssh/vps_root_key authenticates to bms-1, so this run used ssh -o IdentitiesOnly=yes -i ~/.ssh/vps_root_key bms-1. If a future run hits no such identity: …/id_bms1, use vps_root_key the same way (the playbook’s canonical ssh -i ~/.ssh/id_ed25519 root@94.23.26.113 form assumes a different, direct-access host).

Prior recurrence: REGRESSED a SEVENTH time 2026-08-05 (#5525 / #5501 / #5482) — a second container recreation the same day (running container started=2026-08-05T09:49:52Z, still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image, restarts=0) again wiped the earlier 2026-08-05 patch, re-blanking all 15 Chromium libs (ldd → not found). Caught by the scheduled w3-w4-e2e-verification.yml run 30980824591 (06:17Z): [FAIL] pdfgen: HTTP 200 but body is not a PDF (magic=b'Fail') (leading bytes of Failed to launch…), the other three checks green. Auto-filed #5501 (workflow) and #5525 (daily gha-failed-jobs audit); #5482 is the sibling health-check.yml pdfgen issue. The infra-task worker on bms-4 (bms4-cw-1) re-applied the Chromium-libs live patch 2026-08-05 (ldd clean; headless chrome dumps DOM; live POST /api/v1/pdf-gen with Content-Type: application/json returns %PDF; full w3-w4-e2e-verification.yml re-run 30995850189 green — all four checks PASS, pdfgen: PDF generated OK (6298 bytes, %PDF magic)). This now recurs twice in a single day — the recreation cadence is accelerating, which strengthens the case to prioritise the durable compose fix (#5006 successor: repoint the live compose off the dead registry to registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest — see “Corrected root cause” below). This is the sixth successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix lands. Do not close 5006 or treat this file’s live patch as a permanent resolution.

New this recurrence — the libs patch cannot keep pace with active deploy churn (2026-08-05). The patch applied ~09:58Z was reverted ~15 min later: a full p24-back-ts GitLab CI deploy refreshed the entire build dir at 10:03:40Z (/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/{docker-compose.yml,*-environment.env,persistent-patches/*} all re-touched) and recreated pdf-gen-v42-prod at 10:04:04Z (docker events: stop→destroy→create→start), blanking the libs again. Re-patched ~10:08Z (%PDF restored). So on a day with repeated p24-back-ts deploys, the archive.debian.org libs patch is whack-a-mole — each deploy wipes it within minutes. The only remedy that survives deploy churn is the durable image repoint (bake Chromium libs into the image by repointing the live compose off the dead private-registry.dev.pinbox24.com ref to the maintained registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest, #5006 successor). This is now the escalation-priority action, not the libs patch. Repoint is gated on the compose-drift / branch-strategy human decision (see #5006 / #4568 and “Corrected root cause” below) — do not do it unilaterally, but push for it.

Prior recurrence: REGRESSED a SIXTH time 2026-08-05 (#5434) — another container recreation (still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image, restarts=0) again wiped the 2026-08-04 live patch, re-blanking all 15 Chromium libs (ldd → not found). Caught by the 2-hourly health-check.yml pdfgen step (run 30948358920), which auto-filed #5434. The alert was first claimed by a dev-issue worker, correctly rerouted to infra-task, and the infra-task worker on bms-4 (bms4-cw-1) re-applied the Chromium-libs live patch 2026-08-05 (ldd clean, headless chrome launches; live POST /api/v1/pdf-gen with Content-Type: application/json returns %PDF, 6597 bytes; fresh health-check.yml run 30962553565 green). This is the fifth successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix (#5006 successor) lands. Do not close 5006 or treat this file’s live patch as a permanent resolution.

Probe gotcha (confirmed #5434): the endpoint requires Content-Type: application/json. A POST without it returns HTTP 200 + Protocol error (Page.navigate): Invalid parameters url (magic Prot, 77 bytes) even when Chromium is fully healthy — a probe artifact, not a service defect. The health-check.yml step and real p24-back-ts callers both send the header; a hand-rolled curl -d verification must add -H "Content-Type: application/json" or it will falsely read as broken.

Prior recurrence: REGRESSED a FIFTH time 2026-08-04 (#5296) — a container recreation at 2026-08-03T21:19:34Z (still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image) again wiped the 2026-08-03 live patch. Caught by the 2-hourly health-check.yml pdfgen step (run 30850678110, 20:32Z: ✗ pdf-gen-v42-prod: FAIL (HTTP 200, magic=Fail)), which auto-filed #5296. The alert was first claimed by a dev-issue worker, correctly rerouted to infra-task (no in-repo code change fixes the live container), and the infra-task worker on bms-4 re-applied the Chromium-libs live patch 2026-08-04 (ldd clean, headless chrome launches, live POST /api/v1/pdf-gen returns %PDF, 6597 bytes). This is the fourth successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix (#5006 successor) lands. Do not close 5006 or treat this file’s live patch as a permanent resolution.

Prior recurrence: REGRESSED a FOURTH time 2026-08-03 (#5264 / #5258) — a container recreation at 2026-08-03T01:48Z (still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image) again wiped the 2026-08-02 live patch. This time the w3-w4-e2e-verification.yml scheduled run (06:36Z, run 30790736268) caught it hard: its pdfgen check is no longer continue-on-error — the authenticated E2E POST returned [FAIL] pdfgen: HTTP 200 but body is not a PDF (magic=b'Fail') (the leading bytes of Failed to launch…) and the job’s “Fail job if E2E test failed” step failed the run, auto-filing #5258 (workflow) and #5264 (daily audit). The infra-task worker re-applied the Chromium-libs live patch 2026-08-03 (ldd clean, chrome headless launches, re-run 30805637818 green: [PASS] pdfgen: PDF generated OK (6298 bytes, %PDF magic)). This is the third successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix (#5006) lands. Do not close 5006 or treat this file’s live patch as a permanent resolution.

Prior recurrence: REGRESSED a THIRD time 2026-08-02 (#4849) — the predicted revert happened: a container recreation at 2026-08-02T10:37Z (still off the dead private-registry.dev.pinbox24.com/pdf-gen-v42-prod image) wiped the 2026-08-01 live patch, and the live POST /api/v1/pdf-gen began returning HTTP 200 with the Puppeteer crash text again instead of a %PDF. Note the pdfgen health-check step is continue-on-error, so the workflow-level conclusion stayed success — ground-truth the running container (ldd + a real end-to-end POST), never trust the green workflow badge alone. The infra-task worker re-applied the Chromium-libs live patch 2026-08-02 (ldd clean, live POST returns %PDF, 6.6 KB). This is the second successful proof that a scoped p24-back-ts deploy / container recreation reverts this patch — it is still only a temporary live patch and will revert again on the next recreation until the durable compose fix (#5006) lands. Do not close #4849 or treat this file’s live patch as a permanent resolution.

Prior recurrence: REGRESSED 2026-08-01 — the 2026-07-31 live patch was reverted by an unrelated p24-back-ts GitLab CI redeploy at 19:09 UTC that recreated the container from the dead-registry image (see “Corrected root cause” section below); re-patched + verified same day. Source issue: radieu/p24-infra #4849 (recurrence); originally filed alongside this playbook (see References) Related: docs/pdf-gen-v42-prod-operations.md, docs/runbook.md ## Alert: pdfgen section, docs/playbooks/w3-w4-redeploy-idempotency.md Section 8

Trigger

et-lager / W4 frontend reports downloaded PDFs that won’t open, or et-lager UI shows a “file” that is actually plain text starting with Failed to launch the browser process!. Symptom traces back to a specific Pinbox24 register/process/task combination (protocol PDF generation, generate_pdf_protocol task type, template with typeOfGenerator: 'puppeter').

Root cause

pdf-gen-v42-prod (container on bms-1, pdf-gen-api.w4.pinbox24.com) runs Puppeteer ^5.0.0 on a Debian 9 “stretch” base image (EOL since 2022-06-30). The bundled Chromium binary (/app/node_modules/puppeteer/.local-chromium/linux-756035/chrome-linux/chrome) fails to launch — missing shared libraries confirmed via ldd:

libX11-xcb.so.1, libxcb-dri3.so.0, libXtst.so.6, libnss3.so, libnssutil3.so, libsmime3.so,
libnspr4.so, libatk-bridge-2.0.so.0, libXss.so.1, libdrm.so.2, libgbm.so.1, libasound.so.2,
libatspi.so.0, libgtk-3.so.0, libgdk-3.so.0

The Express app’s error handling for this failure returns HTTP 200 with the raw crash text as the body instead of a 5xx — and p24-back-ts’s reportGeneration.helper.ts (getPdfBase64V2 / generateReportFromRecId) does not validate the response is a real PDF before base64-encoding it, uploading to S3, and returning it to the client. That’s why the “PDF” downloads but won’t open.

Two separate defects, two separate owners:

  1. Missing libs on pdf-gen-v42-prod’s image — p24-infra can fix live (this playbook).
  2. p24-back-ts (pinbox24/p24-back-ts, GitLab) has no response validation and the pdf-gen Express app silently succeeds on failure — needs a code fix in a repo p24-infra does not own (push + MR only, human merges — see docs/w3-w4-stack-operations.md).

Confirm (read-only)

# Container status + PM2 boot logs
ssh -i ~/.ssh/id_ed25519 root@94.23.26.113 "docker inspect pdf-gen-v42-prod --format '{{.State.Status}} restarts={{.RestartCount}}'; docker logs pdf-gen-v42-prod --tail 50" # PLAYBOOK: pdf-gen-v42-prod-missing-chromium-libs.md
 
# App-level error log (PM2 writes to a file, not docker stdout)
ssh -i ~/.ssh/id_ed25519 root@94.23.26.113 "docker exec pdf-gen-v42-prod pm2 logs pdf-gen-backend --lines 200 --nostream" # PLAYBOOK: pdf-gen-v42-prod-missing-chromium-libs.md
# Look for: "Error: Failed to launch the browser process!" repeated at request timestamps.
 
# Confirm missing libs
ssh -i ~/.ssh/id_ed25519 root@94.23.26.113 "docker exec pdf-gen-v42-prod ldd /app/node_modules/puppeteer/.local-chromium/linux-756035/chrome-linux/chrome 2>&1 | grep 'not found'" # PLAYBOOK: pdf-gen-v42-prod-missing-chromium-libs.md

Fix — live patch (temporary, lost on container recreation)

Debian 9 stretch is EOL — deb.debian.org no longer serves it, must point at archive.debian.org and disable Valid-Until checking (archived Release files are stale by design).

ssh -i ~/.ssh/id_ed25519 root@94.23.26.113 "docker exec pdf-gen-v42-prod bash -c '\
echo \"deb http://archive.debian.org/debian stretch main\" > /etc/apt/sources.list && \
echo \"deb http://archive.debian.org/debian-security stretch/updates main\" >> /etc/apt/sources.list && \
apt-get -o Acquire::Check-Valid-Until=false update && \
apt-get install -y libx11-xcb1 libxcb-dri3-0 libxtst6 libnss3 libatk-bridge2.0-0 libxss1 \
  libdrm2 libgbm1 libasound2 libatspi2.0-0 libgtk-3-0 libgdk-pixbuf2.0-0 libxrandr2 \
  libxcomposite1 libxdamage1 libxfixes3 libxi6 libcups2 libpango-1.0-0 libpangocairo-1.0-0 \
  fonts-liberation ca-certificates'" # PLAYBOOK: pdf-gen-v42-prod-missing-chromium-libs.md

Verify:

ssh -i ~/.ssh/id_ed25519 root@94.23.26.113 "docker exec pdf-gen-v42-prod ldd /app/node_modules/puppeteer/.local-chromium/linux-756035/chrome-linux/chrome 2>&1 | grep 'not found'" # PLAYBOOK: pdf-gen-v42-prod-missing-chromium-libs.md
# Expect: no output (nothing "not found")
 
ssh -i ~/.ssh/id_ed25519 root@94.23.26.113 "docker exec pdf-gen-v42-prod /app/node_modules/puppeteer/.local-chromium/linux-756035/chrome-linux/chrome --headless --disable-gpu --dump-dom about:blank 2>&1 | tail -5" # PLAYBOOK: pdf-gen-v42-prod-missing-chromium-libs.md

This does NOT survive a container recreation (redeploy pulls the same broken image). It only buys time until the permanent fix below lands.

Permanent fix (requires repo access p24-infra doesn’t have)

  1. Locate the actual source repo for the pdf-gen-v42-prod image (private-registry.dev.pinbox24.com/pdf-gen-v42-prod) — it is not in pinbox24/p24-back-ts (that repo’s Puppeteer-free root Dockerfile builds a different, Alpine-based service). Likely a separate small Express/Puppeteer repo — ask the W4 dev team which repo owns it.
  2. In that repo’s Dockerfile: switch the base image off Debian 9 stretch (EOL) — either add the standard Puppeteer Debian dependency apt-get install block against a maintained Debian release, or move to Puppeteer’s own recommended base / puppeteer-core + a maintained Chromium build.
  3. Fix the Express error handler so a Puppeteer launch failure returns a non-2xx status instead of 200 with the crash text as the body.
  4. In pinbox24/p24-back-ts (src/apps/reportGeneration/reportGeneration.helper.ts, getPdfBase64V2): validate the response starts with the PDF magic bytes (%PDF-) before treating it as pdfBase64 and uploading to S3 — reject/retry/error otherwise instead of silently propagating garbage.

Corrected root cause — the compose fix is NOT a simple line-159 MR (2026-08-01, #5006)

#4792 concluded the revert-source was docs/bms-1/docker-compose-w4.yml (fixed by PR #4844); that was the wrong file. #5006 then said the fix belonged at p24-back-ts/docker-compose.yml ~line 159 — a plausible next guess, but ground-truthing the live bms-1 state proves the fix is one layer deeper still. Do not open a repo MR editing docker-compose.yml line 159 and declare victory — it will not reach production. Verified on bms-1 (94.23.26.113, key ~/.ssh/id_bms1) 2026-08-01:

  • docker inspect pdf-gen-v42-prod → image=private-registry.dev.pinbox24.com/pdf-gen-v42-prod, project=p24-back-ts, service=pdf-gen, config_files=docker-compose.yml, working_dir=/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts.
  • That build dir is on branch fix/v42-prod-compose-persistent-config (not development), and docker-compose.yml shows M (uncommitted working-tree modification).
  • The committed docker-compose.yml on both development and fix/v42-prod-compose-persistent-config is the short (~57-line) backend/s3/wkhtml-only file — it does not contain the pdf-gen service at all. The full 198-line prod compose (with pdf-gen at line 159, plus notify/git-deploy/redis/s3-v2) exists only as uncommitted local drift on the runner, committed to no branch.
  • The committed deploy script docker-deploy-prod.sh runs docker-compose up -d ... pdf-gen ..., so the drift is load-bearing: a clean CI checkout would drop pdf-gen from the compose and the || true would silently leave it unmanaged.
  • The maintained replacement image exists and is fresh: GitLab project pinbox24/p24-ms-pdfgen (id 19942067), registry repo registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest (~592 MB, built 2026-08-01 09:22 UTC).

Durable fix requires a human/architect decision (two layers):

  1. Repoint the live compose — on bms-1 edit line 159 of /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/docker-compose.yml: private-registry.dev.pinbox24.com/pdf-gen-v42-prod → registry.gitlab.com/pinbox24/p24-ms-pdfgen/pdf-gen-v42-prod:latest, then docker-compose pull pdf-gen && docker-compose up -d --force-recreate pdf-gen. This survives the next recreation, but only as long as the working-tree drift itself survives (no clean checkout).
  2. Eliminate the drift — commit the full prod compose (with the corrected image ref) to the branch prod actually deploys from (fix/v42-prod-compose-persistent-config), so a future clean checkout is correct. This is a large diff + a branch-strategy question (why prod deploys from a fix/* branch and why the compose was never committed) → human/architect review, human-triggered prod deploy.

The response-validation half (getPdfBase64V2 %PDF- check) is committable normally and shipped as pinbox24/p24-back-ts MR !796 (2026-08-01, #5006).

Audit log

source /opt/p24-infra/scripts/lib/log_op.sh
log_op "claude" "config_change" "pdf-gen-v42-prod" "success" \
  "Installed missing Chromium runtime libs (libX11-xcb + 14 others) via archive.debian.org; temporary fix, container recreation will revert" "bms-1" <issue-number>

References

Graph View

Backlinks