p24-infra Docs

w4-gitlab-ci-image-pipeline

7 min read

Playbook: W4 v42-prod GitLab CI image-build pipeline (Faza 3C)

Created: 2026-07-11 Source issue: radieu/p24-infra#3768 Plan: docs/plans/w3-w4-cicd-full-deployment.md — Faza 3C Artifact: docs/bms-1/gitlab-ci-w4.yml Compose source of truth: docs/bms-1/docker-compose-w4.yml

Human-execution required. This pipeline builds and deploys the live Pinbox24 W4 backend (v42-prod) on bms-1. p24-infra has push/MR rights but NO merge rights on pinbox24/p24-back-ts (see gitlab-repo-locations.md). An autonomous worker must never apply this to the GitLab repo — a human reviews and applies it.

What this is

docs/bms-1/gitlab-ci-w4.yml is the git-tracked source of truth for the .gitlab-ci.yml that should live in the pinbox24/p24-back-ts GitLab repo (default branch development). It mirrors the pattern established by docker-compose-w4.yml (#3542): land the artifact in p24-infra first, then wire it into the live pipeline as a separate, human-applied step.

Two stages:

StageJobTriggerWhat it does
buildbuild:v42-prodpush to development / manual web triggerBuilds the plain v42-prod app image, pushes :<merged-tag> + :latest to ECR, retags the previous :latest as :old-<date> for rollback
deploydeploy:v42-prodmanual (run in the nightly window)docker compose pull + up -d v42-prod, health-gates on :3000/health
deploydeploy:v42-prod:nightlyCI schedule (set to 20:00–06:00 UTC)Same deploy, hands-off, for the nightly window

All jobs run on the bms-1 shell runner — register the tag v42-bms1-autodeploy on it (same pattern as mailgun-bms1-autodeploy / s3-v2-bms1-autodeploy from Faza 1).

Design decisions honored (issue #3768 scope, 2026-07-11)

Gap 8 — patches are NOT baked into the image

The persistent patches stay volume-mounted at runtime, exactly as in docker-compose-w4.yml. Live mount state on bms-1 (reconfirmed 2026-08-09, #5992):

  • persistent-patches/postbookReport.helper.js → /app/dist/apps/postbook/postbookReport.helper.js
  • persistent-patches/uploadAwsS3.helper.js → /app/dist/globalHelpers/uploadAwsS3.helper.js
  • persistent-patches/socket.middleware.js → /app/dist/middleware/socket.middleware.js (#3966 socket-auth try/catch)
  • persistent-patches/app.js → /app/dist/app.js (#3407 follow-up — disables the GUS-API startup code)
  • ecosystem.config.js → /app/ecosystem.config.js
  • credentials.json → /app/credentials.json (SOPS-managed Google SA creds, #3737)

gus-api-regon-wsdl → /app/dist/wsdl mount was REMOVED (it never took effect — the real require() resolves from node_modules/@pobidowski/gus-api-regon/dist/wsdl/; dropped in the same change as the app.js GUS-code disable). It is no longer in docker-compose.yml.

uploadAwsS3.helper.js and socket.middleware.js are still load-bearing even though their fixes are merged into p24-back-ts development: the deployed v42-prod:latest ECR image’s baked dist/ still ships the pre-fix code (verified #5992). Do not drop these mounts on the strength of the merge alone — verify the baked image dist first. Full patch inventory + still-needed rationale: docs/pinbox24-w3-w4-architecture-spec.md §3.1.

The build:v42-prod job runs a plain docker build (no COPY of these files). This keeps the image reproducible and the patches auditable in the builder dir. Do not change this — it was an explicit decision on 2026-07-11.

Gap 5 — redis-v42 startup ordering

redis-v42 already has depends_on + a healthcheck in docker-compose-w4.yml. The deploy job uses docker compose up -d so compose enforces condition: service_healthy — no ordering logic lives in the pipeline. This prevents the RabbitMQ-consumer crash when v42-prod starts before redis-v42 (P1 Gap #6 in the plan).

D6 — WebSocket-safe restart window (nightly only, 20:00–06:00 UTC)

v42-prod serves WebSocket connections. A mid-day container recreate drops every live socket, so restarts are confined to the low-traffic window 20:00–06:00 UTC. This is a documented operational window, not an app-level graceful-drain — nothing fancier was requested (D6, 2026-07-11).

  • deploy:v42-prod is when: manual: the operator triggers it inside the window.
  • deploy:v42-prod:nightly runs on a CI schedule — set the schedule to fire within the window.
  • Do not run a deploy outside the window without an incident justification.

Out of scope

  • wkhtml-v42-prod Prometheus health check → tracked separately in #3739. Not duplicated here.

Prerequisites (one-time, before first apply)

  1. Runner tag v42-bms1-autodeploy registered on the bms-1 shell runner.
  2. AWS ECR creds as GitLab CI/CD variables (protected + masked), never in the file: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (region eu-central-1).
  3. docker-compose-w4.yml present on bms-1 at the builder path /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/docker-compose-w4.yml, synced from docs/bms-1/docker-compose-w4.yml. The autoheal step (#3611) keeps the env files it references populated from SOPS (pinbox24-w4.env.sops, pinbox24-backends.env.sops).
  4. CI schedule (optional, for deploy:v42-prod:nightly) created in the GitLab project set to a time inside 20:00–06:00 UTC.

Applying to GitLab — single-file PUT (NOT the Commits API)

The multi-action Commits API (POST /projects/:id/repository/commits) times out on this runner. Use the single-file Repository Files endpoint, which commits one file in one call. This is a credential operation (GITLAB_ADMIN_PAT) — a secret-manager / developer runs it, never an autonomous dev-issue worker.

Source file (2026-08-01, #4796): read GITLAB_ADMIN_PAT from secrets/pinbox24-gitlab.env.sops, not secrets/administration.env.sops. The canonical copy still lives in administration.env.sops (developer-only, single recipient — never decryptable on any server), but the bms-4 infra-task worker’s age key was never a recipient of that file, so every command below silently had no path to the token when run on bms-4. pinbox24-gitlab.env.sops is a narrow 2-recipient copy (developer + bms-4 worker key only) created specifically so this playbook’s commands work from bms-4. Both copies must be rotated together (see secret-rotation-access-matrix.md).

Recommended: PUT to a NEW branch, then open an MR (p24-infra has no merge rights — a human on the Pinbox24 side merges). Do not PUT directly to development.

# Read the token silently (never echo it). PROJECT_ID = URL-encoded path or numeric id.
PAT=$(sops -d --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/pinbox24-gitlab.env.sops | grep '^GITLAB_ADMIN_PAT=' | cut -d= -f2-)
PROJECT_ID="pinbox24%2Fp24-back-ts"       # URL-encoded namespace/project
FEATURE_BRANCH="faza-3c-v42-prod-ci"
 
# 1. Update .gitlab-ci.yml on a new branch (start_branch creates it from development).
#    --data-urlencode reads the file content without shell-escaping issues.
curl -sf -X PUT "https://gitlab.com/api/v4/projects/${PROJECT_ID}/repository/files/.gitlab-ci.yml" \
  -H "PRIVATE-TOKEN: ${PAT}" \
  --data-urlencode "branch=${FEATURE_BRANCH}" \
  --data-urlencode "start_branch=development" \
  --data-urlencode "commit_message=ci(faza-3c): v42-prod image build+deploy pipeline (radieu/p24-infra#3768)" \
  --data-urlencode "content@docs/bms-1/gitlab-ci-w4.yml"
# (If the branch already exists, drop start_branch. If .gitlab-ci.yml does not yet exist on the
#  branch, use POST instead of PUT to create it.)
 
# 2. Open a merge request for human review — never merge it yourself.
curl -sf -X POST "https://gitlab.com/api/v4/projects/${PROJECT_ID}/merge_requests" \
  -H "PRIVATE-TOKEN: ${PAT}" \
  --data-urlencode "source_branch=${FEATURE_BRANCH}" \
  --data-urlencode "target_branch=development" \
  --data-urlencode "title=Faza 3C: v42-prod CI image-build pipeline (radieu/p24-infra#3768)"
 
unset PAT
  • Token goes in the PRIVATE-TOKEN header only — never in the URL (leaks into history/logs).
  • Pass -c http.extraHeader=... style only for git, not needed for the REST calls above.

Before applying, diff against the live file — the artifact is proposed. Fetch and compare:

PAT=$(sops -d --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/pinbox24-gitlab.env.sops | grep '^GITLAB_ADMIN_PAT=' | cut -d= -f2-)
curl -sf -H "PRIVATE-TOKEN: ${PAT}" \
  "https://gitlab.com/api/v4/projects/pinbox24%2Fp24-back-ts/repository/files/.gitlab-ci.yml/raw?ref=development" \
  -o /tmp/live-gitlab-ci-w4.yml
unset PAT
diff -u /tmp/live-gitlab-ci-w4.yml docs/bms-1/gitlab-ci-w4.yml || true
# Reconcile job names / stages / runner tag / ECR repo path with reality before applying.

Rollback

The build job retags the previous :latest as :old-<date> before overwriting, so a rollback target always exists (v42-prod has no automatic old- tag — same gap as s3-v2 in pinbox24-ms-deploy.md).

# On bms-1 — repoint :latest at the last known-good old- tag, then redeploy in the nightly window.
docker tag "563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod:old-<date>" \
           "563740926945.dkr.ecr.eu-central-1.amazonaws.com/v42-prod:latest"
docker compose -f /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/docker-compose-w4.yml up -d v42-prod

GitLab pipeline history also records the exact image tag of every past deploy — use it to pin a rollback target.

Error reporting (repo standard)

If any apply/deploy step fails, report it — do not silently work around:

  1. Discord embed via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL.
  2. GitHub issue in radieu/p24-infra labelled bug.

Graph View

Backlinks