p24-infra Docs

w3-gitlab-ci-image-pipeline

7 min read

Playbook: W3 v32-prod GitLab CI image-build pipeline (Faza 3D)

Created: 2026-07-11 Source issue: radieu/p24-infra#3769 Plan: docs/plans/w3-w4-cicd-full-deployment.md — Faza 3D Artifact: docs/bms-1/gitlab-ci-w3.yml Compose source of truth: docs/bms-1/docker-compose-w3.yml W4 counterpart: docs/playbooks/w4-gitlab-ci-image-pipeline.md

Human-execution required. This pipeline builds and deploys the live Pinbox24 W3 backend (v32-prod) on bms-1. p24-infra has push/MR rights but NO merge rights on pinbox24/p24-v-3.2 (see gitlab-repo-locations.md). An autonomous worker must never apply this to the GitLab repo — a human reviews and applies it.

What this is

docs/bms-1/gitlab-ci-w3.yml is the git-tracked source of truth for the .gitlab-ci.yml that should live in the pinbox24/p24-v-3.2 GitLab repo. It mirrors the pattern established for W4 by docs/bms-1/gitlab-ci-w4.yml (#3768): land the artifact in p24-infra first, then wire it into the live pipeline as a separate, human-applied step.

Four jobs across two stages:

StageJobTriggerWhat it does
buildbuild:v32-prodpush to W3 default branch / manual web triggerBuilds the plain v32-prod app image (no patch baking), pushes :<merged-tag> + :latest to ECR, retags old :latest as :old-<date> for rollback
deploydeploy:v32-prodmanual (run in nightly window)docker compose pull + up -d v32-prod, health-gates on api.w3.pinbox24.com/api/i18n/langs
deploydeploy:v32-prod:nightlyCI schedule (20:00–06:00 UTC)Same as above, hands-off
deploydeploy:v32-prod-reso-socket:nightlyCI schedule (20:00–06:00 UTC)Force-recreates v32-prod-reso + v32-prod-socket in the nightly window (picks up env-file refresh; images from private-registry, not ECR)

All jobs run on the bms-1 shell runner — register the tag v32-bms1-autodeploy on it (same pattern as mailgun-bms1-autodeploy / s3-v2-bms1-autodeploy from Faza 1, and v42-bms1-autodeploy from Faza 3C / #3768).

Design decisions honored (issue #3769 scope, 2026-07-11)

Gap 8 — patches are NOT baked into the image

The persistent patches stay volume-mounted at runtime, exactly as in docker-compose-w3.yml:

Patch fileContainer path
persistent-patches/pushNotification.helper.js/app/app-backend/helperFunction/pushNotification.helper.js
persistent-patches/mongoose.js/app/app-backend/config/mongoose.js
persistent-patches/crudService.js/app/app-backend/helperFunction/crudService.js
persistent-patches/filesUpload.js/app/app-backend/controllers/filesUpload.js
persistent-patches/twilioService.js/app/app-backend/services/twilioService.js

All paths are under BUILDER_DIR (/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2). The build:v32-prod job runs a plain docker build (no COPY of these files into the image). Do not change this — it is an explicit decision matching the W4 Gap 8 resolution (2026-07-11).

excel_import.js is a 6th mount but NOT a patch (#4708, verified read-only on bms-1 2026-07-30). docker-compose-w3.yml mounts a 6th file — app-backend/helperFunction/processHelper/tasksService/tasks/excel_import.js → the same path in the container. It used to be listed in the table above, which was wrong twice over: it is a stock, git-tracked file in pinbox24/p24-v-3.2 (tracked + clean, last touched by upstream commit 5f469287, 2020-02-01) rather than a p24-infra-managed hotfix, and it is present in the built image — build-dir copy, mounted copy and image copy all hash identically (md5 659f133a695fb3688e8e6f2b78e61d24), so its mount is a redundant no-op. Gap 8 therefore covers 5 patch files, not 6. Keep the mount (harmless, and dropping it is a live-config change nobody needs), but do not treat it as something the pipeline must preserve or restore: a CI git clean removes only untracked files, and a fresh checkout brings it back regardless. See docs/pinbox24/w3-persistent-patches-committed-2026-07-08.md §Follow-ups.

v32-prod-reso / v32-prod-socket — nightly-window force-recreate

Issue #3740 (WebSocket usage analysis) was resolved 2026-07-11. Verdict: real but marginal usage (1 active client, backend already 502s 99.6% of the time). Recommendation: nightly-window (20:00–06:00 UTC) force-recreate, rather than the indefinite no-restart exemption from PR #3694.

These containers use private-registry.dev.pinbox24.com/v32-prod (not ECR). The nightly deploy:v32-prod-reso-socket:nightly job does docker compose up -d without pulling a new image — the purpose is env-file refresh (autoheal step #3611) in a low-traffic window. It does NOT build or push a new image for these containers.

Nightly restart window (20:00–06:00 UTC)

v32-prod serves WebSocket connections. Restarts are confined to the low-traffic window. This is a documented operational window, not an app-level graceful-drain.

  • deploy:v32-prod is when: manual — the operator runs it inside the window.
  • deploy:v32-prod:nightly and deploy:v32-prod-reso-socket:nightly run on a CI schedule — set the schedule to fire inside the window.
  • Do not run a deploy outside the window without an incident justification.

Rollback policy (plan §Faza 3D)

If the pipeline fails after 4+ attempts: leave on the current image and document as tech-debt. Do not over-engineer retry logic.

The build job retags the previous :latest as :old-<date> before overwriting, so a manual rollback target always exists. GitLab pipeline history records the exact image tag of every past deploy.

Out of scope

  • s3-v32-prod / s3-v32-prod-reso / s3-v32-prod-socket — separate build chain (old-s3 image, different Dockerfile/repo); not included in this pipeline.
  • cron-v32-prod* — static images from private registry, no CI build needed here.
  • Credential rotation — Faza 4, tracked separately.
  • W3 frontend image — tracked in #3736.

Prerequisites (one-time, before first apply)

  1. Runner tag v32-bms1-autodeploy registered on the bms-1 shell runner (ID pattern: same runner as mailgun-bms1-autodeploy / s3-v2-bms1-autodeploy).

  2. AWS ECR credentials as GitLab CI/CD variables (protected + masked), never in the file:

    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY Region: eu-central-1. These are the same ECR credentials used for W4 — verify that they are already set in the p24-v-3.2 project, or copy them from p24-back-ts.

  3. docker-compose-w3.yml deployed to bms-1 at the path /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/docker-compose-w3.yml. Sync from docs/bms-1/docker-compose-w3.yml. The autoheal step (#3611) keeps the env files it references populated from SOPS (pinbox24-w3.env.sops).

  4. Verify W3 default branch in the pinbox24/p24-v-3.2 GitLab project settings. The artifact uses development — adjust the build:v32-prod and deploy:v32-prod rules if the default branch is master or another name.

  5. CI schedules (for the two nightly jobs): create two schedules in the GitLab project, each set to a time inside 20:00–06:00 UTC. Both deploy:v32-prod:nightly and deploy:v32-prod-reso-socket:nightly will fire on any CI_PIPELINE_SOURCE == "schedule" trigger — a single schedule is sufficient if you want both to run together.

Pre-apply: diff against the live file

The artifact is proposed. Fetch the live .gitlab-ci.yml and diff before applying — the existing file contains prod-back-end-deploy, autoheal, and other jobs that must be preserved or merged.

# Read the token silently — never echo it.
PAT=$(sops -d --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/administration.env.sops | grep '^GITLAB_ADMIN_PAT=' | cut -d= -f2-)
PROJECT_ID="pinbox24%2Fp24-v-3.2"
 
curl -sf -H "PRIVATE-TOKEN: ${PAT}" \
  "https://gitlab.com/api/v4/projects/${PROJECT_ID}/repository/files/.gitlab-ci.yml/raw?ref=development" \
  -o /tmp/live-gitlab-ci-w3.yml
unset PAT
 
diff -u /tmp/live-gitlab-ci-w3.yml docs/bms-1/gitlab-ci-w3.yml || true
# Reconcile: job names, stages, runner tag, ECR repo path, branch names.
# Merge the new jobs into the existing file rather than replacing wholesale.

Applying to GitLab — single-file PUT (NOT the Commits API)

The multi-action Commits API (POST /projects/:id/repository/commits) times out on this runner. Use the single-file Repository Files endpoint. This is a credential operation (GITLAB_ADMIN_PAT from secrets/administration.env.sops, developer-only) — run it manually, never in an autonomous dev-issue worker.

Recommended: PUT to a NEW branch, then open an MR (p24-infra has no merge rights on pinbox24/p24-v-3.2 — a human on the Pinbox24 side merges).

PAT=$(sops -d --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/administration.env.sops | grep '^GITLAB_ADMIN_PAT=' | cut -d= -f2-)
PROJECT_ID="pinbox24%2Fp24-v-3.2"
FEATURE_BRANCH="faza-3d-v32-prod-ci"
 
# 1. Create/update .gitlab-ci.yml on a new branch.
#    If .gitlab-ci.yml does not yet exist on the branch, use POST instead of PUT.
#    start_branch creates the feature branch from "development" if it doesn't exist yet.
curl -sf -X PUT "https://gitlab.com/api/v4/projects/${PROJECT_ID}/repository/files/.gitlab-ci.yml" \
  -H "PRIVATE-TOKEN: ${PAT}" \
  --data-urlencode "branch=${FEATURE_BRANCH}" \
  --data-urlencode "start_branch=development" \
  --data-urlencode "commit_message=ci(faza-3d): v32-prod image build+deploy pipeline (radieu/p24-infra#3769)" \
  --data-urlencode "content@docs/bms-1/gitlab-ci-w3.yml"
 
# 2. Open an MR for human review — never merge it yourself.
curl -sf -X POST "https://gitlab.com/api/v4/projects/${PROJECT_ID}/merge_requests" \
  -H "PRIVATE-TOKEN: ${PAT}" \
  --data-urlencode "source_branch=${FEATURE_BRANCH}" \
  --data-urlencode "target_branch=development" \
  --data-urlencode "title=Faza 3D: v32-prod CI image-build pipeline (radieu/p24-infra#3769)"
 
unset PAT
  • Token goes in the PRIVATE-TOKEN header only — never in the URL (leaks into history/logs).
  • If the branch already exists, drop start_branch from the PUT call.

Rollback

# On bms-1 — repoint :latest at the last known-good old- tag, then redeploy in the nightly window.
docker tag "563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod:old-<date>" \
           "563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod:latest"
docker compose -f /home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/docker-compose-w3.yml \
  up -d v32-prod

GitLab pipeline history records the exact merged-<date>-<sha> tag for every past build — use it to pin a rollback target with docker tag ECR_REGISTRY/v32-prod:<tag> ECR_REGISTRY/v32-prod:latest.

Error reporting (repo standard)

If any apply/deploy step fails:

  1. Discord embed via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL.
  2. GitHub issue in radieu/p24-infra labelled bug.

Graph View

Backlinks