p24-infra Docs

v41-prod-ecr-rebuild

5 min read

Playbook: v41-prod (W4 Angular frontend) — reproducible ECR build path

Source issue: #3733. Related: docs/plans/w3-w4-cicd-full-deployment.md (Gap 2), docs/playbooks/bms1-dr-plan.md, docs/playbooks/gitlab-runner-bms1-reregister.md.

What triggers this playbook

  • v41-prod on bms-1 is reported as an untagged local Docker image again (e.g. after a manual hotfix build that wasn’t pushed to ECR) and you need to restore a reproducible build path.
  • You need to rebuild/verify the v41-prod image from source (DR drill, dependency bump, security patch) rather than trust the single running container.
  • docker ps -a on bms-1 shows v41-prod with no repository/tag, only a bare image ID.

Background — what was found and fixed (2026-07-11)

  1. Source location: the Angular app is pinbox24/pinbox24-version-4 on GitLab (https://gitlab.com/pinbox24/pinbox24-version-4, project id 16151783). Confusingly named “p24-v-4.1” in the GitLab UI. master branch = production build source (BACKEND_SERVER_URL= https://api.w4.pinbox24.com), development branch = staging (v41-stage.dev.eat.pl).
  2. A working .gitlab-ci.yml already existed in the repo, with the exact build → push_image (ECR) → deploy pattern used for v42-prod (pinbox24/p24-back-ts). It last ran successfully on master on 2025-04-12 (pipeline 1765528506).
  3. Why it stopped working: the CI jobs are tagged autodeploy / prod-deploy-p24. The runners that originally served those tags for this project (a project-scoped runner + the old group runner “deploy using group runner - autodeploy”) are long offline/stale. A different, currently-live runner — bms-1-autodeploy (GitLab runner id 54112450, tags autodeploy + prod-deploy-p24, already serving pinbox24/p24-back-ts/v42-prod) — was simply never enabled for this project. Enabling it (POST /projects/16151783/runners {runner_id: 54112450}) was the entire fix needed to make the existing pipeline runnable again.
  4. The running container was less exposed than it looked: docker inspect v41-prod’s image digest (sha256:6ad6ea8614177128e9c6a5563fcc67941140d37bd57767d68af2eb331094fbc9) matched an image already present in ECR, pushed by that same 2025-04-12 pipeline run — just without a tag (the local :latest tag on bms-1 had since been reassigned to a different, older image by a manual rebuild, which is what made docker ps -a show it as “untagged”). The ECR repo has no lifecycle policy, so that untagged digest was not at risk of garbage collection — but it was fragile and undocumented. It is now pinned under the explicit tag v41-prod:pre-ci-fix-running-20260711 in ECR as an unambiguous reference for what’s currently live, in addition to the existing Wasabi backup (s3://p24-infra/backups/docker-images/v41-prod-backup-20260708.tar.gz).
  5. Safety change to the CI file: both deploy jobs (stage-front-end-deploy for development/staging, prod-front-end-deploy for master/production) were switched to when: manual. A push to master/development now automatically builds the Angular app and pushes a fresh image to ECR (repo 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v41-prod, tags :latest and :ci-<short-sha>) but will not replace the running container unless a human clicks “play” on the deploy job in the GitLab UI.

Confirm current state

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:GITLAB_ADMIN_PAT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops | Select-String "^GITLAB_ADMIN_PAT=").ToString().Split("=",2)[1]
$headers = @{ "PRIVATE-TOKEN" = $env:GITLAB_ADMIN_PAT }
# Latest pipelines
Invoke-RestMethod -Uri "https://gitlab.com/api/v4/projects/16151783/pipelines?ref=master&per_page=5" -Headers $headers | Select-Object id,status,created_at,web_url
$env:GITLAB_ADMIN_PAT = ""
# On bms-1 — running container + local images
docker ps -a | grep v41
docker images | grep v41-prod

Triggering a rebuild (build + push only — safe, no cutover)

Any push to master (production source) or development (staging source) in the GitLab repo auto-runs the front-end-build and docker-image-build stages on the bms-1-autodeploy runner. No manual trigger is normally needed; if you want to force one without a code change, use “Run pipeline” in the GitLab UI (Project → CI/CD → Pipelines → Run pipeline) on the branch you want.

This is always safe: it never touches the running v41-prod/v41-stage container. It only:

  • docker builds the Angular app (build-arg BACKEND_SERVER_URL baked in, no runtime secrets)
  • Tags the previous ECR :latest as :old-<image> (best-effort, || true)
  • Pushes the new image as :latest and :ci-<short-sha>

Verifying a new image before cutover — REQUIRED, do not skip

  1. Confirm the pipeline’s docker-image-build job succeeded and note the new tag (:ci-<short-sha>).
  2. On bms-1, pull and run the new image side-by-side on a free port, pointed at the same backend, and diff it against the current v41-prod container (a similar side-by-side rig already exists as v41-stage-front, built from the 2026-07-08 Wasabi backup — reuse that pattern): 
    # PLAYBOOK: v41-prod-ecr-rebuild.md
docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v41-prod:ci-<short-sha>
docker run -d -P --name v41-verify \
  -e CONTAINER_NAME=v41-verify -e NODE_ENV=production \
  -e BACKEND_SERVER_URL=https://api.w4.pinbox24.com \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com/v41-prod:ci-<short-sha>
docker port v41-verify
  3. Compare against the live site (https://w4.pinbox24.com) — page loads, login flow, a couple of core screens. Check the container logs for nginx errors.
  4. Only after this passes, proceed to cutover.

Cutover (human-approved, not automated by this playbook)

  1. In GitLab, open the pipeline that produced the verified image and click “play” on prod-front-end-deploy (master) — this pulls :latest from ECR, stops/removes the old v41-prod container, and starts a new one with the same env vars (VIRTUAL_HOST=w4.pinbox24.com, LETSENCRYPT_HOST=w4.pinbox24.com, BACKEND_SERVER_URL=https://api.w4.pinbox24.com).
  2. Confirm the new container is healthy: docker ps | grep v41-prod, curl -I https://w4.pinbox24.com.
  3. Clean up the verification container: docker rm -f v41-verify.

Rollback

  • The pre-cutover running image is pinned in ECR as v41-prod:pre-ci-fix-running-20260711 (or whatever explicit tag was current before your cutover — always re-pin before cutting over again in the future).
  • The Wasabi backup (s3://p24-infra/backups/docker-images/v41-prod-backup-20260708.tar.gz) is a second, independent fallback (docker load < backup.tar.gz).
  • To roll back: docker pull the pinned tag, stop/remove the bad container, docker run with the same env vars as above.

Escalation

If the bms-1-autodeploy runner goes offline again or jobs stay pending with no runner picking them up, see docs/playbooks/gitlab-runner-bms1-reregister.md. If ECR push fails with an auth error, the happy IAM user’s credentials on bms-1 (aws sts get-caller-identity) may need rotation — escalate via docs/playbooks/secret-manager-request.md (this is a legacy/external credential, not currently under p24-infra SOPS management — flag for a human-action decision on whether to bring it under our IAM).

Prevention

  • Any GitLab CI runner re-registration on bms-1 (see gitlab-runner-bms1-reregister.md) should check whether it needs to be re-enabled for pinbox24/pinbox24-version-4 (project id 16151783) alongside the other pinbox24 projects.
  • Consider adding an untagged-image detection check to the nightly infra health check (compare docker images output on bms-1 against expected repo:tag pairs) so this class of SPOF is caught automatically going forward — not yet implemented, tracked as a follow-up idea, not a blocking requirement of #3733.

Graph View

Backlinks