p24-infra Docs

pinbox24-s3v2-master-branch-autodeploy

7 min read

Playbook: pinbox24-ms-s3-v2 master branch auto-deploys to production on merge

Discovered: 2026-08-06, while executing the async-bugs half of p24-infra#2048. Merging a verified, conflict-free MR into pinbox24-ms-s3-v2’s master branch triggered an unscheduled production deploy of s3-v2-v42-prod on bms-1, mid-week — the exact outcome the task’s safety requirement (“no forced live container recreate; defer to the coordinated window, e.g. #4481”) was trying to avoid.

Trigger

Any merge (or direct push) to master in gitlab.com/pinbox24/pinbox24-ms-s3-v2.

Root cause

This repo’s .gitlab-ci.yml defines prod-back-end-deploy (stage deploy) with:

prod-back-end-deploy:
  stage: deploy
  only:
    - master
  when: on_success
  tags:
    - prod-deploy-p24

when: on_success (the GitLab default when the key is omitted, but stated explicitly here) means this job runs automatically the moment the preceding build/push_image stage jobs succeed — no manual approval gate. It stops, removes, and recreates the live s3-v2-v42-prod container on bms-1 with the freshly-built image. Unlike most Pinbox24 GitLab-merge situations in this ecosystem (where “merge is autonomous” refers only to the code review/merge step, and a separate deploy step is still manual or explicitly scheduled), this specific repo’s master collapses merge and deploy into one action.

The sibling feature/scanque branch’s pipeline does not have this behavior — its deploy stage job is when: manual (confirmed via GET /projects/:id/pipelines/:id/jobs after a merge: status manual, not auto-triggered). So merging into feature/scanque is safe from this specific gotcha; merging into master is not.

How to confirm before merging (mandatory pre-merge check for this repo, master target only)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$decrypted = sops --decrypt --input-type dotenv --output-type dotenv secrets\pinbox24-gitlab.env.sops
$line = $decrypted | Select-String "^GITLAB_ADMIN_PAT="
$env:GITLAB_ADMIN_PAT = $line.Line.Split("=",2)[1]
 
$proj = "pinbox24%2Fpinbox24-ms-s3-v2"
curl.exe -s -H "PRIVATE-TOKEN: $env:GITLAB_ADMIN_PAT" `
  "https://gitlab.com/api/v4/projects/$proj/repository/files/.gitlab-ci.yml/raw?ref=master" `
  | Select-String -Pattern "when:\s*on_success" -Context 5,0
 
$env:GITLAB_ADMIN_PAT = ""

If any deploy-stage job for master shows when: on_success (or no when: key at all, which defaults to on_success) and only: [master], merging to master will deploy immediately.

If you need the code merged but NOT deployed yet (the common case — coordinated-window fixes)

  1. Merge into the non-production branch first (feature/scanque here, or whatever the repo’s non-auto-deploy branch is) so the fix is preserved and reviewable without touching prod.
  2. For the master MR: either
    • hold the merge until the actual deploy window (e.g. #4481), and merge master then — the deploy will happen as an intended side effect of that merge, or
    • merge immediately if you accept the deploy risk is negligible (e.g. the change is additive/ safe and a few seconds of container restart is acceptable) — but say so explicitly in the task’s safety assessment, don’t merge on autopilot assuming “merge = safe, deploy = separate step” the way it works for most other Pinbox24 repos.
  3. If a master merge is submitted before this is checked and a deploy pipeline appears with a pending deploy-stage job: cancel it immediately — 
    curl.exe -s -H "PRIVATE-TOKEN: $env:GITLAB_ADMIN_PAT" -X POST "$base/jobs/<job_id>/cancel"
curl.exe -s -H "PRIVATE-TOKEN: $env:GITLAB_ADMIN_PAT" -X POST "$base/pipelines/<pipeline_id>/cancel"
    Race condition warning: the prod-back-end-deploy job goes from pending to success in as little as ~30 seconds once a runner (s3-v2-bms1-autodeploy, always-on shell executor on bms-1) picks it up — a status check immediately after the merge call can still show pending while the cancel request arrives after the job has already completed. There is no reliable window to intercept it via polling; the only sure prevention is the pre-merge .gitlab-ci.yml check above, done before merging.

What actually happened on 2026-08-06 (for reference)

  • MR pinbox24-ms-s3-v2!20 (fix/2048-mailgun-async-bugs-master → master) merged clean, no conflicts, verified byte-exact against the intended fix content before merge.
  • Pipeline #2738081612 ran back-end-build → docker-image-build → prod-back-end-deploy automatically; the deploy job (id 15753390454) completed in 30s (s3-v2-v42-prod stop → rm → run → confirmed Up 10 seconds) before a cancel request (issued as soon as the unexpected pending deploy-stage job was noticed) could take effect.
  • Outcome: the container came back healthy and now runs the intended fix — but the timing was outside the coordinated window the task required, which is the actual violation (an unhealthy redeploy would have been strictly worse, but “it happened to work” is not the same as “it was supposed to happen now”). Reported as a self-flagged incident rather than treated as a no-op because the intent (no mid-week prod touch) was not honored, independent of outcome.
  • The feature/scanque MR (!19) merge in the same session did not auto-deploy (its deploy job stayed manual), confirming the divergence is real and branch-specific, not a fluke.

Prevention

  • Treat “check the target repo’s own .gitlab-ci.yml for auto-deploy triggers on the target branch” as a mandatory step before any GitLab MR merge, ecosystem-wide, not just for this repo — the assumption “merging a Pinbox24 MR is always safe, deploy is a separate manual step” (which holds for several other Pinbox24 repos/branches) does not generalize.
  • Prefer merging into the non-auto-deploy branch (feature/scanque for this repo) whenever the task’s own safety requirement calls for a code-only change with deploy deferred.

Ecosystem audit — which OTHER pinbox24/* repos couple merge⇒deploy (#5762 follow-up)

The #5762 incident report asked, as an open question, whether other pinbox24/* repos share this coupled-merge⇒auto-deploy pattern. They do — s3-v2 is not unique, but it IS the anomaly among backend repos. Audited 2026-08-06 (infra-task worker, read-only via GitLab API GET /projects/:id/repository/files/.gitlab-ci.yml/raw, all 43 pinbox24/* projects, .gitlab-ci.yml fetched per candidate branch — master/main/development/default — because the file differs by branch, which is the whole trap). 27 repos have no .gitlab-ci.yml at all; the rest break down below.

The convention (verified from the actual job definitions)

Job-name patternonly:when:Runner tagDeploys to
stage-*-deploy[development]on_success (auto)autodeploystaging
prod-*-deploy[master]manual (gated)prod-deploy-p24production

So for a well-behaved backend repo, merging master does not auto-deploy prod — the prod job waits for a manual click. pinbox24-ms-s3-v2 breaks this: its prod-back-end-deploy (master) is when: on_success, not when: manual — the single-word deviation that caused the incident. Job name is not a reliable signal (mailgun’s prod job is just called deploy; p24-ext-pages uses front-end-deploy scoped to master). Only the when: key on the branch-scoped deploy job is authoritative — manual = safe, on_success/absent = auto-deploys on merge.

Repos where merging a branch AUTO-deploys to PRODUCTION (the danger class — check these before any merge)

RepoBranchDeploy jobwhen:Runner tagProd target
pinbox24-ms-s3-v2masterprod-back-end-deployon_successprod-deploy-p24s3-v2-v42-prod (bms-1) — the #5762 incident
p24-ms-mailgunmasterdeployon_success (default)mailgun-bms1-autodeploymailgun-v42-prod (bms-1) — live W4 email microservice
p24-ext-pagesmaster and developmentfront-end-deployon_success (default)proddeployfrontend (live-serving status unverified)
fuse2masterprod-front-end-deployon_success (default)prod-deploy-p24frontend (live-serving status unverified)
p24-aotmasterprod-front-end-deployon_success (default)prod-deploy-p24frontend (live-serving status unverified)

p24-ms-mailgun is the operationally significant sibling of the incident repo: it is an active W4 production microservice on bms-1 with a dedicated always-on autodeploy runner, so merging its master recreates mailgun-v42-prod immediately — treat it with the same caution as s3-v2’s master. The three frontends (p24-ext-pages, fuse2, p24-aot) carry the coupled config too; their current live-serving status was not verified in this pass — treat them as coupled until proven otherwise.

Repos that are SAFE on master (prod deploy is when: manual) — but auto-deploy STAGING on development

p24-back-ts (W4 backend), p24-v-3.2 (W3 backend), p24-ms-pdfgen, p24-ms-notification: prod job (prod-back-end-deploy, master) is when: manual. Merging master does not auto-deploy prod. Their development→staging job (stage-back-end-deploy, tags:[autodeploy]) is automatic, so merging development recreates the staging container without a gate — relevant if a task’s safety requirement covers staging too, but not a production-touch. p24-gitlab-build-webhook / p24-monitoring (development→staging) and zsk-archivization (development-test) are staging-only auto as well.

Recommended remediation (not applied in this audit — flagged for a decision)

Making pinbox24-ms-s3-v2’s prod-back-end-deploy (master) when: manual would bring it in line with every other backend repo and remove the merge⇒prod-deploy coupling entirely (the pre-merge check below would then always pass for it). That is a change to upstream pinbox24/* CI behaviour and a deliberate deploy-policy decision, so it is flagged here rather than applied — decide whether s3-v2’s auto-deploy-on-master is intentional (some teams want it) before changing it; the same question applies to p24-ms-mailgun. Until then, the mandatory pre-merge .gitlab-ci.yml check above is the safety net.

Graph View

Backlinks