pinbox24-w4-stage-build-dir-env-restore

Playbook: restore the missing backend-environment.env on the W4 staging GitLab runner

Created: 2026-10-06 Source issue: radieu/p24-infra#7515 Related: #7361 (closed, wrong premise — lint was never the blocker), #5812 (docker-deploy-stage.sh hardening that removed in-script env generation), #5792 (stage/prod Compose project collision), #4934 (sync-pinbox24-w4-stage secrets-sync job), docs/w3-w4-stack-operations.md

---

Symptom

pinbox24/p24-back-ts development pipeline is red. The failing job is stage-back-end-deploy (stage: deploy, only:[development], tag autodeploy, deploys v42-stage). It aborts with:

FATAL: backend-environment.env not found in /home/gitlab-runner/builds/<HASH>/0/pinbox24/p24-back-ts.

back-end-build and docker-image-build in the same pipeline are green — the blocker is purely the missing env file at deploy time. s3-environment.env is required by the same check.

Why it happens (root cause)

Since #5812, docker-deploy-stage.sh no longer generates the env files from hardcoded secrets — it requires backend-environment.env and s3-environment.env to already exist in the build dir and fails loudly otherwise. Those files are host-persistent, untracked, and delivered out-of-band by the sync-pinbox24-w4-stage job in .github/workflows/secrets-sync.yml (the stage “autoheal”: patches V42_STAGE_* keys from secrets/pinbox24-w4.env.sops, stage domains v42-stage-test.dev.eat.pl).

The gap: that secrets-sync job writes to a hardcoded path /root/builds/glrt-HOR/0/pinbox24/p24-back-ts/ (the runner’s old root-user build dir). The bms-1-autodeploy runner was later migrated to run as the gitlab-runner user (gitlab-runner run --user gitlab-runner --working-directory /home/gitlab-runner), so its live checkout moved to /home/gitlab-runner/builds/<NEW_HASH>/0/pinbox24/p24-back-ts/ (e.g. VjDgPZMKY). secrets-sync keeps faithfully patching the orphaned glrt-HOR copy; the live build dir never receives the files → every stage deploy fails the pre-exist check.

A GitLab runner re-registration also changes the build-dir hash, reproducing the same orphaning even without a user migration.

Host / runner facts (verified #7515, 2026-10-06)

Fact	Value
Runner host	bms-1 (94.23.26.113, ns367522) — SSH root@ with ~/.ssh/id_bms1 (see bms4-claude-runner-ssh-client-config.md)
Runner	bms-1-autodeploy (id 55300309), tag autodeploy, shell executor, GitLab Runner 13.8.0
Live stage build dir	/home/gitlab-runner/builds/<HASH>/0/pinbox24/p24-back-ts (<HASH> changes on re-register)
Trustworthy env source	secrets-sync-maintained /root/builds/glrt-HOR/0/pinbox24/p24-back-ts/{backend-environment.env,s3-environment.env} (stage domains, current V42_STAGE_*)
v42-stage backend	down (no container) — not a live consumer; low urgency

This is bms-1, same host as prod, different build-dir hash than prod’s 7N4sbbrB (/root/builds/). It is not the manual prod-deploy-p24 path.

Restore procedure (safe — touches only the stage build dir)

Find the live build dir, back up any existing files, copy the trustworthy source in, fix ownership:

ssh -i ~/.ssh/id_bms1 root@94.23.26.113 'bash -s' <<'REMOTE' # PLAYBOOK: pinbox24-w4-stage-build-dir-env-restore.md
set -eu
SRC=/root/builds/glrt-HOR/0/pinbox24/p24-back-ts        # secrets-sync-maintained source
# Live stage build dir = most-recently-modified gitlab-runner-owned p24-back-ts checkout that
# actually carries docker-deploy-stage.sh. Plain loop (no nested `xargs | sh -c "{}"`) so a build
# path with spaces/special chars can't be shell-injected, and so an empty match is detectable.
DST=""
newest=0
for d in /home/gitlab-runner/builds/*/0/pinbox24/p24-back-ts; do
  [ -f "$d/docker-deploy-stage.sh" ] || continue
  mtime=$(stat -c '%Y' "$d")
  if [ "$mtime" -gt "$newest" ]; then newest="$mtime"; DST="$d"; fi
done
# Guard (MANDATORY): never proceed with an empty $DST. This runs as root, so an empty $DST would make
# `cp "$SRC/$f" "$DST/$f"` write the secret-bearing stage env files to filesystem root
# ("/backend-environment.env") — `set -e` does NOT catch it because the discovery loop itself
# succeeds. Fail closed: nothing is written when no live stage build dir is found.
if [ -z "$DST" ]; then
  echo "FATAL: no live gitlab-runner stage build dir found under" \
       "/home/gitlab-runner/builds/*/0/pinbox24/p24-back-ts — aborting, nothing written"
  exit 1
fi
echo "live stage build dir: $DST"
TS=$(date -u +%Y%m%d%H%M%S)
for f in backend-environment.env s3-environment.env; do
  [ -f "$SRC/$f" ] || { echo "FATAL: source $SRC/$f missing"; exit 1; }
  # cp -a preserves the pre-existing file's mode; if it was group/world-readable the backup would
  # inherit that over-permissive mode on this prod-shared host, so clamp the backup to owner-only too.
  if [ -f "$DST/$f" ]; then cp -a "$DST/$f" "$DST/$f.bak.$TS"; chmod 600 "$DST/$f.bak.$TS"; fi
  cp "$SRC/$f" "$DST/$f"
  chown gitlab-runner:gitlab-runner "$DST/$f"
  chmod 600 "$DST/$f"   # secret-bearing V42_STAGE env file on a prod-shared host — owner-only (never world/group readable)
done
grep -E '^(VIRTUAL_HOST|LETSENCRYPT_HOST)=' "$DST/backend-environment.env"   # must be *-stage-test.dev.eat.pl
REMOTE

Confirm VIRTUAL_HOST/LETSENCRYPT_HOST are the stage domains before trusting the copy. Never copy prod’s /root/builds/7N4sbbrB/.../backend-environment.env — it carries prod VIRTUAL_HOST=api.w4.pinbox24.com, which would make v42-stage hijack the production domain/cert.

⚠️ Re-running the deploy is NOT a pure no-op — two blockers (do not force it blindly)

1. Prod-container collision (shared Compose file). docker-compose.yml is shared by stage and prod and hardcodes prod container names redis-v42 and s3-v2-v42-prod (both live, project p24-back-ts). docker-deploy-stage.sh runs docker-compose up -d under COMPOSE_PROJECT_NAME=v42-stage; those two services then clash by name with the running prod containers. The #5792 COMPOSE_PROJECT_NAME fix stops stage from reconciling prod away, but it does not remove the name clash on the hardcoded services. Only run the deploy in a coordinated window, and prefer scoping it to the stage-only services (docker-compose up -d --no-deps backend s3 wkhtml) so redis-v42/s3-v2-v42-prod are never touched. See pinbox24-docker-compose-staging-prod-collision.md.
2. git clean may wipe the restored (untracked) files. The files are untracked and there is no GIT_CLEAN_FLAGS override at project/group/runner level, so GitLab’s default git clean -ffdx can remove them at the next job’s checkout — before the deploy script runs. Verify the runner’s effective clean behavior before relying on a bare file-restore to make the pipeline green.

Durable fixes (follow-up — needs human/design review)

• secrets-sync path: make sync-pinbox24-w4-stage in .github/workflows/secrets-sync.yml discover the live gitlab-runner-owned stage build dir (as in the restore snippet) instead of the hardcoded /root/builds/glrt-HOR/...; keep its fail-closed docker-deploy-stage.sh guard.
• in-job delivery: add a before_script to stage-back-end-deploy in the GitLab .gitlab-ci.yml that copies the env files from a stable host path into the build dir after checkout — the only way to survive git clean reliably. Note the coupled merge⇒deploy warning in w3-w4-stack-operations.md §2 (merging to development triggers a stage deploy).
• compose split: give stage its own Compose file / override without the hardcoded redis-v42 / s3-v2-v42-prod services.