p24-infra Docs

gitlab-runner-root-traverse-acl

4 min read

Playbook — gitlab-runner cannot traverse /root/s3v2-prod/ (preflight always fails)

Status: ACTIVE (verified live on bms-1 ns367522, 2026-08-09) Scope: bms-1 (94.23.26.113), Pinbox24 W4 microservice s3-v2-v42-prod GitLab CI deploy. Owner: sys-admin (infra-task worker on bms-4). Any change to the /root ACL itself → notify sys-security. Origin: #5971 (rerouted from #5471’s blocked deploy).

Symptom

The GitLab CI deploy job for pinbox24-ms-s3-v2 (runner s3-v2-bms1-autodeploy on bms-1) fails its persistent-patch preflight:

ERROR: Patch missing: /root/s3v2-prod/patches/mailgunFileHandler.helper.js

…even though the file exists with correct perms (644 root:root). The preflight runs test -f <path> as the gitlab-runner OS user (uid 1002), not as root.

Root cause

/root is drwx------ (700, root:root) by default. The gitlab-runner user cannot traverse (x bit) into /root at all — so every path under /root/s3v2-prod/ is unreachable to it, regardless of the (correct) permissions on the subdirectories and files inside. This is structural: the deploy preflight for any file under /root/s3v2-prod/ has never been passable by the runner.

Confirm the diagnosis (read-only):

ssh root@94.23.26.113 '
  stat -c "%a %U:%G %n" /root /root/s3v2-prod /root/s3v2-prod/patches
  sudo -u gitlab-runner test -x /root && echo "traverse-OK" || echo "traverse-DENIED"
  sudo -u gitlab-runner test -f /root/s3v2-prod/patches/mailgunFileHandler.helper.js \
    && echo "file-READ-OK" || echo "file-MISSING-OR-DENIED"
'

The fix — least-privilege ACL (chosen over chmod 755 /root)

The issue listed four options. The applied fix is an ACL (POSIX setfacl), done in a least-privilege form, not chmod 755 /root (which would expose all of root’s home to every local user) and not moving the directory tree (a large cross-repo change touching the GitLab CI script, compose bind-mounts, and repo docs).

PathACL granted to gitlab-runnerWhy this level
/root--x (execute/traverse only, no read)Runner can pass through /root to reach s3v2-prod, but cannot list or read anything else in root’s home. Narrowest grant that unblocks the preflight.
/root/s3v2-prodr-x + default r-xRead+traverse the deploy-artifacts tree; the default ACL makes new files/dirs inherit runner-read.
/root/s3v2-prod/patchesr-x + default r-x (see hardening)Read+traverse the patches dir; default ACL keeps regenerated patch files readable.
…/patches/*.helper.jsr--Read the individual preflight patch files.

Security note — why this is defensible without a full sys-security gate: gitlab-runner is already a member of the docker group (groups … 998(docker)), which is root-equivalent (it can docker run -v /:/host and read anything). Granting it traverse-only on /root therefore adds no meaningful privilege it did not already effectively hold — it only makes an existing capability usable by the CI preflight without a container. The --x (no r) grant on /root specifically avoids letting the runner enumerate root’s home, keeping the blast radius minimal. Still: any edit to the /root ACL is worth a one-line heads-up to sys-security because ACLs on /root are unusual and easy to lose track of.

Commands (idempotent — safe to re-run)

# traverse-only on /root  → runner can pass through, cannot read root's home
setfacl -m u:gitlab-runner:x /root
 
# read+traverse on the deploy tree, with a DEFAULT so new files inherit runner-read
setfacl -m  u:gitlab-runner:rx /root/s3v2-prod
setfacl -d -m u:gitlab-runner:rx /root/s3v2-prod
setfacl -R -m u:gitlab-runner:rx /root/s3v2-prod          # existing files/dirs
setfacl -m  u:gitlab-runner:rx /root/s3v2-prod/patches
setfacl -d -m u:gitlab-runner:rx /root/s3v2-prod/patches  # DEFAULT — durability (see below)

Durability gap this closes (#5971)

The initial fix set the default ACL on /root/s3v2-prod but not on /root/s3v2-prod/patches. POSIX default ACLs are inherited only from the immediate parent at file-creation time, so a patch file regenerated in patches/ via rm+recreate (rather than in-place overwrite) would be born with no runner ACL — silently reintroducing the exact preflight failure on the next deploy that rewrites the patch. Setting default:user:gitlab-runner:r-x on patches/ (last command above) makes every regenerated patch file readable by the runner automatically, matching the pattern already on the parent dir.

ACLs are stored in ext4 extended attributes and survive reboots. They do not survive a later chmod that drops the mask, or the directory being deleted and recreated by a fresh checkout.

Verify (after any change, and as a standalone health check)

ssh root@94.23.26.113 '
  for p in /root /root/s3v2-prod /root/s3v2-prod/patches; do
    sudo -u gitlab-runner test -x "$p" && echo "$p x-OK" || echo "$p x-DENIED"
  done
  sudo -u gitlab-runner test -f /root/s3v2-prod/patches/mailgunFileHandler.helper.js \
    && echo "preflight-file READABLE" || echo "preflight-file BLOCKED"
  getfacl -p /root/s3v2-prod/patches | grep -E "gitlab-runner|default"
'

All lines must read x-OK / READABLE. If any is DENIED/BLOCKED, re-run the fix commands.

  • #5471 — the W4 s3-v2-v42-prod upload-retry deploy blocked on this preflight.
  • docs/w3-w4-stack-operations.md — Pinbox24 W3/W4 ownership + permission matrix (this is a sys-admin/infra-task op).
  • docs/playbooks/pinbox24-s3v2-master-branch-autodeploy.md — the coupled merge⇒deploy behaviour of the same pinbox24-ms-s3-v2 pipeline; relevant when the deploy actually runs after this unblock.

Graph View