p24-infra Docs

git-checkout-ownership-repair

4 min read

Playbook: Repair mixed ownership in a shared /opt/p24-infra checkout

Status: ACTIVE · first written for [#4462] (bms-4). Applies to any server where a claude-runner worker shares the /opt/p24-infra git checkout with root-run deploy jobs.

Symptom

claude-runner cannot git fetch the shared checkout:

error: insufficient permission for adding an object to repository database .git/objects
fatal: failed to write object / unpack-objects failed

So scripts/lib/ensure-fresh-checkout.sh /opt/p24-infra main returns rc=3 (fetch failed) for every worker — the fail-closed freshness gate (#4453 / #4455) then aborts SOPS-touching workers, or (worse) a worker that skips the gate acts on stale rotated secrets.

Root cause

.github/workflows/deploy-vps-scripts.yml SSHes in as root and runs cd /opt/p24-infra && git fetch origin main && git reset --hard origin/main on each server. Those root-run fetches create loose objects (and, over time, object subdirectories) owned root:root inside .git/objects/, whose parent dirs are owned claude-runner. Once a .git/objects/ab/ subdir is root-owned 0755, claude-runner can no longer create the next loose object under it → fetch fails. (The GitHub Actions self-hosted runners run as claude-runner, so they are NOT the culprit — it is the root-over-SSH deploy step.)

A .git/shallow marker (shallow clone) compounds fetch failures and should be removed.

Diagnose (read-only)

ls -la /opt/p24-infra/.git/shallow                 # present => shallow clone
find /opt/p24-infra/.git/objects -user root | wc -l  # count root-owned objects
ls -la /opt/p24-infra/.git/config /opt/p24-infra/.git/HEAD  # often root-owned too
sudo -u claude-runner git -C /opt/p24-infra fetch origin main   # reproduces the rc=3 error

Fix (needs root on the target server)

Run as root (infra-task worker uses the vps_root_key; SSH annotated # PLAYBOOK: git-checkout-ownership-repair.md). Substitute the worker uid/gid for the box (claude-runner on bms-4).

# 1. Normalise ownership of the ENTIRE checkout back to the worker user. Both halves are polluted:
#    root's `git fetch` writes root:root objects under .git/, and root's `git reset --hard` writes
#    root:root files into the WORKING TREE — claude-runner then cannot reset the tree either
#    (the gate fails writing a file whose parent dir is root-owned). Chowning only .git is NOT
#    enough — chown the whole path.
chown -R claude-runner:claude-runner /opt/p24-infra
 
# 2. Make the repo a proper SHARED repo so future root-run fetches create GROUP-writable
#    objects (git honours core.sharedRepository even when run by root). This is what stops
#    the breakage recurring WITHOUT editing the deploy workflow.
sudo -u claude-runner git -C /opt/p24-infra config core.sharedRepository group
find /opt/p24-infra/.git -type d -exec chmod g+rwxs {} +   # setgid dirs => group inherits + is preserved
find /opt/p24-infra/.git -type f -exec chmod g+rw  {} +
 
# 3. Unshallow (remove .git/shallow) so future fetches are clean. Run AS the worker user.
sudo -u claude-runner git -C /opt/p24-infra fetch --unshallow || \
  sudo -u claude-runner git -C /opt/p24-infra fetch origin main   # already-complete repos: plain fetch
 
# 4. Bring HEAD to the remote tip (clean tree => gate returns rc=0 immediately). reset --hard
#    preserves untracked CI-deployed plaintext env files; it only discards TRACKED local edits
#    (there should be none in a deploy checkout).
sudo -u claude-runner git -C /opt/p24-infra merge-base --is-ancestor HEAD origin/main \
  && sudo -u claude-runner git -C /opt/p24-infra reset --hard origin/main

Verify (expect rc=0)

sudo -u claude-runner bash /opt/p24-infra/scripts/lib/ensure-fresh-checkout.sh /opt/p24-infra main
echo "rc=$?"    # MUST be 0
sudo -u claude-runner git -C /opt/p24-infra status --porcelain   # MUST be empty (clean tree)

Why the tree must read clean — the .gitignore companion fix

ensure-fresh-checkout.sh returns rc=0 only when HEAD == origin/main, or when it can cleanly advance a behind checkout. It refuses to reset a dirty tree (rc=4). secrets-sync.yml deploys several plaintext files into the checkout (bms-4/role-*.env, mongodb-bms.env, n8n-bms4-gh.env, docker-compose.override.yml). The gate’s design assumed CI plaintext is named <server>/.env (covered by the top-level .env rule), but these extra names were untracked-but-unignored, so the tree read permanently dirty and the gate would fail rc=4 on every future origin advance — even with perfect ownership. #4462 added .gitignore rules for them so the tree reads clean. Keep the two fixes together: ownership (this playbook) + .gitignore coverage of every CI-deployed plaintext.

Prevent recurrence

The real cure is upstream: deploy-vps-scripts.yml normalises ownership after its root-run git reset --hard on bms-4 (&& chown -R claude-runner:claude-runner /opt/p24-infra), and the repo is set core.sharedRepository=group with setgid .git dirs so even a root fetch creates group-writable objects. Any new root-run git operation on a shared checkout must do the same, or run the git command as the worker user (sudo -u claude-runner git …). vps-i1 uses a different worker user — do not blindly copy the bms-4 chown there; verify the reader uid first.

  • docs/playbooks/sops-checkout-freshness.md — the freshness gate contract and exit codes
  • docs/playbooks/shared-checkout-worktree-isolation.md
  • #4453, #4455 (freshness gate), #4462 (this repair)

Graph View