p24-infra Docs

concurrent-sops-write-worktree-race

6 min read

Playbook: Concurrent SOPS Write — Shared-Checkout Filesystem Race

Created: 2026-08-03 Incident: A w3_app MongoDB password rotation committed a foreign, non-authenticating value because a concurrent agent session was writing the same tracked file at the same time — taking v32-prod (W3 production) into a MongoDB auth-failure loop for several minutes. Issue: #5242 Related incidents: #5224, #5223, #5225 (GH_TOKEN concurrent-rotation race, same window)

Trigger

A credential/shared-file write (typically a sops write to secrets/*.env.sops, but any write to a tracked file works) is executed directly in a shared checkout — the primary C:\code_2026\p24-infra / /opt/p24-infra working directory that other Claude Code sessions or background Agent() calls are simultaneously operating in — instead of an isolated tmp/wt-{issue} worktree mandated by CLAUDE.md §Agent Workflow — Worktree Branch Isolation.

Symptoms

  • git checkout main fails with fatal: 'main' is already used by worktree at .claude/worktrees/agent-…
  • .git/config: Permission denied during git pull / git checkout -b (concurrent .git access)
  • A sops --decrypt read returns a value that matches neither password this session generated — a transient snapshot of another process’s in-flight write to the same file
  • Your own freshly-rotated secret does not authenticate after distribution, because the commit captured a stale/foreign snapshot rather than your value
  • Follow-up “resync” commits are needed to converge on the correct value (and the first resync can itself chase the wrong, race-observed value)

Root cause

Git’s concurrency protections cover the object store and ref updates — they do not cover uncommitted working-tree file writes. When two sessions share one physical checkout, a plain filesystem write to secrets/foo.env.sops in session A can interleave with a sops read/write in session B: the reader sees a half-written or foreign snapshot, and whichever git add runs next stages whatever bytes are on disk at that instant.

This is a distinct failure mode from shared-checkout-worktree-isolation.md (#3230). That incident was a git-branch race (a commit landing on another session’s checked-out branch), and its hook guard blocks git commit / branch-switch in the primary checkout. Neither the git-branch race nor its hook guard touches the filesystem-level race described here — a SOPS write is not a git commit, so the #3230 guard never fires on it.

Immediate recovery

  1. Stop distributing. A value that fails to authenticate must not be pushed further.
  2. Get the correct value out-of-band. Read the intended password from a stable scratchpad copy (never re-derive it from the racing secrets/ file). Reference the key name only — never print the value (CLAUDE.md §Secrets).
  3. Re-converge the SOPS file from that known-good value in a single isolated worktree, run the canary decrypt, then redistribute (SOPS → GH Secrets → live envs → containers).
  4. Follow static-api-key-incident-rotation.md if any value was exposed in the process.

Prevention — pre-flight guard (added for #5242)

Before any write to a shared tracked file (above all secrets/*.env.sops), run the read-only guard and abort the write if it reports UNSAFE:

scripts/check-worktree-write-safety.sh secrets/pinbox24-w3.env.sops || {
  echo "Unsafe — move into an isolated worktree before writing"; exit 1;
}

The guard exits non-zero when the current directory is a concurrent-write hazard:

  • you are in the primary checkout while ≥1 linked worktree is active, or
  • the same branch is checked out by 2+ worktrees (the “already used by worktree” condition).

It reads only git worktree list / git rev-parse — it never touches, decrypts, or prints any secret, so it is safe for any role (including sys-admin/sys-security read-only roles) to run. A non-zero exit is its designed verdict, not a system error, so it deliberately does not fire the Discord/GH error path.

The safe workflow it points you to

# 1. Isolated worktree for this credential op
git worktree add tmp/wt-{issue} -b {branch} origin/main
cd tmp/wt-{issue}
 
# 2. Confirm the directory is race-free, then do the SOPS write here
../../scripts/check-worktree-write-safety.sh secrets/{file}.env.sops
# ... sops write, canary decrypt, git add, git commit ...
 
# 3. Clean up after merge — remove, guarantee the dir is gone, then prune (#6006)
cd ../.. && git worktree remove --force tmp/wt-{issue} && rm -rf tmp/wt-{issue} && git worktree prune -v

A leftover worktree dir silently falls through to the parent repo — see worktree-remove-fallthrough-guard.md.

Dispatch-path audit (#5242 items #1, #3, #4)

Follow-up items 4 asked why a credential-write-capable session was dispatched into the shared checkout, and whether any other dispatch path can do the same. Findings:

  • Queue dispatch (bms-4 / vps-i1) — already isolated. scripts/spawn-worker.sh clones every job (including p24-infra itself and every secret-manager job) into a per-issue working directory /tmp/worker-{issue}-{repo-slug} and cds into it before launching the agent. This closed an earlier shared-/opt/p24-infra branch race (#1689). A queue-dispatched credential rotation therefore already runs in its own filesystem-isolated clone — it was not the #5242 vector.
  • Local orchestration commands — already isolated. issues-review, is-check-nofuture-no-human, process-issues-remote, fix-fast, bug, and auto-issue-pipeline all spawn their workers into dedicated git worktree add checkouts.
  • request-secrets — this was the gap (item #3), now closed. .claude/commands/request-secrets.md spawned the background secret-manager agent with a hardcoded Working directory: C:\code_2026\p24-infra — the shared primary checkout — with no worktree isolation and no pre-flight guard. A rotation dispatched through it (or run manually in the same shared checkout, as in the incident) could race a concurrent session’s in-flight SOPS write. Answer to item #1: the task landed in the shared dir because the dispatch command told it to. The command now requires the agent to create and enter an isolated worktree and run check-worktree-write-safety.sh before any SOPS write (v1.2.0).
  • Cross-reference (item #4). The GH_TOKEN concurrent-rotation race (#5225, PR #5235) in the same window is the same class: two rotation-capable sessions writing the same shared checkout without isolation. The worktree-isolation rule enforced here (and the guard from PR #5244) is the shared mitigation.

Limitations

  • The guard is advisory and opt-in — nothing auto-invokes it yet. It is a fail-fast check the writer (human or agent) is expected to call. The request-secrets dispatch command (v1.2.0) now instructs the spawned agent to run it before any SOPS write, but wiring it into the secret-manager role’s own SOPS-write path (so it fires regardless of dispatch route) is still a secret-manager follow-up (see #5242 deferred items).
  • Like the #3230 hook, it only reasons about worktrees of the local repo on this machine; it cannot see other machines/CI runners operating on a clone.
  • It detects the conditions for a race, not an in-progress write — its value is preventing the write from starting in an unsafe directory at all.

Graph View

Backlinks