p24-infra Docs

worker-bash-tool-nonfunctional

8 min read

Playbook — claude worker spawns but its Bash tool is non-functional

Issue: #4755 (first seen: #4721, 2026-07-31T00:50 UTC on bms-4) Applies to: any host that spawns workers via claude --dangerously-skip-permissions -p "..." as claude-runner — bms-4 (primary dispatch) and vps-i1.

Signature

A worker process spawns and stays alive (visible in ps), but every Bash-tool invocation inside its session returns exit 1 with no stdout/stderr — including trivial commands like true and echo hello, and regardless of whether the sandbox is enabled or disabled. The worker then correctly fail-closes (it cannot run gh/curl/sops/ssh either) and exits.

The failure is silent: because the worker’s own escalation path (gh issue comment, Supabase curl) also needs a working shell, nothing alerts. The only trace is the fail-closed abort text the worker prints to /var/log/p24-infra-workers/<job>-<issue>.log.

The OS shell for claude-runner is fine during this — verify:

ssh root@<host> 'sudo -u claude-runner bash -lc "which claude gh git; claude --version"'

The failure is specific to the Bash tool inside a non-interactive claude -p session, not the login shell.

Detection & alerting (added in #4755)

  • The worker prompts (infra/agent-prompts/infra-task-request-worker.md Step 0-ENV) instruct a worker that detects a dead Bash tool to emit the fixed sentinel line [P24-WORKER-ENV-FAILURE] issue=#<n> ... in its final report. The worker’s text output still reaches the log even when the Bash tool is dead (proven by #4721).
  • monitoring/scripts/claude-worker-metrics.sh (per-minute root cron → node_exporter textfile collector) scans /var/log/p24-infra-workers/*.log touched in the last 6 min for that sentinel (plus the known #4721 prose) and emits claude_worker_env_failure.
  • Prometheus alert WorkerEnvNonFunctional (monitoring/prometheus/rules/ai-workers.yml, severity=critical) fires on claude_worker_env_failure > 0 and routes to Discord/email.

Live deploy is a separate step from the git merge — ansible/roles/bms4-workers/tasks/main.yml now re-syncs /usr/local/bin/claude-worker-metrics.sh from the on-host checkout on every apply (added 2026-08-01, see “Deployment-gap incident” below); reload Prometheus rules via deploy-monitoring-config.yml after any change to monitoring/prometheus/rules/ai-workers.yml.

Live diagnosis — run these first (as claude-runner on the affected host)

The #4755 investigation ruled out the obvious resource causes; re-check them, because a transient spike is the leading suspect and may have cleared by the time you look:

# 1. Reproduce — the definitive test. If this prints REPRO_OK_claude-runner, the env is healthy NOW.
cd /tmp && timeout 120 claude --dangerously-skip-permissions --strict-mcp-config \
  -p 'Use the Bash tool to run exactly: echo REPRO_OK_$(id -un). Reply with only its stdout.'
 
# 2. Disk / inodes (a full FS breaks snapshot/temp writes)
df -h / /tmp /home; df -i / /tmp /home
 
# 3. Memory (OOM pressure)
free -h
 
# 4. Process/fork limits — "every fork() fails with EAGAIN" looks identical to this bug
ulimit -u; ps -L -u claude-runner | wc -l
systemctl show user-$(id -u claude-runner).slice -p TasksMax -p TasksCurrent
 
# 5. Shell snapshots — the leading hypothesis. Heavy pruning + many concurrent claude PIDs = race risk
ls -la /home/claude-runner/.claude/shell-snapshots/
ps -eo pid,user,etime,cmd | grep -F '@anthropic-ai/claude-code' | grep -v grep
 
# 6. The worker's own log — read the fail-closed report it left
tail -40 /var/log/p24-infra-workers/<job>-<issue>.log

Root cause (as of #4755): transient, leading hypothesis = shell-snapshot race

The #4755 investigation could not reproduce the failure — disk, inodes, memory and process/fork limits were all healthy, and a minimal repro succeeded. The 00:50 UTC event was a one-off.

Leading hypothesis (probabilistic, unproven): a concurrent-claude shell-snapshot race. Claude Code’s Bash tool sources a per-session snapshot from ~/.claude/shell-snapshots/ before every command, and its startup also prunes that directory. bms-4 runs many claude processes as the same claude-runner user (4 GitHub-Actions runners, spawned workers, check-sub-usage.py every 10 min — which fires at :50 — and heartbeats). If one session’s startup cleanup deletes the freshly-written snapshot of a concurrently-initialising session, that session’s first Bash command runs source <missing-snapshot> → exit 1, no output, independent of the sandbox — the exact signature.

If it recurs — escalation / hardening options (sys-admin / infra-task)

These are server-side changes, out of scope for a dev-issue PR — open an infra-task if needed:

  1. Isolate per-worker snapshot state. Spawn each worker with its own CLAUDE_CONFIG_DIR (or HOME) so shell-snapshots/ cannot collide. Caveat: ~/.claude/.credentials.json (OAuth) must remain reachable — symlink or copy it into the isolated dir.
  2. Serialise / throttle concurrent claude startups on the host (a spawn lock), so two sessions never run their startup cleanup at the same instant.
  3. Confirm the mechanism next time it fires before it clears: capture strace -f -e trace=openat,execve on the worker PID and look for ENOENT on a shell-snapshots/snapshot-*.sh path.

Until the mechanism is confirmed, the #4755 fix is deliberately scoped to making the failure visible (alerting) rather than blindly changing the spawn path.

strace confirmation pass (2026-07-31, infra-task #4755)

The strace pass in step 3 above was run proactively on bms-4 (claude --version = 2.1.177), tracing unlink,unlinkat,openat,execve across a fresh claude -p startup while two sentinel snapshots (one aged 3 h, one fresh) plus a live session’s snapshot sat in shell-snapshots/.

Result — the cross-session-prune hypothesis (option 1’s premise) is refuted for 2.1.177:

  • The fresh startup unlink()ed exactly one snapshot — its own (the same snapshot-bash-<ts>-<rand>.sh it had just O_CREAT|O_TRUNC-created and appended to). It is self-cleanup on exit, not a directory-wide prune.
  • All three foreign snapshots survived (both planted sentinels and the other live session’s file). No ENOENT on any snapshot-*.sh path. So one session’s startup does not delete another’s snapshot — filenames are per-process-unique and cleanup is scoped to self.

Refined mechanism. The failure is therefore per-session snapshot-generation truncation, not a cross-session race: a session whose own snapshot ends up short (interrupted/slow write during a concurrent-spawn CPU burst) then fails source <own-truncated-snapshot> on every subsequent command — exit 1, no output, sandbox-independent. Two aggravating factors were found and fixed in scripts/setup-claude-env.sh (the script that runs at spawn and whose .bashrc/.bash_profile output the snapshot generator sources):

  1. Non-idempotent profile appends — it appended export P24_INFRA_PATH / export P24_CLAUDE_ROLE with >> on every spawn (145 duplicate blocks apiece on bms-4), despite the file’s “Idempotent” header. A bloated profile lengthens snapshot generation and widens the truncation window. Now rewritten idempotently (grep-out + append into a same-dir temp, atomic mv), self-healing existing duplicates.
  2. Non-atomic truncating writes to shared settings.json, CLAUDE.md, env-version — a concurrent session could read a half-written / zero-byte file at startup. Now all written via temp+mv / os.replace().

On the two heavier options (still available, deliberately not applied): because the strace refuted the prune race that motivated per-worker CLAUDE_CONFIG_DIR isolation, that change is held as a documented follow-up rather than shipped blind (it also carries OAuth-cred-reachability risk on every spawn). Re-open it only if the WorkerEnvNonFunctional alert fires again after the setup-claude-env.sh hardening is deployed — at which point capture a fresh strace of the failing worker (not a healthy one) to confirm truncation before touching the spawn path.

Deployment-gap incident (2026-08-01, sys-admin pass on #4755)

Both fixes above (#4758 observability, #4806 idempotent setup-claude-env.sh) merged to main cleanly, but neither had actually reached the live bms-4 host when re-checked days later:

  1. setup-claude-env.sh’s self-heal hadn’t run yet. The script only executes at Claude Code session start (via the PreToolUse hook → check-env-sync.sh), so the merged idempotent rewrite doesn’t retroactively fix an already-bloated profile until something triggers it. claude-runner’s ~/.bashrc / ~/.bash_profile still carried 147 duplicate export P24_INFRA_PATH blocks apiece on 2026-08-01 — unchanged since the #4755 investigation measured the same number on 2026-07-31. Fix: ran sudo -u claude-runner bash /opt/p24-infra/scripts/setup-claude-env.sh by hand once; both files collapsed to exactly 1 export block each. No spawn should need this manual nudge again going forward (every future spawn’s env-sync hook now runs the idempotent version), but if a host is ever found with a bloated profile again, re-running the script directly is the fix — no need to wait for a coincidental spawn.

  2. The deployed metrics script was stale — the alert was inert since merge. #4758’s claude_worker_env_failure sentinel-scan logic lives in monitoring/scripts/claude-worker-metrics.sh in git, but the file cron actually runs is a separate, standalone copy at /usr/local/bin/claude-worker-metrics.sh (installed once, 2026-06-30, never wired to any Ansible role or CI sync path — confirmed by diffing the two copies: the deployed one was missing the entire env-failure block added in #4758). Consequence: claude_worker_env_failure was never emitted on bms-4, so the WorkerEnvNonFunctional Prometheus alert — despite the rule being correctly loaded — could never fire, even though it looked fully wired end to end. This is the same class of gap as item 1: a merged fix with no deployment path to the actual production file. Fix (immediate, live): backed up the stale copy to /usr/local/bin/claude-worker-metrics.sh.bak-pre4755, copied the current git version over it, chown root:root + chmod 0755, ran it once manually to confirm clean output, and verified end to end: claude_worker_env_failure 0 appears in /var/lib/prometheus/node-exporter/claude-workers.prom, node_exporter serves it on bms-4:9100, and Prometheus on vps-i1 (job=node) now returns a live value for claude_worker_env_failure{instance="54.36.123.110:9100"}. Fix (durable): added two tasks to ansible/roles/bms4-workers/tasks/main.yml (tag claude-worker-metrics) that copy monitoring/scripts/claude-worker-metrics.sh (remote_src, matching the existing queue-analyst/mongodb-restore-drill pattern in the same role) to /usr/local/bin/, and install a newly-tracked scripts/cron.d/claude-worker-metrics fragment to /etc/cron.d/ — both now re-applied on every ansible-playbook playbooks/bms-4.yml run, so this class of drift cannot silently recur.

Prevention takeaway: a script fix landing in git is not the same as it running in production. Any script invoked from a path outside /opt/p24-infra (i.e. any standalone copy under /usr/local/bin, /etc/cron.d, etc.) needs an explicit, idempotent deployment task — manual scp/cp deploys leave no record of what should be kept in sync and drift silently. When closing out a dev-issue PR whose fix targets a deployed copy rather than the /opt/p24-infra checkout itself, explicitly check (diff the two copies) whether the live file actually changed — don’t assume a green merge means the production behavior changed.

Graph View