spawn-worker-sudo-scope-failure

Playbook: spawn-worker fails immediately — systemd-run scope + sudo

Trigger

All workers fail immediately after spawn. Log entries:

[ERROR] spawn failed for #<N> weight=heavy

Dispatcher marks jobs as OOM at max weight (heavy) even when RAM is plentiful.

Root Cause

systemd-run --user --scope exits immediately after scope creation when invoked via sudo -u claude-runner (no PAM session / session leader). PID=$! captures the systemd-run PID, which is already dead within 1 second. kill -0 $PID fails → spawn-worker.sh exits 2 → dispatcher interprets as OOM.

The scope and Claude process continue running — only the PID detection is broken.

Under su -s /bin/bash claude-runner -c '...', systemd-run stays alive as scope controller and PID=$! is valid.

How to Confirm

# On bms-4 as root:
sudo -u claude-runner bash -c '
  export XDG_RUNTIME_DIR=/run/user/$(id -u claude-runner)
  systemd-run --user --scope --quiet -- sleep 10 &
  PID=$!
  sleep 1
  echo "PID=$PID alive=$(kill -0 $PID 2>/dev/null && echo YES || echo NO)"
'
# Expected if broken: alive=NO
 
su -s /bin/bash claude-runner -c '
  export XDG_RUNTIME_DIR=/run/user/$(id -u)
  systemd-run --user --scope --quiet -- sleep 10 &
  PID=$!
  sleep 1
  echo "PID=$PID alive=$(kill -0 $PID 2>/dev/null && echo YES || echo NO)"
'
# Expected if fixed: alive=YES

Check dispatcher command in queue-dispatcher-loop.py:

grep "su -s /bin/bash\|sudo -u" /opt/p24-infra/scripts/queue-dispatcher-loop.py

Fix

Both fixes are in scripts/spawn-worker.sh and scripts/queue-dispatcher-loop.py. They were deployed via PR #1458 (merged to dev 2026-06-25).

Fix 1 — spawn-worker.sh (defensive): Write bash $$ to a temp PID file at scope start; poll 4×0.5s for that file then kill -0 on the worker bash PID instead of the systemd-run PID.

Fix 2 — dispatcher (root fix): Changed sudo -u claude-runner env {...} spawn-worker.sh to su -s /bin/bash claude-runner -c 'export {...}; exec spawn-worker.sh ...'.

If regression occurs:

1. git pull on vps-i1 (/opt/p24-infra) to get the latest dispatcher
2. git pull on bms-4 (/opt/p24-infra) to get the latest spawn-worker.sh

Remediation After Detection

1. Identify all stuck rows:

   SELECT id, github_issue_number, weight, error_message
   FROM dev_r_worker_queue
   WHERE status = 'failed' AND error_message LIKE '%OOM at max weight%';

2. Re-queue as new heavy rows (on vps-i1 where server-side key is available):

   SUPA_URL="https://mwkqmgadqnkkihjdeqsi.supabase.co"
   SUPA_KEY=$(grep -m1 '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/monitoring/.env | cut -d= -f2- | tr -d '"')
   for ISSUE_NUM in <list>; do
     curl -sf -X POST "$SUPA_URL/rest/v1/dev_r_worker_queue" \
       -H "apikey: $SUPA_KEY" -H "Authorization: Bearer $SUPA_KEY" \
       -H "Content-Type: application/json" -H "Prefer: return=minimal" \
       -d "{\"github_issue_number\":${ISSUE_NUM},\"repo\":\"radieu/p24-infra\",\"job_type\":\"dev-issue\",\"weight\":\"heavy\",\"priority\":80,\"status\":\"queued\",\"retry_count\":0,\"max_retries\":2,\"metadata\":{}}"
     echo "Queued #$ISSUE_NUM"
   done

3. Monitor pickup within 2 min:

   tail -f /var/log/p24-infra-workers/dev-issue-<N>.log

Secondary Bug: WORKER_PID_FILE race (PR #1463, 2026-06-25)

Symptom: Sporadic exit-2 failures even after the su/sudo fix, specifically when claude exits in <500ms (auth failure, expired credentials, PATH issue).

Root cause: The inner bash-c string deleted WORKER_PID_FILE as part of its own cleanup:

rm -f "${PROMPT_FILE}" '${WORKER_PID_FILE}'   # ← RACE: deletes PID file before outer poll

If claude exits faster than the outer bash’s first 0.5s poll, the PID file is gone before it can be read. The outer bash sees an empty poll result and exits 2.

Fix (PR #1463): Remove '${WORKER_PID_FILE}' from the inner bash cleanup. The outer bash owns PID file lifecycle and already calls rm -f "${WORKER_PID_FILE}" after the polling loop.

Tertiary Bug: &; syntax error in heartbeat subshell (2026-06-25)

Symptom: PID file is NEVER written — the inner bash exits before echo $$ > PIDFILE. All 4 poll attempts return empty; outer bash exits 2 regardless of RAM, su mode, or WORKER_PID_FILE cleanup state. Confirmed via systemd-run 2>/tmp/err.txt capturing:

/bin/bash: -c: line 1: syntax error near unexpected token `;'

Root cause: The heartbeat subshell in the bash -c string ended with done) &;:

(while kill -0 $$ 2>/dev/null; do
  sleep 60; curl ...
done) &;     ← `;` immediately after `&` is invalid bash syntax

Bash PARSES the entire -c string before executing any of it. The &; syntax error causes the inner bash to exit immediately (exit 2), before writing the PID file. The outer bash polls an empty /tmp/p24-spawn-*.pid for 2s and exits 2.

Why not caught earlier: spawn-diag.sh tests used sleep 5 instead of the full HB block. debug-worker-inner.sh used QUEUE_ROW_ID=0 so the if condition was false — but bash still parses the then-block, causing the same syntax error. The parse error was only discovered by capturing systemd-run’s stderr (2>/tmp/sd-err.txt instead of 2>/dev/null).

Fix (commit f9fbc5e): Remove the ; — change done) &; to done) &:

done) &  \   # ← no semicolon; & already terminates the command

Deployed to both bms-4 and vps-i1 via direct git checkout from the feature branch.

Infrastructure Note: .server-label

spawn-worker.sh reads /opt/p24-infra/.server-label to identify the server in logs. This file must exist on each server. If missing, SERVER_NODE=unknown appears in logs.

Create it manually if missing:

echo "bms-4" > /opt/p24-infra/.server-label   # on bms-4
echo "vps-i1" > /opt/p24-infra/.server-label  # on vps-i1

Prevention

• spawn-worker.sh uses worker PID file approach — robust against future su/sudo changes
• queue-dispatcher-loop.py uses su (not sudo) — correct PAM session
• Inner bash-c cleanup MUST NOT delete WORKER_PID_FILE — outer bash owns that file
• Never use &; in a bash -c string — & already terminates a command; ; after & is a parse error. Use & \ (background + line continuation) or just & with no following ;
• When editing the inner bash -c string in spawn-worker.sh: always test with bash -c '...' 2>&1 locally to catch parse errors BEFORE deploying
• If spawn-worker.sh returns exit 2 and the worker log is empty: capture systemd-run stderr via 2>/tmp/sd-err.txt (replace 2>/dev/null) to see parse errors
• Ensure .server-label is created during VPS provisioning (ansible role)