p24-infra Docs

6094-self-polling-queue-worker

18 min read

Design: Self-polling queue-worker path for wsl1-que-w1 (que-type worker)

⚠️ SUPERSEDED (2026-08-11). This design’s outbound-poll/native-process architecture conflicts with ADR-005’s Docker + CF-Tunnel-inbound design for the same host (wsl1/windows-dev), caught by the /review-plan pass on #6085. Owner decision: build ADR-005’s Docker design instead. Do not implement this design as scoped. Kept as historical record — the job-claiming/race-safety research here (reusing the dispatcher’s guarded PATCH pattern) may still be useful reference material if the Docker build ever needs a self-polling fallback mode, but it is not the current plan.

Issue: #6094 Status: Superseded by ADR-005 (see notice above) — was: Design (plan-iteration workflow) Author: dev-coder worker Depends on: #6091 (wsl1 provisioning, merged)

1. Context & problem

wsl1 (WSL2 Ubuntu on the Windows dev workstation) was provisioned in #6091 as a scoped-down worker node: outbound SSH + CF Tunnel verified, registered in dev_r_server_capacity (enabled=false) and dev_r_services (wsl1-que-w1). Same credential scoping as lap1 — no role-*.env.sops access, no primary SOPS age key, P1–P2 jobs only, never P0 (plan #4556 Phase 3; docs/playbooks/lap1-worker-credentials.md).

No que-type / self-polling worker path exists in the codebase today. The documented que worker-type (docs/playbooks/adding-new-worker.md line 18, docs/environments/lap1.md, docs/environments/wsl1.md) is design intent only — never implemented. Confirmed by grepping the repo: the only consumer of the queue outside the dispatcher itself is a read-only metrics exporter.

Today’s dispatch pipeline (scripts/queue-dispatcher-loop.py, 2910 lines) is entirely SSH-shaped:

leader dispatcher host (bms-4 / vps-i1)
  ├─ reads dev_r_server_capacity.ssh_host / ssh_user for each enabled server
  ├─ claim_job(): GET eligible queued row → guarded PATCH queued→claimed (server_node=<label>)
  └─ spawn_worker(): ssh -i /root/.ssh/p24-dispatcher <ssh_user>@<ssh_host>
        └─ su -s /bin/bash <worker_user> -c 'export …; exec /opt/p24-infra/scripts/spawn-worker.sh …'
              └─ spawn-worker.sh clones repo, resolves prompt file, runs `claude … -p`,
                 heartbeats dev_r_worker_queue, PATCHes final status on exit

spawn-worker.sh (1201 lines) assumes Linux + systemd + a claude-runner account + /opt/p24-infra shared checkout + the universal age key. It is not self-invocable by a poller and is not reusable as a whole.

A que-type worker is architecturally a different path: nothing SSHes into wsl1 (its inbound CF Tunnel SSH is in fact broken — see docs/environments/wsl1.md §Known Issues — and does not matter here). Instead wsl1 reaches out over HTTPS, claims its own rows directly via Supabase REST, and invokes Claude Code locally. This design specifies that path.

Note — this is a new architecture pattern for the queue system (first self-polling worker). It is scoped deliberately narrow (one light-only, single-slot, P1–P2 worker) and adds a parallel claim path that is race-safe against the existing SSH dispatcher by construction (§5). The production dispatch pipeline is not modified.

2. Goals & non-goals

Goals

  1. A self-polling worker on wsl1 that claims dev_r_worker_queue rows via Supabase REST, spawns Claude Code locally, and keeps the shared accounting tables honest — with zero inbound access.
  2. Claim semantics identical to and race-safe against queue-dispatcher-loop.py (no double-claim against the SSH dispatcher or another poller).
  3. P1–P2-only enforcement in the claim query itself, not merely documented (issue requirement 6).
  4. Reuse of the proven claude … -p invocation and the worker-issue.md REST-fallback path (SUPA_MCP_AVAILABLE=0), so wsl1 needs no MCP and no age key to run the standard pipeline.
  5. Supervised, auto-restarting, survives WSL/Windows reboot.
  6. A pattern that generalises to lap1-que-w1/w2 (registered, never activated for the same reason).

Non-goals

  • Modifying queue-dispatcher-loop.py or spawn-worker.sh (the SSH path is untouched).
  • Giving wsl1 any P0 / incident / sys-admin capability, role credentials, or the primary age key.
  • Solving the CF Tunnel inbound-SSH bug (irrelevant to an outbound-only poller).
  • Automating the human-interactive claude auth login — explicitly deferred (§9).

3. Architecture overview

The poller is queue-dispatcher-loop.py’s claim step + spawn-worker.sh’s invoke step, fused into one outbound-only process that dispatches only to itself. One new script, scripts/que-worker-poll.sh (bash — no non-stdlib deps, matches the REST-fallback bash already in worker-issue.md), run as a long-lived systemd service on wsl1:

p24-que-worker.service  (systemd, Restart=always, single slot)
  loop every POLL_INTERVAL_S (default 30s):
    0. self-gate:  is my dev_r_server_capacity row enabled? in a permitted window? under cap?  → else sleep
    1. claim:      GET one eligible P1–P2 row  →  guarded PATCH queued→claimed (server_node=wsl1)
                   (same row lock the SSH dispatcher uses → auto-mutex, §5)
    2. account:    current_workers += 1 (guarded), last_heartbeat = now
    3. prepare:    git clone target repo → /tmp/worker-<issue>-<repo-slug>; resolve prompt file
    4. invoke:     export identity env; run `claude --dangerously-skip-permissions --strict-mcp-config -p "$(cat prompt)"`
                   with a background heartbeat loop PATCHing dev_r_worker_queue.heartbeat_at
    5. finalize:   PATCH final status on the queue row (mirror spawn-worker.sh's exit patch);
                   current_workers -= 1; clean up workdir
    (blocks on the claude run: single slot, so no new claim until this finishes)

Key insight — no new mutex is invented. The SSH dispatcher claims a row with PATCH dev_r_worker_queue?id=eq.{id}&status=eq.queued (queue-dispatcher-loop.py:1254-1256). If the poller issues the identical guarded PATCH, the two serialise on the Postgres row lock and only the first re-matches status=eq.queued; the loser’s PATCH updates 0 rows and re-polls. So a self-polling worker and the SSH dispatcher are mutually exclusive on any row for free — the existing status guard is the cross-path mutex. This is the same property the worker-issue.md REST-fallback supa_claim_task already relies on for the per-issue agent_tasks mutex.

4. Design point 1 — Job-claiming mechanism (REST poll + atomic claim)

Mirrors claim_job() (queue-dispatcher-loop.py:1199-1265) exactly, with server_label=wsl1 and the P1–P2 filter from §6 folded into the GET.

4.1 Claim — two HTTP calls (GET then guarded PATCH)

PostgREST cannot do ORDER BY + LIMIT inside a PATCH, so — like the dispatcher — the claim is:

# Step A — GET one eligible row (read-only). NOW_ISO computed with `date -u +%FT%TZ`.
#   Filters (all ANDed):
#     status=eq.queued
#     job_type=in.(dev-issue,continue-issue,review-pr,review-plan)      # §6 P1-P2 allow-list
#     weight=in.(super-light,light)                                      # §6 wsl1 capacity
#     priority=gte.${P0_FLOOR}                                           # §6 defensive floor (default 5)
#     and=(or(next_attempt_at.is.null,next_attempt_at.lte.${NOW_ISO}),   # backoff gate (dispatcher parity)
#          or(server_preference.is.null,server_preference.eq.wsl1))      # affinity (dispatcher parity)
#   order=priority.asc,queued_at.asc & limit=1
row=$(supa_get "dev_r_worker_queue?status=eq.queued&job_type=in.(dev-issue,continue-issue,review-pr,review-plan)&weight=in.(super-light,light)&priority=gte.${P0_FLOOR}&and=(or(next_attempt_at.is.null,next_attempt_at.lte.${NOW_ISO}),or(server_preference.is.null,server_preference.eq.wsl1))&order=priority.asc,queued_at.asc&limit=1&select=id,issue_number,github_issue_number,job_type,weight,repo,role,repo_context,job_profile,metadata")
[ "$(jq 'length' <<<"$row")" -eq 0 ] && continue   # nothing eligible → next poll tick
 
id=$(jq -r '.[0].id' <<<"$row")
 
# Step B — atomic guarded PATCH (THE mutex — identical guard to the SSH dispatcher).
claimed=$(supa_patch_return "dev_r_worker_queue?id=eq.${id}&status=eq.queued" \
  "{\"status\":\"claimed\",\"server_node\":\"wsl1\",\"server_label\":\"wsl1\",\"claimed_at\":\"${NOW_ISO}\",\"worker_id\":\"wsl1-que-w1\"}")
[ "$(jq 'length' <<<"$claimed")" -eq 0 ] && continue   # race lost to dispatcher / another poller → re-poll

supa_get / supa_patch_return are thin curl wrappers with apikey/Authorization: Bearer ${SUPA_HB_KEY}, Prefer: return=representation on the PATCH — the same helpers already sketched in worker-issue.md Step 0 (supa_patch, supa_upsert).

4.2 Why not reuse spawn-worker.sh’s claim?

spawn-worker.sh does not claim — the dispatcher claims before SSHing, and the Claude worker claims the per-issue agent_tasks mutex itself (worker-issue.md Block B). The poller therefore owns the queue-row claim (this section), and the spawned Claude worker still runs Blocks A/B/C to take the per-issue agent_tasks mutex in REST-fallback mode. Both layers remain intact:

LayerTableGuardWho runs it
Dispatch-row mutexdev_r_worker_queueid=eq.X & status=eq.queuedthe poller (was: dispatcher)
Per-issue mutexagent_tasksgithub_issue_id=eq.N & status=eq.pendingthe spawned Claude worker (unchanged)

4.3 Concurrency-safety summary

  • vs SSH dispatcher: identical guarded PATCH → row lock serialises them; loser gets 0 rows.
  • vs another wsl1 poll tick: single-slot service (§6.2 cap = 1) means only one loop iteration is ever in claim; a crashed-mid-claim row is either still queued (re-claimable) or claimed by the dead session and reclaimed by the same stale-reclaim logic worker-issue.md already documents.
  • vs lap1 pollers (future): same guard, different server_node value → same free mutex.

5. Design point 6 — P1–P2-only enforcement (in the claim query, not just docs)

The problem: dev_r_worker_queue has no P0/P1/P2 class column — priority is a bare integer (lower = higher priority; queue-dispatcher-loop.py:1210). Observed enqueue values: rotation priority=1, monitoring-watchdog 1, normal implementation 10, review-pr 15, low 50. So “never P0” cannot be a single column check; it is enforced by three ANDed filters baked into the Step-A GET (§4.1), defence-in-depth:

5.1 Primary guard — job_type allow-list (the real P0 boundary)

“P0” in the lap1/wsl1 scoping is defined operationally as incident response + sys-admin operations (adding-new-worker.md:309, lap1.md). In the queue those map precisely to specific job types, and the dispatcher already treats them as hard-affinity (bound to a specific host — _HARD_PREF_JOB_TYPES = {nc-alert-batch, alert-triage-batch, infra-task}, queue-dispatcher-loop.py:146-147). So wsl1 claims only the developer-productivity job types:

job_type=in.(dev-issue,continue-issue,review-pr,review-plan)

This excludes by construction: infra-task (SSH/Docker/systemctl on a server), nc-alert-batch / alert-triage-batch (incident triage, vps-i1-bound), secret-manager (credential ops), and rotation jobs — i.e. every P0/sys-admin/incident category. It is an allow-list (fail-closed): a future job type is not claimable by wsl1 until explicitly added here.

5.2 Secondary guard — weight allow-list

wsl1’s capacity row is light-only across all windows (max_weight=max_weight_prime=max_weight_night= light). Enforce it in the claim so a mis-weighted heavy job can never land on wsl1:

weight=in.(super-light,light)

Excludes heavy, playwright, orchestrator.

5.3 Defensive floor — priority

Even within the allow-listed job types, refuse anything enqueued at emergency priority (so a dev-issue hand-queued at priority=1 to jump the line for an incident is still not grabbed by the least-available host):

priority=gte.${P0_FLOOR}          # P0_FLOOR default = 5 (configurable env)

P0_FLOOR=5 sits above the observed emergency band (1) and below normal dev work (10, 15). This is a documented convention, not a schema guarantee — see §11 open question O1 on whether to formalise a priority_class column repo-wide.

5.4 Affinity + backoff parity

and=(or(server_preference.is.null,server_preference.eq.wsl1), or(next_attempt_at.is.null,next_attempt_at.lte.NOW)) — identical to the dispatcher: wsl1 only takes unpreferred rows or rows explicitly preferring wsl1, and honours spawn-failure backoff. It will not steal a row that prefers another host.

5.5 Enforcement test (implementation must include)

A unit/integration test (pytest, mirroring scripts/tests/) asserting the claim query rejects: a priority=1 dev-issue, an infra-task row, a heavy row, and a server_preference=bms-4 row — and accepts a plain priority=10 weight=light job_type=dev-issue row. This is the machine-checkable proof required by issue requirement 6.

6. Design point 2 — Integration with dev_r_server_capacity accounting

Today current_workers / last_heartbeat on the capacity row are only ever touched by the SSH-dispatched path (the dispatcher increments after a successful spawn; queue-dispatcher-loop.py:2768-2771). A self-poller bypassing the dispatcher must keep the row honest itself, or the dispatcher’s own RAM/slot view and Grafana go stale.

6.1 Self-gate (before every claim)

The poller reads its own capacity row each tick and claims nothing unless all hold:

cap=$(supa_get "dev_r_server_capacity?server_label=eq.wsl1&select=enabled,server_status,max_workers_prime,max_workers_night,max_weight_prime,max_weight_night,current_workers")
enabled=$(jq -r '.[0].enabled' <<<"$cap")
[ "$enabled" = "true" ] || { sleep "$POLL_INTERVAL_S"; continue; }     # honour the kill-switch
 
# window: 20:00–02:59 UTC = night (matches dispatcher night-task band); pick prime/night caps
maxw=$(is_night && jq -r '.[0].max_workers_night' <<<"$cap" || jq -r '.[0].max_workers_prime' <<<"$cap")
[ "$maxw" -ge 1 ] || { sleep "$POLL_INTERVAL_S"; continue; }           # night cap 0 → don't work at night

enabled=false is the operational kill-switch: flip it in the DB and the poller drains (finishes its in-flight job, claims nothing more) with no process management on wsl1.

6.2 In-flight cap (count-based, authoritative)

Rather than trust a possibly-drifted current_workers counter, gate on the actual in-flight count the dispatcher itself uses (get_ram_usage, queue-dispatcher-loop.py:988-1018):

active=$(supa_get "dev_r_worker_queue?server_node=eq.wsl1&status=in.(claimed,running)&select=id" | jq 'length')
[ "$active" -lt "$maxw" ] || { sleep "$POLL_INTERVAL_S"; continue; }

Because wsl1 is max_workers*=1, this is effectively “claim only when idle” — and it’s self-healing: a crashed worker’s stale claimed row is swept by the existing retry/stale reclaim path (queue-retry-worker.py + worker-issue.md stale-reclaim), after which active drops and polling resumes.

6.3 Counter + heartbeat writes (best-effort, keep the table honest)

# on successful claim:  guarded increment so concurrent writers don't clobber
supa_patch "dev_r_server_capacity?server_label=eq.wsl1" '{"current_workers": <active_after>, "last_heartbeat":"NOW"}'
# every poll tick (idle or busy): refresh heartbeat so Grafana/health sees wsl1 alive
supa_patch "dev_r_server_capacity?server_label=eq.wsl1" '{"last_heartbeat":"NOW"}'
# on worker exit: decrement

To avoid lost-update races on current_workers, the implementation should set it from the freshly counted active value (§6.2) rather than blind +1/-1, so the count is derived and cannot drift below 0 or above the true in-flight count. last_heartbeat is a plain last-writer-wins timestamp.

Decision: current_workers is maintained best-effort for observability; the authoritative gate is the count-based §6.2 query. If a current_workers write fails (transient REST error), the poller logs and continues — correctness never depends on it. This is called out explicitly because the alternative (making it authoritative) would reintroduce exactly the count-drift class that already paused bms-4 (see its capacity notes).

7. Design point 3 — Claude Code invocation

Reuse the spawn-worker.sh tail verbatim in spirit (spawn-worker.sh:1122):

cd "${WORKDIR}" && nice -n 10 claude --dangerously-skip-permissions --strict-mcp-config \
  ${CLAUDE_MODEL_FLAG} -p "$(cat "${PROMPT_FILE}")" >> "${LOG}" 2>&1
  • --strict-mcp-config ⇒ MCP off ⇒ the worker runs the REST-fallback path (SUPA_MCP_AVAILABLE=0) that worker-issue.md already fully specifies — no age key, no MCP required on wsl1. ✅ matches wsl1’s scoping.
  • --dangerously-skip-permissions — unattended, same as every existing worker.
  • nice -n 10 for the light tier; systemd-run --scope cgroup limits are optional on wsl1 (single slot, 31 GB) and can be added later — not required for a 1-slot node.

7.1 Environment exported before invocation (from the claimed row + host identity)

Env varValueSource
CLAUDE_WORKER_IDwsl1-que-w1fixed on this host
CLAUDE_WORKER_HOST / hostnamewsl1 host idhostname
SERVER_NODE / CLAUDE_SERVER_LABELwsl1fixed
QUEUE_ROW_IDclaimed idrow
SOURCE_TABLEdev_r_worker_queuefixed
QUEUE_JOB_TYPErow job_typerow
CLAUDE_JOB_WEIGHTrow weightrow
CLAUDE_ROLErow role (e.g. dev-coder)row
REPO_CONTEXT_FILEinfra/repo-contexts/<repo>.md if present else ""derived
PR_NUMBERmetadata.pr_number (review-pr only)row
CLAUDE_MODELmetadata.model if setrow
SUPA_MCP_AVAILABLE0fixed (REST fallback)
SUPA_HB_URLhttps://mwkqmgadqnkkihjdeqsi.supabase.cofixed
SUPA_HB_KEYSUPABASE_SERVICE_ROLE_KEYwsl1 env file (§10 — secret-manager)
TELEGRAM_CHAT_IDusually emptyrow/env (optional)

7.2 Prompt-file resolution (mirror spawn-worker.sh:442-534)

  • review-pr → review-pr-worker.md; review-plan / dev-issue / continue-issue → worker-issue.md, upgraded to worker-issue-scoped.md when CLAUDE_ROLE is set and the scoped file exists (spawn-worker.sh:480-499) — the exact case that applies here (this very issue ran under worker-issue-scoped.md).
  • Prompt files are read from a local p24-infra checkout on wsl1 (the poller lives there; §8), path infra/agent-prompts/<file>. wsl1 has no shared /opt/p24-infra, so the poller’s own checkout is the source, kept fresh with git fetch --quiet && git reset --hard origin/main before reading (or ensure-fresh-checkout.sh semantics).

7.3 Repo checkout + heartbeat + exit patch (mirror spawn-worker.sh:244-307, 1107-1141)

  • WORKDIR=/tmp/worker-${ISSUE}-${REPO_SLUG}, git clone --depth=1 --no-single-branch via gh auth git-credential (wsl1 must have gh logged in as the machine account — §10). No inline PAT in the URL (the #4047 at-rest-leak vector).
  • Background heartbeat loop PATCHing dev_r_worker_queue.heartbeat_at every 60s while the claude PID is alive.
  • On exit: classify (ok → status=done,result_summary; nonzero → status=failed, failure_reason=implementation_error,error_message) and PATCH with guard status=in.(running,claimed) (idempotent — the Claude worker’s own Step 11.5 may have already set it). Clean up WORKDIR.

The queue-row status lifecycle is therefore written from two cooperating places, exactly as in the SSH path: the spawned Claude worker sets done/failed in its Step 11.5, and the poller’s exit-patch is the idempotent backstop for crashes where the worker never reached Step 11.5.

8. Design point 5 — Where the poller runs & how it’s supervised

WSL2 on wsl1 has systemd enabled and working (sshd + cloudflared already run as systemd services there — docs/environments/wsl1.md:51-59). Two options were considered:

OptionProsCons
A. Long-lived systemd service (poll loop, Restart=always) ← chosenone process, cheap idle poll, natural single-slot blocking on the claude run, instant kill-switch via enabledmust handle its own sleep/backoff
B. systemd .timer oneshot (claim-one-then-exit)no long-lived processtimer granularity, cold-starts the resolve/clone each tick, harder to hold a single slot cleanly

Chosen: A. p24-que-worker.service:

[Unit]
Description=p24-infra self-polling queue worker (wsl1-que-w1)
After=network-online.target
Wants=network-online.target
 
[Service]
Type=simple
User=claude-runner
WorkingDirectory=/home/claude-runner/p24-infra          # wsl1's own checkout
EnvironmentFile=/home/claude-runner/.config/p24-que-worker.env   # SUPA_HB_KEY etc (0600, §10)
ExecStart=/home/claude-runner/p24-infra/scripts/que-worker-poll.sh
Restart=always
RestartSec=15
# Optional hardening: MemoryMax=, CPUWeight= (single slot, not required day 1)
 
[Install]
WantedBy=multi-user.target
  • Reboot survival: the existing Windows Scheduled Task P24-WSL-Autostart (trigger: at logon) brings the distro + its systemd services up automatically (wsl1.md:67), and C:\Users\konar\.wslconfig vmIdleTimeout=-1 keeps the VM alive. WantedBy=multi-user.target ⇒ the service auto-starts with systemd. No extra Windows wiring needed.
  • Kill-switch: dev_r_server_capacity.wsl1.enabled=false drains it (finishes in-flight, claims nothing) with no shell access. systemctl stop p24-que-worker is the hard stop.
  • Crash recovery: Restart=always + RestartSec=15. An in-flight job whose poller dies leaves a claimed/running row that the existing queue-retry-worker.py stale sweep re-queues.

8.1 Error notification (mandatory — CLAUDE.md §Error Notification Standard)

The poll loop must, on any unhandled error (claim/spawn/REST failure that isn’t a normal race): Discord embed via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL and a bug-labelled GH issue. A repeating transient error must be rate-limited (e.g. one Discord ping per N minutes) so a network blip on a laptop doesn’t spam. P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL is another value wsl1 needs delivered (§10).

9. Design point 4 — Non-interactive auth bootstrap (HUMAN ACTION — deferred)

The poller can claim rows and set up env, but Claude Code itself needs a valid OAuth session at ~/.claude/.credentials.json inside WSL. This requires an interactive browser/terminal OAuth flow that no autonomous worker can perform:

  1. Human, once, on wsl1 (interactive shell): claude auth login — its own independent login on the subscription account. Verify with a trivial claude -p "echo ok".
  2. Never copy .credentials.json from another host — forbidden multi-host refresh-token race (docs/playbooks/claude-oauth-reauth.md Option C; CLAUDE.md §Do NOT). Each host gets its own login.
  3. Then provision the hourly claude-token-refresh cron for the claude-runner account on wsl1 (CLAUDE.md §Claude Code OAuth Token Refresh — mandatory on every claude-runner host; the claude-runner Ansible role templates it). Verify it’s actually live, don’t assume (the #4854 gap).

This is the only step that blocks wsl1-que-w1 going fully live. Per the issue scope note, the implementation PR wires everything else ready-to-run and flags this as a human-action follow-up (sub-issue) rather than leaving #6094 open indefinitely. Suggested follow-up issue title: [human-action] wsl1: run \claude auth login` + verify claude-token-refresh cron to activate wsl1-que-w1`.

10. Prerequisites & cross-role delegation (out of dev-coder scope)

The poller code is dev-coder work; two prerequisites are not and must be delegated:

  1. Secret delivery to wsl1 → secret-manager + a wsl1 secrets-sync path. The poller needs, in /home/claude-runner/.config/p24-que-worker.env (0600):

    • SUPABASE_SERVICE_ROLE_KEY (= SUPA_HB_KEY) — to claim/heartbeat/account.
    • P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL — error notification.
    • GH_TOKEN or a gh auth login on wsl1 — repo clone + gh issue/gh pr in the worker.

    wsl1 has no role-*.env.sops access and no primary age key, so it cannot decrypt these itself. Options for the secret-manager to weigh:

    • (recommended) a narrow secrets/wsl1-worker.env.sops with wsl1 as a recipient (the 2-recipient pattern used for anthropic-admin.env.sops / supabase-ci.env.sops), shipped by a dedicated secrets-sync.yml job — but note wsl1 is outbound-only with no inbound SSH, so the sync must be pull-based (wsl1 fetches) rather than the existing push-over-SSH model. This is itself a design sub-decision (see O2).
    • a dedicated least-privilege Postgres role / scoped PostgREST key limited to dev_r_worker_queue + dev_r_server_capacity writes (tighter than the full service-role key on the least-controlled host) — more work, better blast-radius.

  2. Doc corrections (dev-coder, in the implementation PR):

    • docs/environments/wsl1.md, lap1.md, adding-new-worker.md all say the que workers poll p24_worker_queue. The live table is dev_r_worker_queue — the poller MUST target that. Fix the doc references (or add a clarifying note) so the next reader isn’t misled.

11. Failure modes, idempotency & open questions

Failure modes

ScenarioBehaviour
Race lost to SSH dispatcher on a rowguarded PATCH returns 0 rows → poller re-polls (no double-run)
Poller crashes mid-jobclaimed/running row swept by queue-retry-worker.py; current_workers re-derived on next tick
SUPA_HB_KEY missing/expiredclaim GET/PATCH fail → error-notify (rate-limited), sleep, retry; no rows lost
enabled=false flippeddrains: finishes in-flight, claims nothing
Night window, max_workers_night=0self-gate sleeps; claims nothing until prime window
Mis-weighted heavy / P0 row enqueuedexcluded by the §5 claim filters — never claimed
Claude OAuth not bootstrapped (§9)claude exits nonzero → exit-patch marks row failed; this is why activation is gated on §9

Idempotency: every write is a guarded/last-writer PATCH; the exit-patch guards on status=in.(running,claimed) so it never overwrites a terminal status the Claude worker already set.

Open questions (for /review-plan):

  • O1. Should P0/P1/P2 be formalised repo-wide as a priority_class (or a documented numeric band) column, instead of wsl1 relying on a job-type allow-list + a P0_FLOOR convention? Cleaner and reusable, but a cross-cutting schema/dispatcher change beyond this issue. Recommendation: ship the allow-list now (fail-closed, sufficient), file O1 as a separate enhancement.
  • O2. Secret delivery mechanism to an outbound-only host (pull-based sync vs. one-time manual drop). Needs secret-manager + possibly a new secrets-sync variant. Recommendation: for first activation, a one-time manual 0600 env-file drop by the operator during the §9 human step (it’s the same session), and design the repeatable pull-sync as a fast-follow (it’s shared with lap1).
  • O3. Should current_workers accounting be dropped entirely in favour of the count-based gate (§6.2) being the only mechanism? The count query is authoritative; the counter is cosmetic. Keep it best-effort for Grafana continuity, revisit if it drifts.

12. Implementation plan (for the follow-up implementation PR)

Deliberately not implemented in this design issue — this is the design phase of the plan-iteration workflow. The implementation PR should:

New files

  • scripts/que-worker-poll.sh — the poll loop (§3–§7), bash, REST helpers, error-notify.
  • scripts/systemd/p24-que-worker.service — the unit (§8).
  • scripts/tests/test_que_worker_claim.py — the §5.5 enforcement test (rejects P0/heavy/infra-task/ other-host rows; accepts a normal light dev-issue). Runnable in CI without wsl1.
  • docs/playbooks/self-polling-que-worker.md — ops playbook (deploy, enable/disable, drain, logs, activation checklist) + wire into adding-new-worker.md.

Edits

  • Doc corrections p24_worker_queue → dev_r_worker_queue (§10.2).
  • adding-new-worker.md / wsl1.md: mark wsl1-que-w1 active only after §9.
  • Register/confirm compliance: dev_r_services row already exists (wsl1-que-w1); ensure compliance_workbook/ops-doc link updated to the new playbook.

Deferred to human-action (§9)

  • claude auth login on wsl1 + claude-token-refresh cron verification.
  • Secret delivery (§10.1) — secret-manager.
  • Flip dev_r_server_capacity.wsl1.enabled=true after a supervised soak.

Rollout / soak / rollback

  1. Land code + unit test (this + follow-up PR) — inert while enabled=false.
  2. Human: secrets drop + claude auth login + token-refresh cron (§9, §10).
  3. systemctl enable --now p24-que-worker; watch logs claim one real P2 dev-issue end-to-end.
  4. Soak: keep max_workers_prime=1, observe a few jobs; confirm no double-claim vs dispatcher (grep both logs for the same row id), heartbeat visible in Grafana.
  5. Rollback = enabled=false (drain) or systemctl disable --now p24-que-worker. Zero blast radius on the SSH path.

Generalisation: the same que-worker-poll.sh + unit activates lap1-que-w1/w2 (Windows: run under its own WSL or as a scheduled PowerShell wrapper; the REST/claim core is identical, only the supervisor shell differs). Prove on wsl1 first, then lift to lap1.

Graph View