p24-infra Docs

agent-parallelism-protection

6 min read

Agent Parallelism Protection — How We Prevent Double-Work

Answers issue #1246: “How do we prevent separate environments / parallel sessions from picking up the same issue?” — and the follow-on questions: is the Supabase session registration still needed now that we have a queue, and what should the human’s workflow look like.

Operational manuals for the moving parts already exist — this doc explains the protection model and the human workflow, and links out to them: worker-queue-operations.md · queue-dispatcher-operations.md · agent-session-audit.md

TL;DR

Two agents never do the same issue because claiming is an atomic database write, not a suggestion. There are three independent guards, each at a different layer:

LayerMechanismProtects against
1. DispatchOne dispatcher holds dev_r_dispatcher_lease (leader election)Two dispatchers running at once (bms-4 vs vps-i1)
2. Queue rowclaim_worker_queue_item() — UPDATE … WHERE id=? AND status='queued'Two dispatch cycles claiming the same queued job
3. Workerclaim_task() — … FOR UPDATE SKIP LOCKED WHERE status='pending'Two workers implementing the same GitHub issue

If any layer loses the race, the loser gets NULL / 0 rows back and stops cleanly — it does not retry the same item.

The three guards in detail

Layer 1 — Dispatcher leader election

Only one dispatcher process may dispatch at a time. Before doing anything, the dispatcher calls claim_dispatcher_lease(<server>), which atomically takes a time-limited lease row in dev_r_dispatcher_lease:

-- claim_dispatcher_lease: acquire only if free or expired
UPDATE dev_r_dispatcher_lease
SET holder = p_holder, acquired_at = NOW(), expires_at = NOW() + INTERVAL '5 minutes'
WHERE lease_name = 'queue-dispatcher'
  AND (holder IS NULL OR expires_at < NOW());
-- returns TRUE only to the winner

A dispatcher that does not win the lease exits immediately. (The dev_r_dispatcher_lease table also carries a 150 s ttl_seconds/60 s heartbeat default from migration 035; the claim_dispatcher_lease function the dispatcher actually calls uses the 5-minute window above.) Today bms-4 is the sole active worker host; vps-i1 is the warm standby (see dev_r_server_capacity.enabled). The lease is what makes adding a second dispatcher safe — they cannot both run.

  • Code: scripts/queue-dispatcher.sh, monitoring/supabase/migrations/035_dev_r_dispatcher_lease.sql, 036_worker_queue_operational_extensions.sql

Layer 2 — Queue-row claim (status guard)

The dispatcher loop selects the top job (status='queued' ordered by priority, then queued_at) and then claims it with a conditional update:

UPDATE dev_r_worker_queue
SET status = 'claimed', claimed_at = NOW(), dispatcher_session_id = p_dispatcher_session
WHERE id = p_queue_id AND status = 'queued'   -- ← the guard
RETURNING *;                                   -- NULL if another claim already flipped it

Because the WHERE status='queued' is evaluated under the row lock the UPDATE takes, two concurrent claims cannot both succeed — the second sees 0 rows and moves on. The lifecycle is queued → claimed → running → done | failed | oom_killed | cancelled; each transition has the same kind of status-guarded update (start_worker_queue_item, complete_worker_queue_item).

  • Code: monitoring/supabase/migrations/034_dev_r_worker_queue.sql, scripts/queue-dispatcher-loop.py

Layer 3 — Worker claim (claim_task, the original lock)

This is the guard that predates the queue and is still the last line of defence. Every worker, at Step 0 of infra/agent-prompts/worker-issue.md, calls:

SELECT * FROM claim_task('<session_id>', <issue_number>);
-- claim_task: take the oldest pending task for this issue, skipping rows another txn holds
UPDATE agent_tasks SET status='claimed', claimed_by=p_session_id, claimed_at=NOW()
WHERE id = (
  SELECT id FROM agent_tasks
  WHERE github_issue_id = p_issue_id AND status = 'pending'
  ORDER BY created_at ASC LIMIT 1
  FOR UPDATE SKIP LOCKED      -- ← concurrency-safe selection
)
RETURNING *;

If it returns NULL, another session already owns the issue → the worker sets $shouldStop and exits at Step 12 without touching any files. FOR UPDATE SKIP LOCKED means two workers that fire simultaneously each grab a different row (or one grabs nothing) — never the same one.

Question: now that we have a queue, is the session registration still needed?

Yes — keep it. The queue and the session/task registration solve different problems; they are complementary, not redundant.

ConcernSolved byWhy the other can’t do it
Don’t dispatch the same job twicedev_r_worker_queue (Layers 1–2)The queue only governs queue-spawned work
Don’t implement the same issue twiceagent_tasks + claim_task() (Layer 3)Catches any path — manual /new-issue, a human VS Code session, an /issues-review wave — not just queue spawns
Liveness / heartbeat / stale-worker recoveryagent_sessions.last_heartbeatQueue status alone can’t tell a crashed worker from a slow one; reset_stale_workers() reads heartbeats
Audit & EU AI Act Art. 13 traceability (which physical host ran what)dev_r_agent_sessions (server_node, issues_worked)Queue rows are work items, not the agent-identity ledger

So the recommendation is option “extend, don’t remove”:

  • The queue is the front door for autonomous dispatch and concurrency/RAM budgeting.
  • claim_task/agent_tasks remains the universal mutex on a GitHub issue, covering paths that never go through the queue (including a human working two VS Code windows).
  • agent_sessions + dev_r_agent_sessions stay as the heartbeat + audit layer.

Removing the session/task layer would re-open double-work for every non-queue path and break stale-worker recovery and compliance traceability.

Question: how do I (the human) stop two of my own parallel sessions picking the same issue?

The agents are already protected by claim_task. For your own parallel VS Code sessions, the protection is the issue’s pipeline state, surfaced as labels/milestone:

  1. Route picks through the pipeline, not ad-hoc. When an issue is taken it gets the WIP label and is moved to In Progress; queued-but-not-started issues carry ai-dev-queued. A session should pick the next issue that has neither — that is the convention /issues-review and hourly-devops-triage already follow.
  2. Treat “no WIP / no ai-dev-queued” as the eligibility filter. If you (or another window) start an issue, add WIP first; the other window then skips it.
  3. For anything automated, submit to the queue instead of spawning by hand — then Layers 1–3 do the deduplication for you and you never see the same issue twice.

In short: humans dedupe via labels/milestone; agents dedupe via claim_task; the queue dedupes via leader election + status guard.

Suggested human workflow now

With autonomous dispatch + three-layer claiming in place, the high-value human activities are the ones agents cannot safely self-approve:

  1. Plan / triage new issues — write clear specs so the design phase has something to work from (vague one-word issues get bounced to human-action).
  2. Review Code-change-design comments before implementation — this is the cheapest place to redirect an agent.
  3. Review and merge PRs, especially anything left at Human review or labelled human-action (arch-gate hits, merge conflicts, insufficient-info bounces).
  4. Handle escalations — human-action issues are the explicit “agent stopped, needs you” queue. Drain those first.
  5. Let the queue run the rest. Implementation, type-checks, tests, PR open/merge for well-specified issues are autonomous; you don’t need to hand-pick them.

You generally should not be manually picking implementation issues to code yourself — that is exactly the double-work the system is built to avoid. Your leverage is at the edges: specifying work in, and reviewing work out.

References

  • infra/agent-prompts/worker-issue.md — Step 0 claim logic
  • monitoring/supabase/migrations/034_dev_r_worker_queue.sql — queue table + lifecycle RPCs
  • monitoring/supabase/migrations/035_dev_r_dispatcher_lease.sql — lease / leader election
  • scripts/queue-dispatcher.sh, scripts/queue-dispatcher-loop.py — dispatcher
  • worker-queue-operations.md — full operations manual
  • queue-dispatcher-operations.md — RAM-aware dispatch
  • agent-session-audit.md — session ledger & audit

Graph View