queue-dispatch-cross-project

Playbook: How Projects Enqueue Tasks via the Queue API

Service: p24-meta-dispatcher Cloudflare Worker
Related docs: docs/queue-api-operations.md, docs/meta-dispatcher-operations.md, docs/queue-dispatcher-operations.md

---

Dispatch Protocol Summary

Project	repo field	job_profile (auto-derived)	Keys location
et-operational-platform	radieu/et-operational-platform	nextjs-supabase-vercel	Vercel env vars
brandpilot	radieu/brandpilot	nextjs-supabase-vercel	secrets/brandpilot.env.sops (synced to Vercel)
Art-Agency	radieu/Art-Agency	nextjs-supabase-vercel	secrets/art-agency.env.sops
radekkonarski-personal-brand	radieu/radekkonarski-personal-brand	content-automation	secrets/radekkonarski-brand.env.sops (pending C1)
amazon-kdp-tango	radieu/amazon-kdp-tango	kdp-python	secrets/kdp-tango.env.sops (pending)
whatsup-android-chat-puller	radieu/whatsup-android-chat-puller	generic	secrets/whatsup.env.sops
p24-infra	radieu/p24-infra	infra-ops	secrets/monitoring.env.sops

---

0. Which repos can rely on dispatch-to-queue.yml vs. the orphan backstop vs. POST /queue-issue

There are three ways an issue (or PR) reaches dev_r_worker_queue, and not every repo has all three:

1. dispatch-to-queue.yml — a GitHub Actions workflow living in the target repo that fires on issues.opened / issues.milestoned (and pull_request.* for review) and inserts the row within seconds. Only repos that ship this workflow get immediate event-driven enqueue. Only p24-infra has it. et-operational-platform does not (verified 2026-10-05 on main/dev/staging — its /new-issue = scripts/new-issue.ps1 creates a Triage issue + Trello card and spawns a local agent, nothing remote), and neither do whatsup-android-chat-puller or other sibling repos.

2. Orphan backstop poll (#6831 for issues, #7707 for PRs) — the p24-meta-dispatcher CF Worker’s scheduled run (crons = ["0 */4 * * *"], i.e. every 4 h) re-scans each repo in DEFAULT_BACKSTOP_REPOS (infra-src/meta-dispatcher/src/index.ts — includes radieu/et-operational-platform) and queues any open active-milestone issue that has no dev_r_worker_queue row (as dev-issue, or continue-issue for a Review-milestone issue) and any open non-draft PR that has no in-flight review-pr row. This is the default automatic path for et-op and the other backstop repos — a Triage-milestoned et-op issue is picked up within one producer cycle (≤ 4 h) with no manual call. Issue dedup is handled by getQueuedIssueNumbersForRepo plus the insertQueueRow guard, so it never double-queues.

   PR-review ownership is split — single owner per repo (#7715). The backstop’s PR scan does not enqueue PRs for a repo already covered by the Python dispatcher’s scan_prs_for_review() (scripts/queue-dispatcher-loop.py, config scripts/dispatcher-review-config.json REVIEW_REPOS = p24-infra, et-op, Art-Agency, radekkonarski-personal-brand, gmail-tools). So et-op PRs are reviewed by the dispatcher, not the backstop; the backstop PR scan only owns repos the dispatcher does not cover (today: only whatsup-android-chat-puller). The excluded set is DISPATCHER_OWNED_PR_REVIEW_REPOS in src/index.ts (override: env BACKSTOP_PR_SCAN_EXCLUDE_REPOS); it must stay in sync with REVIEW_REPOS. The backstop PR scan is further bounded by head-sha dedup (skip a PR already reviewed at its current head sha; the sha is stored in the queue row metadata.head_sha), a per-cycle cap (BACKSTOP_REVIEW_PR_CAP, default 3 per repo per tick, oldest-updated PR first — the rest defer to later ticks), and skips for ai-review-queued-labelled PRs and PRs whose mergeable_state is dirty. Kill switch: set the wrangler var BACKSTOP_PR_SCAN=0 to disable the PR scan only — the #6831 issue backstop keeps running. In-flight dedup (getInflightReviewPrNumbersForRepo) + the insertQueueRow guard still apply on top. Background: the unbounded first cut (#7709) would have fanned out ~19 full 9-lens councils every 4 h (#7663 repeat-review burn).

3. POST /queue-issue on p24-meta-dispatcher (this playbook, §2) — the repo-agnostic, immediate path. Use it when you need dispatch faster than the ≤ 4 h backstop (e.g. /new-issue’s verify-and-fallback, an urgent issue, or a one-off PR review) — it inserts the row directly, for any repo, regardless of whether that repo ships its own workflow.

4. dispatch-cross-repo.yml (workflow in p24-infra) — the one-step operator path for sibling repos. A workflow_dispatch workflow in radieu/p24-infra that takes repo, issue_number, job_type (dev-issue|continue-issue) and priority, validates them against the allowlist, checks the issue is dispatchable (open, not a PR, not human-action/no-auto-dispatch, not a pinned tracker), then calls POST /queue-issue using QUEUE_API_URL/QUEUE_API_KEY from p24-infra’s own GH Secrets and a GitHub App token scoped to the one target repo (issues:write, used only to read the issue and move Triage -> Queue). The operator never handles QUEUE_API_KEY, and nothing is added to the sibling repo. Usage — from any p24-infra session:

   /new-issue --repo radieu/et-operational-platform <description>        # create + enqueue, one step
   # or, for an issue that already exists:
   gh workflow run dispatch-cross-repo.yml --repo radieu/p24-infra \
     -f repo=radieu/et-operational-platform -f issue_number=<N> -f job_type=dev-issue

   Single config source: scripts/dispatcher-review-config.json. CROSS_REPO_DISPATCH_REPOS is the allowlist (today only et-op; every entry must also be in REVIEW_REPOS), and REVIEW_REPO_BASE_BRANCHES gives the PR base branch (et-op: dev, promotion dev -> staging is et-op’s own gitflow; p24-infra / unlisted: main). worker-issue.md and worker-nextjs-dev.md read that same map (when /opt/p24-infra/scripts/ is present on the worker host) and fall back to the older “dev exists on origin” heuristic otherwise. To add a repo: add it to REVIEW_REPOS, CROSS_REPO_DISPATCH_REPOS and (if not main) REVIEW_REPO_BASE_BRANCHES in one reviewed PR; scripts/tests/test_dispatch_cross_repo.py enforces the invariants.

   Guardrails: trigger is workflow_dispatch only (no issue/PR event, so no self-trigger loop); per-(repo, issue) concurrency group; dedup is the Worker’s (dedup.ts in-flight + done-row guard) and the orphan backstop (mechanism 2) skips any issue that already has a row, so a later 4 h tick never double-queues. Not covered: POST /queue-issue itself still accepts any repo for any holder of QUEUE_API_KEY (the Brandpilot/Art-Agency Vercel callers depend on that), so the allowlist is enforced by this workflow, not by the Worker.

   PR approval identity: workers open PRs and stop (E1, #7612); merge is done by the review-pr worker after APPROVE. et-op (GitHub Free, private) has no branch protection, so require_last_push_approval is not enforced there; the guarantee is process-level only. See docs/playbooks/git-identity-and-roles.md and docs/playbooks/ai-pr-auto-review.md §Enforcement model.

The /new-issue skill trap (#6813). /new-issue creates a Triage issue and reports “queued to the remote worker”. On a repo without dispatch-to-queue.yml the immediate event-driven enqueue does not happen — the create fires no workflow, so with only the 4 h backstop the issue would otherwise sit in Triage for up to one cycle (three whatsup-android-chat-puller issues sat >1 h with no row before the skill was fixed). The skill now verifies a row landed after creating the issue and falls back to POST /queue-issue when it did not — so it enqueues immediately or fails loudly on every repo. The orphan backstop (mechanism 2) is the safety net that catches anything the skill missed within ≤ 4 h; the POST /queue-issue fallback is what makes dispatch immediate. When adding a new sibling repo, either give it its own dispatch-to-queue.yml or add it to DEFAULT_BACKSTOP_REPOS; never assume the create-and-milestone alone enqueues anything instantly.

1. Overview

The p24-meta-dispatcher Cloudflare Worker exposes an HTTP API that any sibling project can call to dispatch a GitHub issue into the p24-infra worker queue. Within ~2 minutes, the queue-dispatcher on bms-4 picks up the job and spawns a Claude worker via spawn-worker.sh.

Your project → POST /queue-issue → CF Worker → dev_r_worker_queue
                                                      ↓ (within 2 min)
                                            queue-dispatcher-loop.py (bms-4)
                                                      ↓
                                            spawn-worker.sh → systemd-run → Claude worker

bms4 proxy (planned — ADR-005 §9, #6112). A centralization component on bms-4 is planned to sit between remote/ephemeral workers and Anthropic: a single static egress IP, centralized account/task routing (reusing the dispatcher’s existing _account_order headroom ranker), and OAuth session custody for ephemeral workers (windows-dev Docker containers, #6110). It does not change how sibling projects enqueue — the POST /queue-issue path below is unchanged. Design-of-record: docs/bms4-proxy-operations.md.

Use this API when: a sibling project (BrandPilot, et-operational-platform, Art-Agency, radekkonarski-personal-brand) needs to trigger a p24-infra worker to implement a GitHub issue automatically.

Do NOT use this API for:

• Dispatching issues that belong to the calling project’s own repo without moving them to radieu/p24-infra first (or supply the repo field to target the right repo)
• Bypassing the issue pipeline — the issue must exist on GitHub first

---

2. Endpoint Reference

Base URL

https://p24-meta-dispatcher.radieu.workers.dev

Authentication

All endpoints require:

Authorization: Bearer <QUEUE_API_KEY>

Missing or invalid token returns 401 {"error":"Unauthorized"}. The key is validated with a constant-time comparison to prevent timing attacks.

---

POST /queue-issue — Dispatch a single issue

Enqueues one GitHub issue for processing by a Claude worker.

Request:

POST https://p24-meta-dispatcher.radieu.workers.dev/queue-issue
Authorization: Bearer <QUEUE_API_KEY>
Content-Type: application/json
 
{
  "issue_number": 1234,
  "repo": "radieu/et-operational-platform",
  "job_type": "dev-issue",
  "weight": "light",
  "priority": 10
}

Field	Required	Default	Description
issue_number	Yes	—	GitHub issue number
repo	No	radieu/p24-infra	Target repo (e.g. radieu/et-operational-platform)
job_type	No	dev-issue	dev-issue, infra-task, or infra-alert
weight	No	auto-derived	light (3 GB) or heavy (8 GB) — omit to let the Worker derive from issue labels/complexity
priority	No	auto-derived	Lower = processed first (1–20). Omit to let Worker compute
job_profile	No	auto-derived	Tech-stack profile. Auto-derived from repo name if omitted. Values: nextjs-supabase-vercel | python-fastapi | infra-ops | content-automation | kdp-python | generic

When weight/priority are omitted, the Worker fetches the issue from GitHub and auto-derives weight from acceptance-criteria count and labels.

Successful response — 201:

{
  "id": 42,
  "status": "queued",
  "job_type": "dev-issue",
  "weight": "light",
  "priority": 10,
  "github_issue_number": 1234,
  "repo": "radieu/et-operational-platform",
  "queued_at": "2026-06-25T10:00:00Z",
  "metadata": { "source": "queue-api" }
}

The returned id is the dev_r_worker_queue row ID. Use it to poll job status.

Error responses:

Status	Body	Cause
400	{"error":"issue_number required"}	Missing issue_number in body
401	{"error":"Unauthorized"}	Invalid or missing Bearer token

---

GET /queue-issue?job_id=N — Poll job status

Check the status of a previously enqueued job.

GET https://p24-meta-dispatcher.radieu.workers.dev/queue-issue?job_id=42
Authorization: Bearer <QUEUE_API_KEY>

job_id is the id returned by POST /queue-issue — not the GitHub issue number.

Successful response — 200:

{
  "id": 42,
  "status": "running",
  "job_type": "dev-issue",
  "weight": "light",
  "priority": 10,
  "github_issue_number": 1234,
  "repo": "radieu/et-operational-platform",
  "queued_at": "2026-06-25T10:00:00Z",
  "claimed_at": "2026-06-25T10:02:15Z",
  "started_at": "2026-06-25T10:02:16Z",
  "metadata": { "source": "queue-api" }
}

Status lifecycle:

queued → claimed → running → done
                           → failed
                           → cancelled
                           → oom_killed

Error responses:

Status	Body	Cause
400	{"error":"job_id required"}	Missing or non-numeric job_id
401	{"error":"Unauthorized"}	Invalid or missing Bearer token
404	{"error":"not found"}	No row with that id

---

POST /trigger — Manual meta-dispatcher run

Triggers the wave-aware batch queueing logic immediately (normally runs every 4 hours via cron). Returns 202 immediately; dispatch runs in the background.

POST https://p24-meta-dispatcher.radieu.workers.dev/trigger
Authorization: Bearer <QUEUE_API_KEY>
Content-Type: application/json
 
{ "all_waves": false }

Response: 202 {"ok":true}

---

3. Per-Project Setup — Where to Get the Keys

3.1 BrandPilot

QUEUE_API_KEY and QUEUE_API_URL are already present.

SOPS location: p24-infra/secrets/brandpilot.env.sops

# Decrypt to verify (Windows dev workstation)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\brandpilot.env.sops | Select-String "QUEUE_"

Vercel env vars: synced automatically by secrets-sync.yml on merge to dev or main. Available as:

• QUEUE_API_KEY
• QUEUE_API_URL → https://p24-meta-dispatcher.radieu.workers.dev

No manual setup needed — keys are live in both SOPS and Vercel.

---

3.2 et-operational-platform

Auto-dispatch is live via the orphan backstop (§0 mechanism 2, 7707). et-op issues on a Triage milestone are picked up automatically within ≤ 4 h (one meta-dispatcher cron cycle) — no manual POST /queue-issue needed. The manual POST below is the immediate path for when you cannot wait for the next cycle, not the default. et-op does not ship its own dispatch-to-queue.yml, so only these two paths apply to it.

et-op PR review is owned by the dispatcher, not the backstop (#7715). et-op is in the Python dispatcher’s REVIEW_REPOS, so its open PRs are auto-reviewed by scan_prs_for_review() — and the meta-dispatcher backstop deliberately excludes et-op from its PR scan (single owner per repo; see §0 mechanism 2). Do not expect the backstop to review et-op PRs.

QUEUE_API_KEY and QUEUE_API_URL are in secrets/monitoring.env.sops (they cover vps-i1/bms-4 env as well).

SOPS location: p24-infra/secrets/monitoring.env.sops

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "QUEUE_"

To add to et-operational-platform Vercel project:
Open a p24-infra session and run:

vercel env add QUEUE_API_KEY production preview
vercel env add QUEUE_API_URL production preview

Values come from secrets/monitoring.env.sops.
Or file a radieu/p24-infra issue with label infra-task-request — the n8n pipeline will handle it.

---

3.3 Art-Agency

QUEUE_API_KEY and QUEUE_API_URL are now configured in secrets/art-agency.env.sops (added in PR #1849).

SOPS location: p24-infra/secrets/art-agency.env.sops

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\art-agency.env.sops | Select-String "QUEUE_"

To sync to the Art-Agency Vercel project, open a p24-infra session and run:

vercel env add QUEUE_API_KEY production preview
vercel env add QUEUE_API_URL production preview

Or use /request-infra from any Art-Agency session — see docs/playbooks/cross-project-infra-requests.md.

SOPS target: p24-infra/secrets/art-agency.env.sops

---

3.4 radekkonarski-personal-brand

Keys are not yet in this project’s SOPS file. To add them:

1. Open a p24-infra session.
2. Decrypt radekkonarski-personal-brand/secrets/radekkonarski-brand.env.sops, append the keys, re-encrypt, push.
3. Reference in n8n workflows on bms-4 via the n8n credential or env injection.

SOPS target: radekkonarski-personal-brand/secrets/radekkonarski-brand.env.sops

---

4. Integration Examples

4.1 curl — manual dispatch

curl -s -X POST "https://p24-meta-dispatcher.radieu.workers.dev/queue-issue" \
  -H "Authorization: Bearer $QUEUE_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"issue_number": 1234, "repo": "radieu/et-operational-platform", "job_type": "dev-issue"}' \
  | jq .

Poll status:

JOB_ID=42
curl -s "https://p24-meta-dispatcher.radieu.workers.dev/queue-issue?job_id=$JOB_ID" \
  -H "Authorization: Bearer $QUEUE_API_KEY" | jq .status

---

4.2 n8n HTTP Request node (bms-4)

Node settings:

Setting	Value
Method	POST
URL	{{ $env.QUEUE_API_URL }}/queue-issue
Authentication	Predefined Credential Type → Header Auth
Credential	p24-infra-queue-api (or create Header Auth with name Authorization, value Bearer {{ $env.QUEUE_API_KEY }})
Specify Body	JSON (not keypairs — see note below)
JSON	={{ JSON.stringify({ issue_number: $json.issue_number, repo: "radieu/et-operational-platform", job_type: "dev-issue" }) }}

n8n 2.26.x bug: specifyBody=keypairs silently sends an empty body. Always use specifyBody=json with a JSON.stringify(...) expression. See memory file feedback_n8n_2_26_specifyBody_bug.md.

Complete HTTP Request node JSON (copy into Code node or import):

{
  "parameters": {
    "method": "POST",
    "url": "={{ $env.QUEUE_API_URL }}/queue-issue",
    "authentication": "predefinedCredentialType",
    "nodeCredentialType": "httpHeaderAuth",
    "specifyBody": "json",
    "jsonBody": "={{ JSON.stringify({ issue_number: $json.issue_number, repo: \"radieu/et-operational-platform\", job_type: \"dev-issue\" }) }}",
    "options": {}
  },
  "credentials": {
    "httpHeaderAuth": { "id": "<credential-id>", "name": "p24-infra-queue-api" }
  }
}

Credential setup (once per n8n instance):
Settings → Credentials → New → Header Auth:

• Name: p24-infra-queue-api
• Name field: Authorization
• Value field: Bearer <QUEUE_API_KEY>

Never hardcode the key value in node parameters — always use the credential reference.

---

4.3 TypeScript / Next.js (BrandPilot or et-operational-platform)

// lib/queue.ts
 
export interface QueueJob {
  id: number;
  status: "queued" | "claimed" | "running" | "done" | "failed" | "cancelled" | "oom_killed";
  job_type: string;
  weight: "light" | "heavy";
  priority: number;
  github_issue_number: number;
  repo: string;
  queued_at: string;
  claimed_at?: string;
  started_at?: string;
}
 
export async function dispatchIssue(
  issueNumber: number,
  repo = "radieu/p24-infra",
  jobType = "dev-issue",
): Promise<QueueJob> {
  const url = process.env.QUEUE_API_URL;
  const key = process.env.QUEUE_API_KEY;
  if (!url || !key) throw new Error("QUEUE_API_URL or QUEUE_API_KEY not set");
 
  const resp = await fetch(`${url}/queue-issue`, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${key}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ issue_number: issueNumber, repo, job_type: jobType }),
  });
 
  if (!resp.ok) {
    const err = await resp.json().catch(() => ({}));
    throw new Error(`Queue API ${resp.status}: ${JSON.stringify(err)}`);
  }
 
  return resp.json() as Promise<QueueJob>;
}
 
export async function pollJobStatus(jobId: number): Promise<QueueJob> {
  const url = process.env.QUEUE_API_URL;
  const key = process.env.QUEUE_API_KEY;
  if (!url || !key) throw new Error("QUEUE_API_URL or QUEUE_API_KEY not set");
 
  const resp = await fetch(`${url}/queue-issue?job_id=${jobId}`, {
    headers: { Authorization: `Bearer ${key}` },
  });
 
  if (resp.status === 404) throw new Error(`Job ${jobId} not found`);
  if (!resp.ok) throw new Error(`Poll failed: ${resp.status}`);
 
  return resp.json() as Promise<QueueJob>;
}
 
// Usage in a Server Action or API Route:
//
// const job = await dispatchIssue(1234, "radieu/et-operational-platform");
// console.log(`Queued as job #${job.id}, weight=${job.weight}`);
//
// // Poll until done (simple loop — use exponential backoff in production):
// while (true) {
//   const s = await pollJobStatus(job.id);
//   if (["done","failed","cancelled","oom_killed"].includes(s.status)) break;
//   await new Promise(r => setTimeout(r, 10_000)); // wait 10s
// }

---

5. What Happens After Dispatch

POST /queue-issue
  → CF Worker inserts row into dev_r_worker_queue (status=queued)
  → returns { id, status:"queued", ... }

  ↓ within ~2 min

queue-dispatcher-loop.py on bms-4 (runs every 2 min via p24-queue-dispatcher.timer):
  1. Claims dispatcher lease via claim_dispatcher_lease() RPC
  2. Checks available RAM via available_ram_gb('bms-4')
  3. If RAM available: PATCHes row to status=claimed, SSH-executes spawn-worker.sh
  4. spawn-worker.sh allocates a systemd cgroup, runs claude -p with the worker prompt
  5. Row status → running; PID recorded
  6. Worker completes: row status → done (or failed / oom_killed)

Exit-code escalation (if worker can't fit in current weight tier):
  light → heavy → playwright → orchestrator → failed

Stuck jobs auto-recovered: reset_stale_workers() re-queues running rows after 2 hours.

Monitoring:

• Mezmo: filter app:p24-queue for dispatch events (dispatch_cycle, spawn_ok, spawn_oom)
• Grafana: worker-queue-v1 dashboard — queue depth, running count, RAM headroom
• Discord: #infra-errors channel for spawn_oom and dispatch_failed events

Typical end-to-end time:

• Queued → claimed: ≤ 2 min (next dispatcher tick)
• Claimed → running: < 30 seconds (SSH + systemd-run startup)
• Running → done: depends on issue complexity (light: 5–30 min, heavy: 15–60 min)

---

6. Troubleshooting

Job stays queued for > 5 minutes:

-- Check dispatcher lease
SELECT holder, expires_at FROM dev_r_dispatcher_lease;
-- Check available RAM
SELECT available_ram_gb('bms-4');
-- Check queue depth
SELECT status, COUNT(*) FROM dev_r_worker_queue GROUP BY status;

If dispatcher is dead, see docs/playbooks/worker-queue-operations.md § Fix — Queue not draining.

Job status is failed immediately:

The Worker could not insert the row (Supabase auth issue) or the GitHub issue was not found. Check:

• QUEUE_API_KEY matches the CF Worker secret
• issue_number exists in repo
• Supabase service key is valid (SUPABASE_SERVICE_ROLE_KEY in CF Worker)

401 Unauthorized from the API:

• Verify QUEUE_API_KEY was copied from secrets/monitoring.env.sops (or brandpilot.env.sops)
• Check that the Authorization header is exactly Bearer <key> with a space after Bearer
• The key is validated with constant-time comparison — even a trailing newline causes failure

OOM (oom_killed):

Dispatcher auto-re-queues as heavier weight (light → heavy). If it OOM-kills again at heavy, the issue needs splitting. See docs/playbooks/worker-queue-operations.md § Fix — OOM worker.

---

7. Related

• docs/queue-api-operations.md — full API reference (all endpoints, CORS, rate limits)
• docs/meta-dispatcher-operations.md — CF Worker deployment, cron scheduling, wave logic
• docs/queue-dispatcher-operations.md — bms-4 dispatcher internals, RAM budget, exit codes
• docs/playbooks/worker-queue-operations.md — stuck jobs, OOM recovery, failover
• docs/playbooks/cross-project-infra-requests.md — how to request infra changes from non-p24-infra sessions
• infra-src/meta-dispatcher/src/index.ts — Worker source
• scripts/spawn-worker.sh — worker spawn logic and exit code reference