p24-infra Docs

meta-dispatcher-operations

7 min read

meta-dispatcher — Operations

Purpose

Cloudflare Worker (p24-meta-dispatcher) that replaces the local /issues-queue PowerShell skill. Combines two functions in one Worker:

  1. HTTP Queue API — single ad-hoc dispatch: POST /queue-issue
  2. Cron meta-dispatcher — wave-aware: reads GitHub issues, builds dependency waves, inserts dev_r_worker_queue rows every 4 hours

The existing queue-dispatcher-loop.py on bms-4 is unchanged — it remains the consumer.

Deployment

Automated (CI/CD — preferred)

infra-src/meta-dispatcher/ci/deploy-meta-dispatcher.yml (install to .github/workflows/) deploys the Worker and binds all 5 secrets automatically. It runs on:

  • push to main touching infra-src/meta-dispatcher/** (or the workflow itself)
  • manual: gh workflow run deploy-meta-dispatcher.yml --repo radieu/p24-infra

The workflow installs deps, runs npm test, deploys via wrangler deploy, then binds the secrets — each value decrypted from secrets/monitoring.env.sops with the CI age key (AGE_KEY_GHA) and piped to wrangler secret put (never echoed). It uses the GH secret CF_API_TOKEN (needs Workers Scripts:Edit scope) and account ID acc5b2c956dd488265003364e2c7f84a. A smoke test confirms GET /queue-issue returns 404 (not 500) after deploy. On failure it sends a Discord alert and opens a GH issue.

Manual (fallback)

cd infra-src/meta-dispatcher
npm install
wrangler deploy

After first deploy, set all 5 secrets (see below).

Rollback:

wrangler rollback

Restores the previous CF Worker version instantly. DB migration rollback (if needed):

DROP TABLE dev_r_orchestrator_runs;
-- Also remove dev_r_services rows if rolling back the registration

Secrets

All 5 secrets are set via wrangler secret put from infra-src/meta-dispatcher/:

Secret nameSourceSOPS file
QUEUE_API_KEYGenerated: python3 -c "import secrets; print(secrets.token_hex(32))"secrets/monitoring.env.sops + secrets/brandpilot.env.sops
SUPABASE_SERVICE_KEYSupabase dashboard → API keys → Secretsecrets/monitoring.env.sops
GH_TOKENGitHub PAT with repo scope (issues + comments read on private repo)secrets/monitoring.env.sops
DISCORD_WEBHOOK_URLP24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URLsecrets/monitoring.env.sops
MEZMO_SERVICE_KEYP24_INFRA_MEZMO_SERVICE_KEYsecrets/monitoring.env.sops
wrangler secret put QUEUE_API_KEY
wrangler secret put SUPABASE_SERVICE_KEY
wrangler secret put GH_TOKEN
wrangler secret put DISCORD_WEBHOOK_URL
wrangler secret put MEZMO_SERVICE_KEY

QUEUE_API_KEY Rotation

  1. Generate a new key: python3 -c "import secrets; print(secrets.token_hex(32))"
  2. Set on Worker: wrangler secret put QUEUE_API_KEY (in infra-src/meta-dispatcher/)
  3. Update secrets/monitoring.env.sops — follow Windows SOPS procedure in docs/playbooks/sops-windows-crlf.md
  4. Update secrets/brandpilot.env.sops — same procedure
  5. Sync to Vercel: vercel env add QUEUE_API_KEY (for BrandPilot integration)
  6. Append rotation entry to docs/secrets-rotation-log.md

The old key stops working immediately after step 2 — coordinate with any callers (BrandPilot, n8n) before rotating.

Cron Schedule

Runs every 4 hours: 0 */4 * * * (00:00, 04:00, 08:00, 12:00, 16:00, 20:00 UTC).

Verify in Cloudflare dashboard → Workers → p24-meta-dispatcher → Triggers tab.

Use wrangler tail to stream live logs during a cron invocation or manual trigger.

Monitoring

Grafana panel — recent orchestrator runs

Add a Table panel to the p24-infra Grafana dashboard using the grafana_readonly PostgreSQL datasource:

SELECT id, triggered_by, status, issues_analyzed, waves_planned, issues_queued,
       issues_skipped, started_at, completed_at
FROM dev_r_orchestrator_runs
ORDER BY started_at DESC
LIMIT 10;

Search app:meta-dispatcher in the Mezmo UI for structured logs from the Worker. Key log lines: "run started", "wave N queued", "run complete", "wave in progress — skipped".

Distributed Lock

The Worker uses a unique partial index on dev_r_orchestrator_runs to prevent overlapping runs:

CREATE UNIQUE INDEX dev_r_orchestrator_runs_one_running
  ON dev_r_orchestrator_runs (status)
  WHERE status = 'running';

A second concurrent trigger attempt gets a 23505 unique violation and aborts cleanly. Stale locks (CF Worker hit the 30s wall-clock limit) are auto-cleaned on the next run start (any running row older than 10 minutes is patched to failed before inserting the new run row).

Wave Logic

  • Fetches all open issues with milestones: Triage, Design, In Progress
  • Enriches In Progress issues only (reads comments to find ## Code-change-design block)
  • Builds a dependency graph from: explicit depends on #N references in body + file-overlap edges
  • Topological sort (Kahn’s algorithm), max 6 issues per wave
  • On the cron trigger: queues first incomplete wave only (default)
  • On POST /trigger with all_waves: true: queues all incomplete waves

Priority formula: (wave_number × 10) + labelAdj + sizeAdj

  • labelAdj: bug=-5, patch=-2, else=0
  • sizeAdj: S=-1, M=0, L=+1, XL=+2
  • Size from AC count in design comment: 1-2→S, 3-4→M, 5+→L; XL if arch-gate keyword in title/body
  • Weight: S→light, M/L/XL→heavy

Role Classification (#1706)

Both singleDispatch() (queue-api) and runMetaDispatcher() (wave builder) call classifyRole() (src/classify.ts) at enqueue time and write role, repo_context, server_preference to each queue row. The worker reads role to select its agent prompt; the dispatcher uses server_preference to route rows to the right server. Full rule table: see worker-queue-operations.md §Role-Based Routing.

Troubleshooting

All endpoints return HTTP 500 / CF error 1101 (issues #1507, #1550)

Symptom: every request to https://p24-meta-dispatcher.radieu.workers.dev/* returns an opaque 500 with a Cloudflare 1101 (“Worker threw exception”) page.

Root cause (2026-06-27): the Worker source was complete but had never been deployed with its secrets bound — there was no deploy workflow, so the Worker either did not exist or ran without QUEUE_API_KEY/SUPABASE_SERVICE_KEY bound.

Fix: run the deploy workflow (gh workflow run deploy-meta-dispatcher.yml), which deploys the code and binds all 5 secrets. Confirm with:

# 404 (not 500) means the handler is reachable and secrets are bound
curl -s -o /dev/null -w '%{http_code}' \
  -H "Authorization: Bearer $QUEUE_API_KEY" \
  "https://p24-meta-dispatcher.radieu.workers.dev/queue-issue?job_id=999999999"

Then verify QUEUE_API_URL in secrets/monitoring.env.sops points to the live host p24-meta-dispatcher.radieu.workers.dev (not a placeholder). Note: the routing handler already wraps all routes in a try/catch that converts downstream throws into a diagnosable JSON 500 with a detail field instead of a bare 1101 (see src/index.ts).

POST/GET /queue-issue returns 401 or 403 (issue #2222)

First, distinguish the two — they have different causes:

  • 401 {"error":"Unauthorized"} comes from the worker itself (verifyApiKey). It means the Authorization: Bearer <key> header was missing, malformed, or did not match the bound QUEUE_API_KEY. The worker has no code path that returns 403.
  • 403 Forbidden on https://p24-meta-dispatcher.radieu.workers.dev/* is a Cloudflare edge response (Bot Fight Mode / WAF custom rule / rate-limit / Access policy) — it never reaches the worker. A 403 is therefore not a key mismatch; check the Cloudflare dashboard (Security → Events) for the blocking rule and the source IP.

Verify the key against the live worker (status code only — never print the key):

QK=$(sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops \
  | grep '^QUEUE_API_KEY=' | cut -d= -f2- | tr -d '"')
# 404 = handler reached + key accepted; 401 = bad key
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Bearer $QK" \
  "https://p24-meta-dispatcher.radieu.workers.dev/queue-issue?job_id=999999999"
unset QK

QUEUE_API_URL must be the BASE host — https://p24-meta-dispatcher.radieu.workers.dev, without a /queue-issue suffix. Callers append the path themselves (${QUEUE_API_URL}/queue-issue, ${QUEUE_API_URL}/trigger); a suffixed value produces a doubled path. (The worker routes on path.endsWith(...) so a doubled /queue-issue is currently tolerated, but /trigger and any future route are not — keep the env var as the base host.)

Misconfiguration is fail-closed (#2222): if QUEUE_API_KEY is unbound or empty, the worker returns a clean 401 rather than crashing to a CF 1101/500 or (for an empty key) accepting an empty Bearer token. If you see 401 on every request, confirm the secret is bound: wrangler secret list for p24-meta-dispatcher.

Endpoint returns HTTP 500 Supabase … Invalid API key (issue #4621)

Symptom: a specific data-path endpoint (e.g. POST /admin/cleanup-failed, seen when nightly-queue-cleanup.yml fails) returns:

HTTP 500: {"error":"internal error","detail":"Error: Supabase UPDATE cleanupFailedJobs: undefined Invalid API key"}

The worker is reachable (routes resolve, try/catch produces a JSON 500 with detail), so this is not the “all endpoints 1101” case above — it is Supabase rejecting the bound SUPABASE_SERVICE_KEY.

Root cause (2026-07-30): the Wrangler SUPABASE_SERVICE_KEY secret on the deployed worker was a stale legacy JWT (eyJ…). Supabase disabled legacy API keys, so every call is rejected with Invalid API key. The current new-format sb_secret_* value already lives in secrets/monitoring.env.sops — only the worker’s bound copy was stale.

Fix: re-bind the worker secrets from SOPS via the deploy workflow (never edit SOPS for this — the value there is already correct):

gh workflow run deploy-meta-dispatcher.yml --repo radieu/p24-infra   # re-binds all 5 secrets
# verify the originally-failing path (dry run — no rows mutated):
gh workflow run nightly-queue-cleanup.yml --repo radieu/p24-infra -f dry_run=true
# expect: HTTP 200: {"ok":true,...,"dry_run":true}

Confirm which SOPS keys are new-format without printing values: … | grep '^SUPABASE_SERVICE_KEY=' | cut -d= -f2- should start with sb_secret_, not eyJ.

  • docs/queue-api-operations.md — HTTP API endpoint reference
  • infra-src/meta-dispatcher/ci/deploy-meta-dispatcher.yml (install to .github/workflows/) — CI/CD deploy + secret binding
  • infra-src/meta-dispatcher/ — source code
  • supabase/migrations/20260625_queue_api.sql — DB schema
  • queue-dispatcher-loop.py (bms-4) — unchanged consumer

Graph View