Playbook: AI PR Auto-Review System

Overview

Every dispatcher cycle (every ~2 minutes) queue-dispatcher-loop.py scans open PRs in configured repos and auto-enqueues a review-pr job for each PR that is:

Not labeled human-action (human must decide)
Not labeled ai-review-queued (already in queue / being reviewed)
Not a draft

The review worker (infra/agent-prompts/review-pr-worker.md) runs two-pass analysis (Haiku scout → Sonnet analyst) and then takes one of three actions:

Verdict	Action
APPROVE or COMMENT (only MINOR issues)	Squash-merge PR, delete branch, remove `ai-review-queued` label
REQUEST_CHANGES (CRITICAL or MAJOR issues)	Close PR with explanation, comment on linked issue, remove `ai-review-queued` + add `ai-review-rejected`, re-queue issue for reimplementation
SKIP (draft / CI failing)	Post a comment explaining skip, mark queue row done

Labels

Label	Set by	Meaning
`human-action`	Worker or human	Do NOT auto-review — human decision required
`ai-review-queued`	Dispatcher scanner	PR is in the review queue (prevents double-dispatch)
`ai-review-rejected`	Review worker	PR was closed by AI review; issue re-queued

Add these to a repo’s standard labels via new-repo-setup.md §Step 4 or:

$repo = "radieu/<repo>"
gh label create "ai-review-queued"  --repo $repo --color "0075ca" --description "PR queued for AI auto-review"
gh label create "ai-review-rejected" --repo $repo --color "d93f0b" --description "PR rejected by AI review — issue re-queued"

Configuration

The dispatcher reads two env vars. They must be set in the EnvironmentFile of whichever host is the current dispatch leader (p24-queue-dispatcher.service is HA leader-elected — only the leader runs scan_prs_for_review()). Each host reads a different EnvironmentFile (see the unit header):

Dispatch host	EnvironmentFile the dispatcher reads
vps-i1 (nominal “primary” per the lease note)	`/opt/p24-infra/monitoring/.env`
bms-4 (nominal “backup”; actual lease holder since 2026-07-30)	`/opt/p24-infra/bms-4/.env`

Because the leader can be either host, set both vars in BOTH files so the config survives an HA failover. The #5607 roll-out set them in both (2026-08-05).

Env var	Meaning	Default
`REVIEW_REPOS`	Comma-separated repos to scan for PRs to auto-review. Empty = disable.	`radieu/p24-infra`
`REVIEW_REPO_BASE_BRANCHES`	Comma-separated `repo=branch` overrides for the PR-list base filter (#5607). Any repo not listed defaults to `main`.	(empty)

Current fleet-wide value (#5607) — appended to both /opt/p24-infra/bms-4/.env (leader) and /opt/p24-infra/monitoring/.env (vps-i1, failover):

REVIEW_REPOS=radieu/p24-infra,radieu/et-operational-platform,radieu/Art-Agency,radieu/radekkonarski-personal-brand,radieu/gmail-tools
# et-operational-platform's PRs target `dev`, not `main` — override it (all others default to main):
REVIEW_REPO_BASE_BRANCHES=radieu/et-operational-platform=dev

After editing .env, the next dispatcher cycle (a Type=oneshot timer fire that re-reads the EnvironmentFile) picks both vars up automatically — no restart needed.

⚠️ Known blocker — scanner needs GH_TOKEN on the active leader (found 2026-08-05, #5607). scan_prs_for_review() starts with if not GH_TOKEN or not REVIEW_REPOS: return. GH_TOKEN is present in vps-i1’s /opt/p24-infra/monitoring/.env but absent from bms-4’s /opt/p24-infra/bms-4/.env (the isolated secrets/n8n-bms4-gh.env.sops GH_TOKEN is deployed by secrets-sync.yml into worker/role env files, not the dispatcher’s EnvironmentFile). Since bms-4 has held the dispatch lease since 2026-07-30 (vps-i1 healthy but logs “Not leader” every cycle — no preemption toward the nominal primary), the scanner has been inactive fleet-wide — zero review-pr rows enqueued since 2026-07-30. p24-infra’s own reviews were unaffected only because they also arrive via the event-driven dispatch-to-queue.yml (pull_request trigger), a path the other repos do not have. So the REVIEW_REPOS config above is correct but inert until a GH_TOKEN-bearing host is the scan leader. Fix options (a decision-stop, escalated on #5607): (A) deliver GH_TOKEN to the bms-4 dispatcher EnvironmentFile (secret-manager: add to the SOPS source feeding bms-4/.env, or add the role/worker-keys env as a second unit EnvironmentFile=); (B) restore vps-i1 (which has GH_TOKEN) as the dispatch leader; or (C) a code fallback in queue-dispatcher.sh to export GH_TOKEN from an existing bms-4 token when unset. Until one lands, verify via the event path on p24-infra only.

Prerequisites when adding a repo (do not silently add to REVIEW_REPOS):

Confirm the repo’s current PR-target branch from its own CLAUDE.md §Branching — do not assume. Verified 2026-08-05: radieu/et-operational-platform targets dev (its CLAUDE.md: “Każdy nowy branch feature/fix tworzymy od dev. PR celuje w dev”), so it must be listed in REVIEW_REPO_BASE_BRANCHES. Art-Agency, radekkonarski-personal-brand and gmail-tools default to main.

Confirm the repo is wired for job_type=review-pr execution (correct role/job_profile per docs/playbooks/cross-repo-worker-isolation.md) before adding it — otherwise the enqueued review job fails at spawn.

radieu/brandpilot is intentionally excluded (2026-08-05): it is not resolvable by the dispatcher’s gh/GH_TOKEN (both radieu/brandpilot and p24-infra/brandpilot return “Could not resolve to a Repository”). Resolve the repo location/access first, then add it.

Add the standard labels (see §Labels) to the repo before or when adding it — scan_prs_for_review() sets ai-review-queued, and the review worker sets ai-review-rejected.

Base-branch filter (per-repo since #5607). scan_prs_for_review() lists PRs with a base branch resolved per repo: REVIEW_REPO_BASE_BRANCHES.get(repo, "main"). Repos not in the map use main (this repo’s policy — CLAUDE.md §Branching). Before #5607 the filter was hardcoded base=main for every repo (and base=dev before #4579), so it silently no-oped — returning zero PRs with no error — for any repo whose PRs target another branch. That is why et-operational-platform=dev is required. The event-driven pull_request handler in .github/workflows/dispatch-to-queue.yml has no branches: filter and is unaffected. scripts/tests/test_scan_prs_base_branch.py locks the default, the per-repo lookup, and the no-hardcoded-branch rule.

Should REVIEW_REPOS move into SOPS? These two vars are non-secret config but live only in the hand-edited on-server .env, so a monitoring redeploy that regenerates .env from secrets/monitoring.env.sops would drop them. Moving them into secrets/monitoring.env.sops (a secret-manager operation) would make them survive a redeploy. Flagged as a judgment call for secret-manager/sys-admin — not actioned by this change.

Priority of review-pr jobs: 15. Both insertion paths must stay aligned to this value — scan_prs_for_review() here and the pull_request handler in .github/workflows/dispatch-to-queue.yml. Before #3526 they had drifted apart (15 vs 20); since the GH Actions path fires immediately on PR events (ahead of the ~2min scanner cycle), the drift meant review-pr jobs effectively ran at 20 in practice.

This sits between the default (unlabeled) dev-issue priority (12) and the P2 dev-issue tier (50): P1 (10) < default dev-issue (12) < review-pr (15) < P2 (50). Reviews no longer crowd out unlabeled implementation work, but still outrank the lowest-priority dev-issue backlog.

Flow diagram

Dispatcher cycle (every 2 min)
  └─ scan_prs_for_review()
       ├─ gh pr list (open, base=REVIEW_REPO_BASE_BRANCHES.get(repo, "main"), repo=REVIEW_REPOS)
       ├─ skip: human-action | ai-review-queued | draft | already in queue
       ├─ INSERT dev_r_worker_queue (job_type=review-pr, priority=15)
       └─ add label ai-review-queued to PR

Worker (bms-4 or vps-i1)
  └─ review-pr-worker.md
       ├─ Phase 0: Haiku scout (pre-scan, CI check, diff)
       ├─ Phase 1: Sonnet deep analysis
       ├─ Phase 2: verdict logic
       ├─ Phase 3: post GitHub review comment
       ├─ Phase 3.5: ACTION
       │    ├─ APPROVE / COMMENT → gh pr merge --squash --delete-branch
       │    └─ REQUEST_CHANGES  → gh pr close
       │                          → gh issue comment (linked issue)
       │                          → INSERT dev_r_worker_queue (issue re-queue)
       │                          → label ai-review-rejected
       └─ Phase 4: cleanup + mark queue row done

Monitoring

Grafana: worker-queue dashboard — filter job_type=review-pr to see review jobs.

Supabase: dev_r_worker_queue where job_type = 'review-pr' — result_summary field shows outcome (merged / rejected+requeued / skipped).

GitHub PR labels: ai-review-queued (in progress) → removed on completion. ai-review-rejected (blocked PRs that need rework).

What triggers human-action (escapes auto-review)

The review worker sets human-action on a PR’s linked issue (not the PR itself) when:

OOM / spawn failure exceeds max retries (set by dispatcher)
The re-queued issue fails 3+ times

The dispatcher sets human-action on the linked issue when:

spawn_failures > MAX_SPAWN_FAILURES

A PR itself is never labeled human-action by the review worker — it is either merged or closed. If neither worked, the queue row is marked failed and Discord alerts fire.

Disabling for a PR

To opt a specific PR out of auto-review, add the human-action label to it. The scanner skips any PR with this label.

Disabling globally

Set REVIEW_REPOS= (empty string) in /opt/p24-infra/monitoring/.env on vps-i1. The dispatcher scan_prs_for_review() no-ops when the list is empty.

Escalation

Review job stuck in queued > 10 min: check dev_r_server_capacity — all slots may be full with heavier jobs. review-pr jobs are light and should run quickly once a slot opens.

Merge fails after APPROVE: likely a branch protection rule (required reviews, CI gate). Check gh pr view ${PR_NUMBER} for merge errors. The review worker will log the error and mark the queue row done (not failed) — the PR stays open for human merge.

Confirmed live (2026-08-09, PR #6035): when the bot’s own review can’t be posted as an approval (Can not approve your own pull request — the reviewing identity and the pushing identity are the same underlying GitHub account/App), the worker posts the verdict as a comment instead, and gh pr merge fails with New changes require approval from someone other than the last pusher — even with --admin. This is a hard GitHub-side limit, not a bug in this pipeline: a genuinely different reviewing identity is required to satisfy require_last_push_approval, and none is currently wired up. Until a second, distinct bot identity is added for review-only use, this class of PR always needs a human merge click — don’t loop retrying gh pr merge on it.

Re-queue loop: if an issue keeps getting rejected, check spawn_failures in dev_r_worker_queue — after MAX_SPAWN_FAILURES (5) the dispatcher escalates to human-action and stops re-dispatching.

p24-infra Docs

Explorer

ai-pr-auto-review