p24-infra Docs

plan-4245-gh-app-token-fleet-migration

15 min read

Plan: GH Actions fleet → App token + agent gh throttling (rate-limit fix)

Issue: #4245 Type: Code-change-design (plan only) Status: Draft — iteration 1 Last updated: 2026-07-17 Author: Claude Opus 4.8 (worker session i1-cw-1 on vps-i1) Related: #4065 (parent App-migration plan) · #4068 (Wave A — bms-4 workers) · #4180 (queue-exporter exhaustion) · #4139 (existing triage backoff)

0. Summary — recommendation

The issue’s premise does not survive measurement, and the recommendation changes as a result.

#4245 is framed as: ~40 workflows drain one 5000/hr PAT bucket → migrate the Actions fleet onto App 2109526 to fix it. Two of those three clauses are wrong:

Claim in #4245MeasuredVerdict
”~40 workflows … drain that ONE bucket”18 workflows reference secrets.GH_TOKEN on a non-comment line; 16 are migratable here❌ over-counted ~2.2×
“5 high-frequency crons” to prioritise3 of the 5 burn the PAT; 2 use the built-in token❌ 2 false positives
Migrating the fleet fixes 0/5000 dailyFleet core burn ≈ 20–100 req/hr (0.4–2% of budget). queue-exporter alone burns ≈ 21,540 req/hr = 430% of the entire budget❌ fixes <2% of the problem

The rate limit is not an Actions-fleet problem. It is a queue-exporter problem, and it is already owned by #4180. Migrating all 16 workflows while queue-exporter runs unchanged leaves the bucket at ~430% over budget — the 0/5000 symptom would persist unchanged. Conversely, landing #4180 alone drops burn to ~2% of budget and the symptom disappears without migrating a single workflow.

So the recommendation is a re-ordering and a re-justification, not a cancellation:

  1. Land #4180 first — it is the actual fix. Do not gate it on anything in #4245. Until it lands, every other number here is noise against a 4.3× overrun.
  2. Keep Move 1 (fleet → App token), but re-justify it under #4065, not capacity. The migration is still worth shipping — it removes 16 consumers from the PAT-rotation surface (#4065’s actual goal), cuts blast radius from a long-lived PAT to a 1h-TTL token, and unblocks PAT deprecation. It should be **re-parented under #4065 as a Wave A sibling of 4068, and explicitly de-scoped from the rate-limit narrative.
  3. Ship Move 2 (throttling guard) as an extraction, not an invention. A tested bounded-backoff implementation already exists (scripts/prometheus-alerts-ai-triage.py, added by #4139, tests in scripts/tests/test_triage_rate_limit.py). Extract it to scripts/lib/gh_rate_limit.py and reuse. Do not write a second one.
  4. Drop Move 2 item 5 (cron back-off) as a rate-limit measure. Halving */10 → */30 on the two */10 crons saves ~12 req/hr against a 21,540 req/hr overrun. It is a detection-latency regression bought for nothing. If those crons should be slower, argue it on cost/noise grounds in a separate issue.

Decision required from the human reviewer (ARCH GATE, §8): given Move 1 does not fix the rate limit, do we still want it now under the #4065 PAT-elimination banner, or does it defer until after #4180 + #4068 land?

1. Evidence

All counts from branch plan/4245-gh-app-token-fleet-migration at origin/main, 2026-07-17.

1.1 The fleet is 18 references / 16 migratable, not ~40

68 workflows total in .github/workflows/. Token references, after stripping comments:

# true PAT consumers — secrets.GH_TOKEN on a non-comment line
for f in .github/workflows/*.yml; do
  hits=$(sed 's/#.*//' "$f" | grep -cE 'secrets\.GH_TOKEN')
  [ "$hits" -gt 0 ] && echo "$f"
done | wc -l
# => 18
BucketCountConsumes the PAT 5000/hr?
secrets.GH_TOKEN (the PAT)18✅ yes
├─ must stay on the PAT1 (gh-pat-expiry-check.yml)✅ by design — see §1.4
├─ not App-compatible, re-route to #40681 (update-claude-env.yml)✅ — see §1.6
└─ migratable here16✅
secrets.GITHUB_TOKEN only (built-in)14❌ separate per-repo bucket
Already on App 21095263❌ App installation bucket
No token at all33❌

The ~18, not ~40 magnitude is independently corroborated by the repo’s own alert text — gh-pat-expiry-check.yml:142 already says “All ~17 GHA workflows using secrets.GH_TOKEN are failing”. The “~40” in #4245 appears to come from grep -l GH_TOKEN matching the env-var name GH_TOKEN: rather than the secret reference secrets.GH_TOKEN — the same conflation that produces the two false positives in §1.2.

1.2 Two of the five “priority” workflows are false positives

atrax-data-freshness.yml and nc-alert-instant-dispatch.yml both do this:

env:
  GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}   # env var NAMED GH_TOKEN, VALUE = built-in token

The env var is named GH_TOKEN (because that is what the gh CLI reads) but its value is secrets.GITHUB_TOKEN — the built-in Actions token, which has its own 1,000/hr per-repository bucket and never touches the user PAT. GITHUB_TOKEN cannot be shadowed by a repo secret (GitHub reserves the GITHUB_ prefix), so this is unambiguous.

Corrected high-frequency PAT burner list — 3, not 5:

WorkflowCronRuns/hrPAT?
dispatch-health-check.yml*/106✅
prometheus-alerts-ai-triage.yml*/154✅
alertmanager-escalation.yml*/154✅
atrax-data-freshness.yml*/106❌ built-in
nc-alert-instant-dispatch.yml*/154❌ built-in

reregister-ionos-runners.yml is a fourth false positive — it is already migrated; its two secrets.GH_TOKEN hits are comments at lines 67 and 89 describing the pre-migration state.

1.3 The fleet is not the burn — queue-exporter is

First — “the bucket” is not one bucket. gh api rate_limit on this account returns 14 independent resources. The three that matter here:

ResourceLimitWho hits it
core5,000/hrPOST/PATCH /repos/{r}/issues*, GET /repos/{r}/issues/{n}, /actions/runners
search30/min/search/issues — the dedup lookups in both triage scripts
actions_runner_registration10,000/hrprovision-new-vps.yml runner tokens

The 0/5000 symptom is a core exhaustion. The triage scripts’ dedup /search/issues calls do not touch core at all (scripts/prometheus-alerts-ai-triage.py:249,418; scripts/alertmanager-escalation.py:144,288), so they are excluded from the core column below.

Amortised hourly core burn:

ConsumerCalccore req/hr
dispatch-health-check6 runs/hr × ~3 calls~18
prometheus-alerts-ai-triage4 runs/hr × (1 POST/PATCH per firing alert; searches are not core)~0–40
alertmanager-escalation4 runs/hr × same shape~0–40
13 other PAT workflowsdaily / weekly / manual~0 amortised
Actions fleet subtotal~20–100 (0.4–2% of 5,000)
queue-exporter359 issues × 60 cycles/hr (app.py:476 time.sleep(60)), all GET /repos/{r}/issues/{n} → core~21,540 (430%)

queue-exporter reads the same PAT — monitoring/exporters/queue-exporter/app.py:233 → os.environ.get('GH_TOKEN'), wired in monitoring/docker-compose.yml as GH_TOKEN=${GH_TOKEN} from /opt/p24-infra/monitoring/.env. #4180 evidences it directly: 9,922 × HTTP 403 Forbidden in 2h, and a clean hourly sawtooth on p24_worker_queue_analyst_escalated_stale.

The conclusion is robust to large estimate error. The fleet numbers above are bounded, not exact — the per-run count depends on how many alerts are firing. But even at a 10× over-estimate (~1,000 core req/hr, 20% of budget) the fleet still cannot produce a 0/5000, and queue-exporter still exceeds the entire budget 4.3× on its own. No plausible refinement of the fleet estimate changes the recommendation.

One consumer is ~99% of core burn, and it is not in this issue’s scope.

1.4 gh-pat-expiry-check.yml must NOT be migrated

It uses the PAT as the subject under test, not as a credential of convenience:

HTTP_RESPONSE=$(curl -sI -H "Authorization: Bearer $TOKEN" ...)
EXPIRY_HEADER=$(echo "$HTTP_RESPONSE" | grep -i '^github-authentication-token-expiration:' ...)

Swapping $TOKEN to an App token would report the App token’s expiry (always ~1h) instead of the PAT’s, silently destroying the p24_github_pat_expires_days metric and the expiry alert. It already correctly uses secrets.GITHUB_TOKEN for its own gh CLI alerting (line 134). Exclude permanently.

1.5 Agent gh calls only partly share the PAT bucket

This worker’s gh CLI authenticates as AI-Dev-IO1 — a distinct machine account with its own 5000/hr bucket, measured at 5000/5000 remaining mid-session. So “Claude agents’ gh CLI … drain that ONE bucket” is not true for the gh CLI path on vps-i1.

Two caveats keep it partly true:

  • update-claude-env.yml writes export GITHUB_TOKEN=<the PAT> into a persistent $HOME/.claude-env on each runner (§1.6), and the claude-env-sync Ansible role does the same across vps-i1 / bms-4 / vps-h1. Any agent code path that reads $GITHUB_TOKEN (rather than shelling to gh) does hit the PAT bucket.
  • The bms-4 telegram-claude-bot authenticates with radieu’s personal owner-level PAT (priorities.md P1).

Move 2 is worth shipping as defence-in-depth, but it is not load-bearing for the 0/5000 symptom — §1.3 is.

1.6 update-claude-env.yml cannot take an App token

It is not a gh consumer — it is a PAT distributor:

runs-on: ${{ inputs.runner_label }}
env:
  GITHUB_TOKEN_SECRET: ${{ secrets.GH_TOKEN }}
run: |
  ENV_FILE="$HOME/.claude-env"
  cat > "$ENV_FILE" <<EOF
  export GITHUB_TOKEN=${GITHUB_TOKEN_SECRET}
  EOF

$HOME persists on a self-hosted runner, and agents source this file for the life of the machine. An App installation token has a 1-hour TTL — writing one here produces a credential file that is dead an hour later and silently 401s every agent that sources it. A long-lived credential written to a persistent file is structurally the opposite of what an App token is.

The correct fix is on-demand minting, which is exactly bin/gh-app-token.sh in #4068 §Scope. This workflow therefore belongs to #4068’s agent-credential class, not to this issue’s Actions-fleet class. Re-route, do not migrate. (Same applies to the claude-env-sync Ansible role.)

1.7 What #4245 gets right

  • The owner:/repositories: scoping trap is real. close-merged-pr-issues.yml and dispatch-to-queue.yml both omit owner:/repositories: and therefore silently scope to the current repo only. reregister-ionos-runners.yml sets them correctly (owner: radieu, repositories: et-operational-platform,amazon-kdp-tango,p24-infra). Any migration must set both.
  • One App is the right answer. All tokens from one App share one installation bucket, so a 2nd App buys isolation, not capacity. Correct — and given §1.3, capacity was never the constraint.
  • Issues stay on GitHub. The rejected Supabase alternative is correctly rejected.

2. Changes required

Move 1 — fleet → App token (16 workflows, re-justified under #4065)

Pattern per workflow — insert a token step, swap the consumer reference:

- name: Generate GitHub App token
  id: app-token
  uses: actions/create-github-app-token@v1
  with:
    app-id: ${{ secrets.GH_APP_ID }}
    private-key: ${{ secrets.GH_APP_PRIVATE_KEY }}
    owner: radieu                 # REQUIRED — default scopes to current repo only
    repositories: p24-infra       # REQUIRED — widen only where the workflow needs it
 
- name: <existing step>
  env:
    GH_TOKEN: ${{ steps.app-token.outputs.token }}   # was: ${{ secrets.GH_TOKEN }}
WaveFilesWhat changesWhy
1 — high-frequency (3)dispatch-health-check.yml, prometheus-alerts-ai-triage.yml, alertmanager-escalation.ymlApp-token step + GH_TOKEN env swapHighest run count → fastest signal on regressions
2 — scheduled (5)db-maintenance.yml, db-maintenance-check.yml, nightly-queue-cleanup.yml, cloudflare-security-check.yml, nightly-devops-triage.ymlsameDaily/weekly; failure is visible next cycle
3 — event/manual (8)apply-supabase-migrations.yml, auto-fix-gh-actions.yml, deploy-meta-dispatcher.yml, deploy-p24-auth-worker.yml, resource-incident-triage.yml, provision-new-vps.yml, update-vps-h1-sops.yml, wasabi-sops-update.ymlsameManual trigger → verify on demand
excluded (2)gh-pat-expiry-check.ymlno change§1.4 — PAT is the subject under test
update-claude-env.yml**re-route to 4068§1.6 — 1h TTL incompatible with a persistent credential file

Waves 1+2+3 = 3 + 5 + 8 = 16, plus 2 excluded = 18 — reconciles with §1.1.

Runner risk is already retired. 9 of the 18 run on [self-hosted, bms4]. dispatch-to-queue.yml and close-merged-pr-issues.yml already run create-github-app-token@v1 on that exact runner today — the Node action is proven there. No runner provisioning needed.

update-vps-h1-sops.yml — verify it is still live before migrating. It checks out ref: dev (CLAUDE.md: “dev branch kept for historical reference only — no new work targets it”) and its PR body says “Closes #232”. It looks stale. Confirm it still runs; if dead, delete it rather than migrate it — that is strictly cheaper and removes a PAT consumer outright.

Move 2 — extract the existing backoff guard

FileWhat changesWhy
NEW scripts/lib/gh_rate_limit.pyExtract _rate_limit_delay() + RATE_LIMIT_MAX_RETRIES / RATE_LIMIT_MAX_WAIT_S + the bounded-retry wrapper verbatim from scripts/prometheus-alerts-ai-triage.py#4139 already built and tested this. One implementation, not two.
scripts/prometheus-alerts-ai-triage.pyImport from the new lib; delete the inlined copyBehaviour-preserving refactor
scripts/alertmanager-escalation.pyAdopt the lib in gh_request()Currently unguarded; same 403 exposure
scripts/tests/test_triage_rate_limit.pyRe-point at scripts/lib/gh_rate_limitExisting coverage carries over
NEW scripts/tests/test_gh_rate_limit.pyDirect unit tests for the extracted libLib is now a shared contract

Guard semantics (already proven by #4139, preserve exactly): honour Retry-After, else X-RateLimit-Reset when X-RateLimit-Remaining: 0; retry ≤3× with ≤30s wait; on persistent exhaustion skip remaining work and exit 0 — a rate limit is transient and must not mark the workflow failed.

Rejected — the <500 remaining pre-flight check proposed in #4245. A pre-flight gh api rate_limit costs a call to save a call and races the actual request. Reacting to the 403/429 the API already sends is strictly better and is what #4139 shipped.

Move 2 item 4 — per-session caching helper

Defer. No measured evidence of repeat-fetch cost, and §1.5 shows agents on vps-i1 have their own unexhausted bucket. Re-raise with a measurement if it ever matters.

Move 2 item 5 — cron back-off

Drop. */10 → */30 on dispatch-health-check saves ~12 req/hr against a 21,540 req/hr overrun, at the cost of tripling detection latency on the dispatch health path. Not a rate-limit measure. (atrax-data-freshness, the other */10, does not touch the PAT at all — §1.2.)

New files

scripts/lib/gh_rate_limit.py, scripts/tests/test_gh_rate_limit.py.

DB schema changes: no.

3. Prerequisite — App permission audit (inherited from #4065 §2.4)

#4065 §2.4 requires confirming the installation grants every permission its consumers rely on before the migration PR opens. For this fleet specifically:

WorkflowNeedsStatus
most (issue create/comment/close)issues: write✅ per #4065 §2.4
apply-supabase-migrations.yml, deploy-*contents: write, actions: write✅ per #4065 §2.4
update-vps-h1-sops.yml, wasabi-sops-update.ymlcontents: write + pull_requests: write — they pass the PAT as actions/checkout token: and open a PR✅ per #4065 §2.4
provision-new-vps.ymladministration: write (runner registration token)⚠️ accepted 2026-07-17 (#4238/#4221) — verify live, acceptance ≠ exercised

Verified: no workflow in this fleet calls gh secret set or the /actions/secrets API, so secrets: write is not required. (An earlier draft of this plan assumed it was — checked and withdrawn.)

Record the audit output in the implementation PR, per #4065 §2.4.

4. Regression risks

RiskWhy it could breakMitigation
Un-scoped App token 403s on foreign reposDefault scope is current-repo-only. This exact trap already hides in close-merged-pr-issues.yml / dispatch-to-queue.yml.Set owner: radieu + explicit repositories: on every step. Grep the PR for a create-github-app-token block missing either key.
1h token TTL expires mid-runalertmanager-escalation.yml / prometheus-alerts-ai-triage.yml export the token over SSH into vps-i1 and run a Python script there.Both scripts start immediately and run ≪1h — safe. Generate the token in the same job that consumes it; never pass one across jobs or persist it. Net security win: blast radius drops from a long-lived PAT to 1h.
gh-pat-expiry-check.yml migrated by a mechanical sweepA sed-style bulk swap would silently break the expiry metric (§1.4).Explicit exclusion list; assert secrets.GH_TOKEN still present in that file as a test.
administration: write accepted but never exercisedprovision-new-vps.yml mints a runner registration token. The scope was accepted 2026-07-17 but acceptance is not proof it works.Dry-run it in Wave 3 (§5) before removing its PAT path.
**Double-migration with 4068Both touch the App. update-claude-env.yml is a genuine overlap — it distributes the PAT to agents, which is #4068’s bin/gh-app-token.sh domain, not the Actions-fleet domain (§1.6).Exclude update-claude-env.yml here and hand it to #4068. secrets-sync.yml is #4068’s and is untouched here (it references no secrets.GH_TOKEN today — verified). No other file is shared.
update-vps-h1-sops.yml may be deadChecks out ref: dev (deprecated) and closes an issue from the #232 era.Confirm liveness; delete rather than migrate if dead.
PAT deprecation before consumers are gonequeue-exporter + telegram bot still read the PAT after this lands.Do not remove GH_TOKEN from SOPS/GH Secrets at the end of this issue. Removal is gated on #4180 + #4068 + telegram-bot resolution.

5. Manual test checklist

  • gh workflow run dispatch-health-check.yml → run green, issue create/comment path exercised
  • prometheus-alerts-ai-triage.yml + alertmanager-escalation.yml dry-run (-f dry_run=true) → SSH step reaches vps-i1, script authenticates, no 401/403
  • provision-new-vps.yml dry-run → runner registration token still issues (administration: write confirmed live, not just accepted)
  • gh-pat-expiry-check.yml unchanged → p24_github_pat_expires_days{token_name="GH_TOKEN"} still populates on the pushgateway
  • update-claude-env.yml unchanged in this PR → agents’ $HOME/.claude-env still sources a working token (§1.6)
  • python -m pytest scripts/tests/test_gh_rate_limit.py scripts/tests/test_triage_rate_limit.py -v → pass
  • ruff check scripts/ → clean
  • Grep guard: every create-github-app-token block in the PR has both owner: and repositories:
  • No secrets.GH_TOKEN remains outside gh-pat-expiry-check.yml and comments

6. Dependencies & sequencing

#4180  queue-exporter throttle   ──► THE rate-limit fix. Land first, independently.
#4068  Wave A — bms-4 workers    ──► owns secrets-sync.yml + git-credential + rotate/*.js
         └─ ALSO take update-claude-env.yml + claude-env-sync role (§1.6 — needs
            on-demand bin/gh-app-token.sh, not a static token in a file)
#4245  this issue                ──► 16 GH Actions workflows + scripts/lib/gh_rate_limit.py
         └─ re-parent under #4065 as a Wave A sibling; de-scope from the rate-limit narrative

After moving update-claude-env.yml to #4068, no file is touched by both issues. #4245 does not block #4180 and must not be sequenced ahead of it.

7. Cost/benefit — why still ship Move 1

With the capacity argument gone, Move 1 stands on #4065’s actual goal:

  • Removes 16 of the 18 secrets.GH_TOKEN call sites in .github/workflows/ → materially advances #4065’s “cut the PAT-consumer count”. (#4065 §2.3 counts consumer classes, not call sites; the Actions fleet is not itemised there, so this is net-new coverage relative to #4068.)
  • Replaces a long-lived PAT with 1h-TTL tokens in 16 places → smaller blast radius on the next exposure (cf. 2824, both still open P0).
  • App tokens need no browser/2FA → the migrated set becomes Tier 1 autonomous, no human-action on rotation.
  • Mechanical, reversible, per-workflow. Low risk once §3’s audit passes.

It is a security/autonomy win, not a capacity win. Ship it on that basis or not at all.

8. ARCH GATE — human decision required

worker-issue.md §Step 3 flags migrate as an architecture-gate keyword, and this plan’s findings make that gate substantive rather than procedural: the issue’s stated goal (fix the rate limit) is not achieved by its stated method (migrate the fleet).

Decide before any implementation issue opens:

  1. Accept the re-frame? Move 1 re-justified under #4065 (PAT elimination), explicitly not a rate-limit fix.
  2. Sequence: land #4180 first (recommended), or run in parallel?
  3. **Re-parent #4245 under 4065 as a Wave A sibling of #4068, or keep standalone?
  4. Accept the de-scopes? Move 2 item 4 (caching) deferred, item 5 (cron back-off) dropped.
  5. Hand update-claude-env.yml (+ claude-env-sync Ansible role) to #4068? §1.6 — it needs on-demand minting, not a token in a file. #4068’s scope grows slightly.
  6. update-vps-h1-sops.yml — confirm dead and delete, or keep and migrate?

No workflow YAML changes in this PR — this is the plan artifact only.

9. File-overlap signature

docs/plans/plan-4245-gh-app-token-fleet-migration.md (this PR — plan only).

Implementation issues, once approved, would touch the 16 migratable workflows — .github/workflows/{dispatch-health-check,prometheus-alerts-ai-triage,alertmanager-escalation,db-maintenance,db-maintenance-check,nightly-queue-cleanup,cloudflare-security-check,nightly-devops-triage,apply-supabase-migrations,auto-fix-gh-actions,deploy-meta-dispatcher,deploy-p24-auth-worker,resource-incident-triage,provision-new-vps,update-vps-h1-sops,wasabi-sops-update}.yml — plus scripts/lib/gh_rate_limit.py, scripts/prometheus-alerts-ai-triage.py, scripts/alertmanager-escalation.py, scripts/tests/test_gh_rate_limit.py, scripts/tests/test_triage_rate_limit.py.

Not touched here: gh-pat-expiry-check.yml (§1.4), update-claude-env.yml + ansible/roles/claude-env-sync (§1.6 → #4068), secrets-sync.yml (#4068), monitoring/exporters/queue-exporter/ (#4180).

Auto-generated by Claude Code worker i1-cw-1 (session on vps-i1) for #4245.

Graph View

Backlinks