p24-infra Docs

gh-pat-rotation

11 min read

Playbook: GH PAT Rotation (GH_TOKEN)

Trigger: GithubPatExpiringSoon or GithubPatExpired Prometheus alert, or Discord embed from gh-pat-expiry-check.yml. Audience: p24-infra sys-admin role Related issues: #3650 (monitor), #2750, #2926, #3634 (prior incidents)

GH_PAT is mostly migrated but NOT yet retired — see §0. Since #5841 five workflows that used GH_PAT (secrets-sync.yml’s sync-gh-secrets job and credential-rotation.yml in Part A; n8n-workflow-snapshot.yml, n8n-backup.yml, rotation-log-reimport-check.yml in the Part A follow-up) authenticate with a short-lived GitHub App installation token, not a static PAT. One consumer remains — sops-sync-receiver.yml pushes directly to protected main via a branch-protection bypass, and whether the App installation token is an authorized bypass actor is unverified; until that is resolved it stays on GH_PAT, so GH_PAT still exists and a GH_PAT expiry alert still means “mint a new GH_PAT” (§0). The original issue under-counted GH_PAT as “2 workflows” — it was actually 6 (incl. gh-pat-expiry-check.yml, which monitors it).

GH_TOKEN is now mostly migrated too — see §0.5 (#5841 Part B + third attempt). All but 2 of the workflows that consumed secrets.GH_TOKEN now mint an App installation token at job start — auto-fix-gh-actions.yml joined them 2026-08-09 on the third attempt, and deploy-email-sender-worker.yml (a late-added consumer the earlier sweeps missed — it only uses the token for a failure-path gh issue create) was migrated 2026-08-11, once the real blocker (the installation had never accepted a pending permission upgrade — not just a manifest gap) was fixed. Only 2 deliberately stay on the GH_TOKEN PAT (update-claude-env.yml, gh-pat-expiry-check.yml — reasons in §0.5). So GH_TOKEN still exists but has a much smaller blast radius; full retirement is a secret-manager task pending those 2.

0. GH_PAT is no longer a PAT (migrated to GitHub App installation tokens, #5841)

GH_PAT was a static classic PAT stored as a GH Actions repo secret + in secrets/administration.env.sops. Because GitHub exposes no API to create/revoke PATs, every expiry/rotation forced a human to mint a replacement at github.com/settings/tokens — the recurring toil behind #5524, #5232, #3650, #2750, #2926, #3634, #5715, #5717.

That is now fixed for five of the six consumers, which mint a fresh, auto-expiring (1h) GitHub App installation token at job start via actions/create-github-app-token@v3, using secrets.GH_APP_ID + secrets.GH_APP_PRIVATE_KEY (App 2109526 / installation 142989418, account radieu, repository_selection: all, granted secrets:write + contents:write + issues:write + pull_requests:write, verified 2026-07-20 — see secret-rotation-access-matrix.md §2.4):

WorkflowJob / stepToken scope minted
.github/workflows/secrets-sync.ymlsync-gh-secrets → Generate GitHub App tokenowner: radieu, repositories: p24-infra,et-operational-platform, permission-secrets: write
.github/workflows/credential-rotation.ymlGenerate GitHub App tokencurrent repo (radieu/p24-infra), default app permissions (git push + gh secret set + gh issue/gh pr)
.github/workflows/n8n-workflow-snapshot.ymlGenerate GitHub App token → Commit if diffcurrent repo, default app perms — PR create/merge (contents:write + pull_requests:write)
.github/workflows/n8n-backup.ymlGenerate GitHub App token → Commit if workflows changedcurrent repo, default app perms — PR create/merge
.github/workflows/rotation-log-reimport-check.ymlGenerate GitHub App token → Open PR if the fallback file was drainedcurrent repo, default app perms — PR create/merge

The three follow-up workflows all follow the same PR-to-automation-branch + gh pr merge --auto shape: they never push to main directly (only to a non-protected automation/* branch), so the App token needs only contents:write + pull_requests:write, and the App-created PR fires pull_request events (a scoped GITHUB_TOKEN PR does not) so required checks run and auto-merge proceeds — exactly the property the old GH_PAT provided.

Still on GH_PAT (blocks retirement):

WorkflowWhy it can’t migrate yet
.github/workflows/sops-sync-receiver.ymlPushes rotation commits directly to protected main via a branch-protection bypass (“as credential-rotation-bot”). An App installation token pushes under a different actor identity; whether App 2109526 is an authorized bypass actor for main is unverified (the machine account can’t read branch protection — 403). Migrating blind risks a silent GH006 that would break the credential-rotation delivery path. Resolve the bypass-actor question first (confirm/add App 2109526 as a bypass actor, or convert the bypass to commit-message-pattern based so it is token-agnostic), then migrate.
.github/workflows/gh-pat-expiry-check.ymlIts purpose is to monitor the GH_PAT PAT — retire the workflow when GH_PAT is retired.

There is nothing to rotate for GH_PAT — the App token is regenerated on every run and expires by itself. The same App-token path is also used on-host by bin/gh-app-token.sh / scripts/lib/gh_app_secret_set.sh (bms-4 workers) and by dispatch-to-queue.yml, close-merged-pr-issues.yml, reregister-ionos-runners.yml.

If a sync-gh-secrets / credential-rotation run 403s with “Resource not accessible”: that is an App problem, not a PAT expiry. Verify App 2109526 is still installed on the failing repo with secrets:write (and, for credential-rotation, contents:write) rather than minting a PAT.

Retiring the leftover GH_PAT secret material (removing it from secrets/administration.env.sops and deleting the GH_PAT GH Actions repo secret) is a secret-manager task (tracked in #5842) — and it is blocked until sops-sync-receiver.yml also comes off GH_PAT (see the “Still on GH_PAT” table above), because retiring the secret while that workflow still reads it would break the credential-rotation delivery path on its next run. Until then a monitor on GH_PAT expiry may still fire; treat any such alert as “mint a new GH_PAT” (the migrated 5 consumers don’t care, but sops-sync-receiver.yml still needs a live GH_PAT), not as “the migration is done”.

0.5. GH_TOKEN is now mostly App-token too (migrated, #5841 Part B)

Same root cause, same fix as §0, applied to the broader GH_TOKEN PAT. As of #5841 Part B, 15 workflows mint a fresh App installation token at job start (actions/create-github-app-token@v3, secrets.GH_APP_ID + secrets.GH_APP_PRIVATE_KEY) instead of secrets.GH_TOKEN:

alertmanager-escalation, apply-supabase-migrations, cloudflare-security-check, db-maintenance, db-maintenance-check, deploy-meta-dispatcher, deploy-p24-auth-worker, dispatch-health-check, nightly-devops-triage, nightly-queue-cleanup, prometheus-alerts-ai-triage, provision-new-vps, resource-incident-triage, wasabi-sops-update, weekly-supabase-advisor-audit. (reregister-ionos-runners was already on the App token before Part B.) All of their GH_TOKEN operations — issue/PR create/comment/edit, git push to contents, and the cross-repo runner registration-token calls in provision-new-vps — are covered by the App’s granted issues:write / pull_requests:write / contents:write / administration:write. provision-new-vps uses the cross-repo owner: radieu / repositories: p24-infra,et-operational-platform scoping (like reregister-ionos-runners) because it hits et-operational-platform’s registration-token API.

auto-fix-gh-actions.yml — three attempts, third one succeeded (2026-08-09, #5841):

Attempt 1 (PR #5927 → reverted PR #5949): the App manifest was edited to add workflows: write at github.com/settings/apps/2109526/permissions, so a migration PR requested permission-contents/workflows/issues/actions: write on the token-mint step and merged. The very first live dispatch failed at token-mint with a generic 422 The permissions requested are not granted to this installation. (GitHub does not say which permission when several are requested together in one call). A disposable diagnostic workflow (test/5841-app-token-permission-probe, PR #5940, deleted immediately after use) isolated the exact cause by requesting permission subsets one at a time:

Probe (added to contents:write + issues:write baseline)Outcome
baseline only (contents+issues)✅ success
+ workflows: write❌ failure
+ actions: write (no workflows)❌ failure
default (no permission-* inputs — full current installation grant)✅ success

The actual root cause, found by the probe: editing a GitHub App’s manifest to add a permission does not retroactively re-grant it to an already-existing installation. GitHub instead queues a pending-permissions upgrade that the installing account must separately approve — normally via a prompt at github.com/settings/installations/142989418. Attempt 1 only did the manifest edit, not the installation-side acceptance, so the mint kept 422ing even though the manifest looked correct. actions:write had the same problem plus hadn’t even been added to the manifest yet (the original 2026-07-20 §2.4 gap). Because auto-fix-gh-actions.yml is dispatched automatically by the gh-actions-monitor n8n workflow (infra-src/n8n-workflows/gh-actions-monitor.json) on every real GH Actions failure — live production automation, not dormant — attempt 1 was reverted back to secrets.GH_TOKEN the same session rather than left broken.

Attempt 2: deferred — the human actions identified by the attempt-1 finding (add actions: write to the manifest; separately accept the pending installation-permission-upgrade prompt) had not yet been done. Re-attempting without both would have reproduced the same 422.

Attempt 3 (succeeded, 2026-08-09): both human actions were completed — actions: write was added to the App manifest, and the pending permission-upgrade prompt at github.com/settings/installations/142989418 was accepted (the step attempt 1 was missing). Live verification via bin/gh-app-token.sh --check --permissions <perm> on bms-4 confirmed all four permissions (contents:write, workflows:write, actions:write, issues:write) individually PASS against the installation before the migration was re-applied. The Generate GitHub App token step from attempt 1 was re-applied unchanged (same four permission-* inputs) and the live workflow_dispatch end-to-end run succeeded, including the workflows:write-dependent push and the actions:write-dependent gh calls inside /fix-gh-actions.

Lesson for next time this class of gap appears: “the App manifest lists permission X” is not sufficient evidence that a specific installation can use X. Always verify at the installation level (bin/gh-app-token.sh --check --permissions X, or the PR #5940 probe-workflow pattern) — and if the manifest was just edited, check github.com/settings/installations/<id> for a pending upgrade prompt that still needs a human to accept it, separately from the manifest edit itself.

2 workflows deliberately stay on secrets.GH_TOKEN (documented inline at each usage):

WorkflowWhy it can’t use the App token
update-claude-env.ymlPersists the token into ~/.claude-env as a long-lived GITHUB_TOKEN read by worker sessions for hours/days. App installation tokens expire in ~1h, so they can’t back a static exported credential.
gh-pat-expiry-check.ymlIts purpose is to monitor the GH_TOKEN PAT as the subject under test — swapping the token would check the wrong credential. Retire the whole workflow when GH_TOKEN is retired.

Retiring GH_TOKEN entirely (removing it from secrets/n8n-bms4.env.sops + deleting the GH Actions repo secret) is a secret-manager task and can happen only once those 2 workflows are resolved — either by finding a long-lived alternative for the runner env, or by accepting they stay on a (now-narrower-scope) PAT. Until then, do not delete GH_TOKEN; a GH_TOKEN expiry alert still means “mint a new GH_TOKEN” per §3 below, because the 2 kept workflows depend on it.

1. Identify which PAT is affected

Secret nameScope requiredWho uses it
GH_TOKENrepo, workflow, write:packages2 GHA workflows post-#5841 (Part B + third attempt): update-claude-env, gh-pat-expiry-check (see §0.5). Every other consumer (incl. auto-fix-gh-actions, migrated 2026-08-09 on the third attempt, and deploy-email-sender-worker, migrated 2026-08-11) is on App tokens.
GH_PATrepo (+ secrets:write use)1 GHA workflow post-#5841: sops-sync-receiver.yml (see §0). The other 5 migrated to App tokens; retirement is blocked on this last one.

The Prometheus alert annotation and Discord embed both name the token_name label. A GH_PAT label means sops-sync-receiver.yml still needs a live GH_PAT — mint/rotate it as a real PAT (§3) until that workflow is migrated (§0), do not treat it as fully retired.

2. Confirm the expiry

# Run from any shell with the current GH_TOKEN value
curl -sI -H "Authorization: Bearer $GH_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/user \
  | grep -i 'http\|github-authentication-token-expiration'
  • HTTP 200 + expiry header present → token is still valid, days shown
  • HTTP 401 → already expired

3. Generate a new PAT (GH_TOKEN only)

  1. Open github.com/settings/tokens as the radieu account.
  2. Click Generate new token (classic).
  3. Set expiry: 365 days (do not select “No expiration” — the monitor needs a header to track).
  4. Select scopes for GH_TOKEN: repo, workflow, write:packages.
  5. Copy the generated token value — you only see it once.

4. Update GH Secret

# Replace with the new token value — use --body to avoid trailing newline
gh secret set GH_TOKEN --repo radieu/p24-infra --body "<new-token>"

Do not pipe via stdin — the pipe adds a trailing newline that breaks base64 decode in CI (see memory: feedback_gh_secret_set_newline.md).

5. Update SOPS (for tokens that are also in SOPS)

GH_TOKEN lives in secrets/n8n-bms4.env.sops (used by bms-4 services).

Follow docs/playbooks/sops-edit-operations.md to update the value in SOPS, then:

# Windows — canary before git add
sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }
git add secrets/n8n-bms4.env.sops
git commit -m "chore: rotate GH_TOKEN in n8n-bms4.env.sops (#3650)"

After merge to main, secrets-sync.yml distributes the new value to bms-4 automatically.

6. Verify the alert clears

The gh-pat-expiry-check.yml workflow runs daily at 06:00 UTC. To verify immediately:

gh workflow run gh-pat-expiry-check.yml --repo radieu/p24-infra

Watch the run — it should report the new expiry days and push updated metrics to Pushgateway. The Prometheus alert should resolve within 2 minutes of the next scrape (Pushgateway scrape interval: 60 s).

7. Update the rotation log

Add an entry to docs/credential-rotation-log.md (or equivalent) with:

  • Date of rotation
  • Which token(s) rotated
  • New expiry date
  • Operator

Prevention

  • GH_TOKEN MUST be created with a fixed expiry (365 days max). “No expiration” tokens are invisible to the monitor.
  • The gh-pat-expiry-check.yml workflow runs daily and warns 14 days before expiry — this is the scheduled rotation trigger.
  • Prometheus alerts GithubPatExpiringSoon (warning, <14 d) and GithubPatExpired (critical, ≤0 d) provide additional Grafana/Alertmanager coverage.
  • Most GH_TOKEN consumers were migrated to GitHub App installation tokens in #5841 Part B + the 2026-08-09 third-attempt success (§0.5); only 2 workflows still depend on the PAT. The remaining barrier to full retirement is the need for a long-lived credential in the runner env (update-claude-env) and gh-pat-expiry-check’s role as the PAT’s own monitor. Do not add fresh GH_TOKEN consumers — new workflows that need a token should mint an App token via actions/create-github-app-token@v3 (see the 16 migrated workflows or close-merged-pr-issues.yml as reference).
  • Manifest ≠ installation grant. If a future App-token migration 422s with “permissions requested are not granted to this installation” even though the App manifest lists the permission, check github.com/settings/installations/<id> for a pending permission-upgrade prompt — a manifest edit queues an upgrade, it does not apply it. This exact gap caused auto-fix-gh-actions.yml’s first migration attempt (PR #5927) to fail; see §0.5 for the full three-attempt history.

Graph View

Backlinks