gh-actions-failed-jobs-taxonomy

GH Actions Failed Jobs — Recurring Failure Taxonomy

Living document. Update after each “failed jobs” review session. Last reviewed: 2026-09-14 (#6997 — added "Password Rotation Check" to SKIP_WORKFLOWS in gha-failed-jobs-audit.py; its exit-1 is a Group X true-positive rotation alert already owned by the CredentialRotationOverdue issue, so the daily audit’s duplicate [failed-gh-actions] issue was pure noise — same treatment as Group Y’s #6589) Previously: 2026-09-03 (added Group Z — trivy Permission denied exit-126 shared-$HOME parallel-install race in Image CVE Scan; 22-image matrix on bms-4’s 3 shared-$HOME runners all wrote ~/.local/bin/trivy-bin/trivy at once; fixed #6835 with per-job $RUNNER_TEMP install + skip-setup-trivy, same class as Groups L/S) Previously: 2026-08-05 (added Group W — startup_failure runs, 0 jobs / no logs, GitHub emitting conclusion: failure on push to non-default branches for workflow_dispatch-only / path-filtered workflows like wasabi-sops-update.yml; the audit now skips these off-main and no longer files “unknown — log analysis required” noise, #5528) Previously: 2026-07-20 (added Group S — shared-$HOME race on ~/.ssh/id_bms1 between the concurrent bms-1 sync jobs in secrets-sync.yml; sibling job’s if: always() scrub rms the shared key mid-SSH, #4381 stale re-surface, already fixed by #4376) Previously: 2026-07-20 (added Group R — gh issue create --label aborts when the label is missing, ansible-drift.yml reporter swallowed its own drift signal, #4377) Previously: 2026-07-19 (added Group Q — compose container-name conflict on a container the compose project does not own, sync-pinbox24-backends/mailgun-prod, #4290; previously listed as a still-open failure under Group L) Previously: 2026-07-12 (added Group P — sops --encrypt input path misses .sops.yaml creation rule, 3880; added Group O — bot auto-commit push rejected by main branch protection GH006, 3640; Group N is sops-sync-receiver exit-2 canary invalid dotenv input line: {, #3633; Group M is GH API rate-limit 403 in sync-gh-secrets, #3697; Group L is Text file busy sops install race, #3625)

---

How to use this playbook

1. Open GH Actions → filter by status: failure
2. For each failing workflow, match against a Group below
3. Apply the listed fix; record any new root cause at the bottom

Quick lookup:

gh run list --status failure --limit 30
gh run view <id> --log-failed

---

Group A — Python test regression in test_sync_n8n_supabase_creds.py

Symptom:

FAILED scripts/tests/test_sync_n8n_supabase_creds.py::TestExportCredential::test_happy_path_returns_dict
FAILED scripts/tests/test_sync_n8n_supabase_creds.py::TestExportCredential::test_retries_empty_export_then_succeeds
RuntimeError: exported credential file empty or unreadable

Affected workflows: Python Tests, P24-Infra Self Test, n8n Workflow Backup, any workflow with pytest step

Root cause pattern: The sync-n8n-supabase-creds.py script was refactored but the test mocks were not updated to match the new export_credential API. Mock doesn’t return a non-empty file so retry exhaustion raises RuntimeError.

History:

• 2026-06-30 morning: ~10 failed runs (both bms-4 and GH-hosted runner)
• Fixed by PR #2193 (fix(#2122): update export_credential tests for #2120 single-exec refactor) merged ~10:24

Fix: Update test mocks to match new export_credential signature. Prevention: When refactoring a function that has unit tests, always run pytest -x locally on the test file before pushing.

---

Group B — Missing SSH key for backup jobs

Symptom:

getKeyFile error: open /.ssh/grafana_backup_key: no such file or directory
ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
Process completed with exit code 1.

Affected workflows: Grafana Backup, n8n Backup (any using appleboy/drone-ssh)

Root cause pattern: The GH Secret GRAFANA_BACKUP_SSH_KEY (or equivalent) is missing or empty. The workflow writes it to ~/.ssh/grafana_backup_key but there is nothing to write.

History:

• 2026-06-30: Grafana Backup failing (run 28426550041)

Fix:

1. Check the GH Secret exists: gh secret list | grep GRAFANA_BACKUP_SSH_KEY
2. If missing: generate or retrieve the SSH private key for backup user on vps-i1, re-add via gh secret set GRAFANA_BACKUP_SSH_KEY
3. The secret must be the raw private key content (PEM format, no passphrase)

Prevention: After any credential rotation affecting backup SSH keys, immediately re-run the affected workflow manually to confirm.

---

Group C — Expired/rotated n8n API key

Symptom:

curl: (22) The requested URL returned error: 401
# n8n API endpoint returns 401 Unauthorized

Affected workflows: n8n Workflow Snapshot, n8n Backup

Root cause pattern: BMS4_N8N_API_KEY GH Secret is stale — n8n API key was regenerated in n8n Settings but the GH Secret was not updated.

History:

• 2026-06-30: n8n workflow snapshot failing (run 28428432130)

Fix:

1. SSH into bms-4: ssh root@54.36.123.110
2. In n8n: Settings → API → regenerate key (or check if existing key works)
3. Update GH Secret: gh secret set BMS4_N8N_API_KEY --body "<new-key>"

Prevention: When rotating n8n API key, add a checklist step to update all GH Secrets that reference it.

---

Group D — Stale Supabase DB password (direct postgres connection)

Symptom:

FATAL: password authentication failed for user "postgres"
curl: (22) The requested URL returned error: 404  # on Discord notification step

Affected workflows: DB Maintenance (db-maintenance.yml)

Root cause pattern: SUPABASE_DB_PASSWORD GH Secret contains an old password. Supabase postgres password was rotated (e.g. during sb_secret key migration) without updating this secret.

History:

• 2026-06-30: DB Maintenance failing (run 28425367096)

Fix:

1. Get current DB password from Supabase dashboard → Project Settings → Database → Connection string
2. Update: gh secret set SUPABASE_DB_PASSWORD --body "<new-password>"
3. Re-run workflow manually to confirm

Note: The SUPABASE_SERVICE_KEY → sb_secret migration (PR 896, 2026-06-21) may have also rotated the direct postgres password. Always check both after any Supabase credential rotation.

---

Group E — BOM character in HTTP header (UnicodeEncodeError)

Symptom:

UnicodeEncodeError: 'latin-1' codec can't encode character '' in position 7: ordinal not in range(256)

Affected workflows: compliance-audit-due-check

Root cause pattern: A file being read and used as an HTTP Authorization header contains a UTF-8 BOM (). This happens when:

• A secret file was written with BOM on Windows (Out-File, Set-Content, > redirection)
• A SOPS file was decrypted with BOM-adding Windows tools

History:

• 2026-06-30: compliance-audit-due-check failing (run 28434831724)

Fix:

1. Identify which file/secret is being read as the auth header in the script
2. Strip BOM: sed -i 's/^\xEF\xBB\xBF//' <file> or re-write with UTF8Encoding($false) on Windows
3. If the secret was written to GH Secrets with BOM, delete and re-add the secret

Prevention: Always use [System.IO.File]::WriteAllText($path, $content, [System.Text.UTF8Encoding]::new($false)) on Windows. See docs/playbooks/sops-windows-crlf.md.

---

Group F — vps-i1 unstaged changes blocking git pull

Symptom:

error: cannot pull with rebase: You have unstaged changes.
error: Please commit or stash them.
Process completed with exit code 128.

Affected workflows: Prometheus Alerts → AI Sysadmin Triage, any workflow that SSHes into vps-i1 and runs git pull

Root cause pattern: An agent or manual operation left uncommitted changes in /opt/p24-infra on vps-i1. The git pull --ff-only or git pull --rebase in the workflow step then fails.

History:

• 2026-06-30: Prometheus AI Triage failing (run 28436140120)

Fix:

ssh root@217.154.82.162
cd /opt/p24-infra
git status           # identify the dirty files
git stash            # stash if changes are temp / exploratory
# OR
git diff             # review changes
git add -A && git commit -m "chore: save agent workdir state"

Prevention:

• Agents on vps-i1 should never modify tracked files in /opt/p24-infra directly
• If a script must write config, write to /opt/p24-infra/.local/ (gitignored) or a temp file

---

Group G — GH Token / App credential issues

G1 — GitHub App app-id missing (Dependabot context)

Symptom:

throw new Error("Input required and not supplied: app-id");
Error: Input required and not supplied: app-id
Secret source: Dependabot

Affected workflows: Dispatch to Worker Queue (dispatch-to-queue.yml)

Root cause: Dependabot PRs run with a restricted token that does not have access to repository Actions secrets. The actions/create-github-app-token@v1 step requires APP_ID + PRIVATE_KEY which are not available.

Fix: Add a condition to skip dispatch when triggered by Dependabot:

if: github.actor != 'dependabot[bot]'

Or alternatively, grant Dependabot access to these specific secrets in repo Settings → Secrets.

History: 2026-06-30 (run 28425183531)

G2 — GH_TOKEN insufficient for GraphQL labels

Symptom:

error fetching labels: GraphQL: Resource not accessible by integration (repository.labels)

Affected workflows: n8n workflow snapshot (error-handler step), Infra Task Request — enqueue

Root cause: The GITHUB_TOKEN on bms-4 runner does not have issues: write permission, or the PAT used lacks repo-level label access. The error occurs in the error-handler step that tries to add a bug label.

Fix: The workflow needs permissions: issues: write in the job definition, OR the GH_TOKEN secret needs broader scope.

---

Group H — Dead Discord webhook

Symptom:

{"message": "Unknown Webhook", "code": 10015}
# OR
curl: (22) The requested URL returned error: 404

Affected workflows: Grafana Backup, any workflow sending Discord alerts via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL or DISCORD_WEBHOOK_URL

Root cause: The Discord webhook URL stored in GH Secrets has been deleted or expired in the Discord server. Error notifications from all infrastructure jobs silently drop.

Status: Already tracked in docs/priorities.md P2 section.

Fix:

1. Create new Discord webhook in target channel → Integrations → Webhooks → New Webhook → Copy URL
2. Update GH Secret: gh secret set DISCORD_WEBHOOK_URL --body "<new-url>"
   And also: gh secret set P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL --body "<new-url>"
3. Test with: curl -X POST "$URL" -H "Content-Type: application/json" -d '{"content":"test"}'

---

Group I — Gitflow enforcement block (OBSOLETE since #2513)

⚠️ No longer applies. The feat/* → dev → rc/* → main flow was retired on 2026-07-08 (#2513); enforce-gitflow.yml was updated to pass unconditionally. p24-infra feat/*/fix/* branches now PR directly to main — that is the correct, expected flow, not a failure. This section is kept for historical reference; a genuine “PRs to main must come from ‘dev’/‘rc/*’” block should no longer fire. If you see it, the enforcing workflow is itself stale — fix the workflow, do not reroute the PR through the frozen dev branch.

Historical symptom (no longer expected):

BLOCKED: PRs to main must come from 'dev' or 'rc/*'
Got: fix/2007-waha-monitoring-off
Process completed with exit code 1.

Affected workflows: Enforce gitflow — main branch (now passes unconditionally)

---

Group J — Workflow YAML file issues on branch

Symptom:

X This run likely failed because of a workflow file issue.
(no log output available)

Affected workflows: db-maintenance-check, cloudflare-security-check

Root cause pattern: A push on a branch containing a broken workflow YAML file causes GitHub to report “workflow file issue” for ALL workflows triggered by that push. The branch hotfix/delete-waha-cf-worker caused this on 2026-06-30.

Fix:

1. Identify the bad branch: gh run view <id> --json headBranch
2. Check the workflow YAML on that branch for syntax errors
3. If the branch is merged/deleted, the issue auto-resolves

Note: These failures do NOT indicate the workflow itself is broken on main/dev.

---

Group K — Runs stuck queued + job-level EACCES on the self-hosted runner (NOT a runner outage)

Symptom:

# Multiple GH Actions runs sit in `queued` for 20–30+ min, then one job that DID start logs:
Error: EACCES: permission denied, stat '/root/.local/bin/git'

Affected workflows: Sync secrets to VPSes, P24-Infra Self Test, Enforce gitflow, Dispatch to Worker Queue — anything routed to the bms-4 / secrets-deploy self-hosted runner.

Root cause pattern: The runner process is healthy — it never crashed. Jobs stall because a job-level step needs elevated/sudo access the claude-runner user no longer has (sudoers drift): the sops/git install/stat step trips EACCES: permission denied, stat '/root/.local/bin/git'. The queued backlog is a side effect of secrets-deploy failover waiting on the specific labelled runner, not evidence the runner is offline. Tracked separately as the drift-sudoers gap: **#3291 / 3344 (claude-runner not in sudoers for the sops install step).

Why this matters — do NOT misdiagnose as a runner crash: the natural first read of “many runs queued 25min” is “runner is down → svc.sh restart” (see gh-runner-restart-offline.md). That restart is wasted here — the runner was never down. Confirm liveness BEFORE restarting:

# Runner alive? (registration Online/Idle → it is NOT a runner-outage case)
gh api repos/radieu/p24-infra/actions/runners --jq '.runners[] | {name, status, busy}'
# A recent job that reached "in_progress" (even if it later failed) also proves the runner is live.
gh run list --repo radieu/p24-infra --limit 10 --json databaseId,status,conclusion,name

Distinguishing signature (Group K vs. a real runner outage):

Observation	Group K — job-level drift sudoers	Real runner outage (gh-runner-restart-offline.md)
Runner status via API	online	offline
Any job reaches in_progress?	yes (then fails on EACCES)	no — all stay queued
Error text	EACCES … stat '/root/.local/bin/git'	Failed to create a session / Unauthorized / none
Fix	restore claude-runner sudoers (#3291/#3344) — NOT a restart	systemctl restart or re-register

Fix: restore the claude-runner sudoers entry for the sops install step (see #3291 / #3344). Do not restart the runner. Once sudoers is fixed, requeued/re-run jobs proceed.

History:

• 2026-07-08 (#3324): 4 workflows queued >25min after 03:58 UTC; initially reported as “runners offline”. Owner confirmed the runner was healthy throughout — root cause was job-level drift sudoers (EACCES). secrets-sync partially completed (sync-pinbox24-backends OK, W4 up HTTP 200); old Wasabi key deleted manually. Side gaps split out: drift sudoers (#3291/#3344) and a stale sync-bms-4 SSH key (#3345, since closed).

Prevention: before treating “queued workflows” as a runner outage, check runner API status and whether any job reached in_progress. If a job started and failed on EACCES, it is a job-level permission gap — not a runner crash.

---

Group L — Text file busy (exit 126) on the shared ~/.local/bin/sops — parallel-install race

Symptom:

/opt/gh-runner-3/_work/_temp/<uuid>.sh: line 6: /home/claude-runner/.local/bin/sops: Text file busy
##[error]Process completed with exit code 126.

Several secrets-sync jobs fail simultaneously in their Install sops step with exit 126. Distinct from Group K (that is EACCES/exit 1 from sudoers drift; this is ETXTBSY/exit 126).

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — any run where 2+ of its 13 self-hosted, secrets-deploy jobs execute concurrently.

Root cause pattern: All secrets-deploy runner instances on bms-4 (gh-runner, gh-runner-3, …) run as the same claude-runner user and therefore share one $HOME — so every job’s Install-sops step wrote the same file ~/.local/bin/sops. When job A’s curl -o ~/.local/bin/sops is still writing the binary while job B tries to ~/.local/bin/sops --version, the kernel returns ETXTBSY (cannot exec a file open for writing) → exit 126. It is a pure concurrency race: it fires only when jobs overlap, so it looks intermittent.

Fix (#3625): made the install idempotent and atomic in all 13 blocks — skip the download if the pinned version (3.9.1) is already on PATH; otherwise curl to a per-job mktemp file, chmod, then mv -f it into ~/.local/bin/sops. The shared path is never held open for writing during an exec, so ETXTBSY cannot occur, and concurrent installs converge on an identical binary.

Distinguishing signature (Group L vs Group K):

Observation	Group L — sops install race	Group K — drift sudoers
Exit code	126	1
Error text	… /sops: Text file busy	EACCES … stat '/root/.local/bin/git'
Trigger	2+ jobs overlapping	any job needing sudo the user lost
Fix	atomic/idempotent install (#3625)	restore sudoers (#3291/#3344)

Prevention: never write a shared-$HOME tool binary in place from parallel jobs. Download to a temp file and atomically mv into place, and short-circuit when the pinned version already exists.

Note — separate, still-open failures in the same run (not fixed here; track individually):

• sync-pinbox24-backends exit 1 — mailgun-v42-prod container name conflict on docker compose up. Now classified as Group Q and fixed with in-step conflict recovery (#4290).
• sync-gh-secrets exit 1 — failure while processing P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (transient GH API rate-limit — now classified as Group M and hardened with retry/backoff, #3697).

---

Group M — GH API rate-limit exhaustion in sync-gh-secrets (HTTP 403, transient)

Symptom:

  P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
failed to fetch public key: HTTP 403: API rate limit exceeded for user ID 11704852 …
  (https://api.github.com/repos/radieu/p24-infra/actions/secrets/public-key)
##[error]Process completed with exit code 1.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — the sync-gh-secrets job. Any workflow that calls gh secret set in a loop is vulnerable.

Root cause pattern: gh secret set must first GET .../actions/secrets/public-key to encrypt each value. That request counts against the shared GH_PAT account’s 5000/hr core rate budget (user ID 11704852). Many agents, dispatchers, and the daily audits all authenticate as that same user, so the budget can be momentarily exhausted. When the public-key fetch lands during exhaustion it returns HTTP 403 rate-limit exceeded and — before #3697 — failed the entire sync on the first key. This is transient and self-healing: the budget resets hourly, and prior/subsequent runs succeed.

Distinguishing signature: error text is HTTP 403: API rate limit exceeded for user ID … on the .../actions/secrets/public-key endpoint. NOT EACCES (Group K), NOT Text file busy/exit 126 (Group L). It is a GitHub-API quota issue, not a runner or sudoers problem.

Diagnosis / confirm transient:

gh run list --repo radieu/p24-infra --workflow secrets-sync.yml --limit 8 \
  --json databaseId,status,conclusion,createdAt      # neighbours succeed → transient
gh api rate_limit --jq .resources.core               # remaining/reset for the current token

Fix / recovery:

1. Re-run just the affected job (idempotent — re-writes the same SOPS values): gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=gh-secrets
2. If the core budget is still near-zero, wait for reset before retrying.

Prevention (#3697 + #4089):

• #3697 — wrapped gh secret set in a retry-with-backoff helper (set_secret_with_retry, 5 attempts, 15s→30s→60s→120s ≈ 4 min total window) in the sync-gh-secrets job, so a single transient 403/network blip no longer fails the whole sync.
• #4089 — added wait_for_core_budget: called once at job start (pre-flight) and again from set_secret_with_retry whenever the captured error matches API rate limit exceeded|HTTP 403. It reads gh api rate_limit, and if core.remaining < 50 it sleeps until core.reset + 10s (capped at 3600s). Handles sustained exhaustion — a multi-minute drain where the ~4 min retry backoff window from #3697 wasn’t long enough. timeout-minutes bumped 10 → 65 so the wait can complete inside the job.

History:

• 2026-07-11 (#3697): 1 failed run (29131871037, 00:06 UTC) — 403 on the first key (P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL). Confirmed transient (neighbouring runs OK); a target=gh-secrets re-run (29148035117) succeeded. Added retry/backoff hardening.
• 2026-07-12 → 2026-07-13 (#4089): 5 failed runs in 25h, e.g. 29241072075 at 10:01:44 UTC — 403 on the same key across all 5 retry attempts. The retry backoff (max 120s between attempts) landed inside the same exhausted hourly window every time, so retries burned without progress. Fixed by pre-flight wait_for_core_budget + rate-limit-aware retry that sleeps until reset.

---

Group N — sops-sync-receiver exit 2: canary decrypt failed … invalid dotenv input line: {

Symptom:

[SOPS-RECEIVER] sops replaced key <CREDENTIAL>
[SOPS-RECEIVER] ERROR exit=2 canary decrypt failed after encrypt — NOT committing: invalid dotenv input line: {
##[error]Process completed with exit code 2.

Affected workflows: SOPS Sync Receiver (sops-sync-receiver.yml → scripts/sops-sync-receiver.py)

Root cause pattern: A rotation producer built the payload with a non-scalar value — either a nested JSON object ({"value": "..."}) or a double-encoded JSON string of that object ('{"value": "..."}') — instead of a plain string. update_sops() then serialized the dotenv line KEY={"value": "..."}, which begins with {; SOPS’s dotenv parser rejects it at the post-encrypt canary decrypt, so the receiver aborts with exit 2 BEFORE committing. This is the safety fence working as designed — the SOPS file is left intact, no corruption ships. The new credential is simply never persisted (the rotation must be re-run once the producer is fixed).

Not a runner/secret problem: the receiver did its job. The bug is upstream in whatever built the payload. Do not “fix” by editing SOPS by hand.

Fix (already shipped — verify it is present before re-deriving):

• _coerce_string_value() unwraps both a dict value and a str value that (stripped) starts with { and JSON-decodes to a dict — extracting the scalar via the value/new_value field.
• validate_payload() rejects any value still {-prefixed after coercion as an explicit exit-1 validation error, so this class never again reaches the SOPS writer as a confusing exit-2 canary failure.
• Covered by scripts/tests/test_sops_sync_receiver.py. If a NEW variant slips through, add the offending shape as a test case and extend _coerce_string_value().

Drift check after any occurrence: the rotate script verifies + applies the new credential to the live service before dispatching to the receiver, so an exit-2 abort can leave live ≠ SOPS drift (new value live, SOPS still old). Confirm the live service health and, if a real rotation (non-empty LOG_ID / a dev_r_rotation_log row) was in flight, re-run the rotation or reconcile SOPS. A log_id=none + workflow_dispatch run with no dev_r_rotation_log row is a manual repro/test, not a tracked production rotation — usually no drift.

History:

• 2026-07-08: W3_APP_MONGODB_PASSWORD rotation payload malformed → exit-2 canary abort (run 28951846700). Real live/SOPS drift for w3_app did occur in this window and caused a v32-prod auth-fail crash-loop, reconciled by #3609 (rotate w3_app password to match). The receiver code path was hardened by #3171 (dict unwrap) and **#3412 / PR 3531 (double- encoded JSON-string unwrap + exit-1 reject; 18 tests). Failure-notification dedup: #3413.
• 2026-07-09: daily gha-failed-jobs-audit.py re-surfaced the now-stale 28951846700 run as #3633 (root cause already fixed the previous night; no further code action).

Prevention: rotation producers must send plain-string scalars in new_value/new_values. When adding a new scripts/rotate/*.sh producer, assert the payload value is a JSON string (not an object) before dispatch.

---

Group O — Bot auto-commit git push rejected by main branch protection (GH006)

Symptom:

[warn] git push failed: remote: error: GH006: Protected branch update failed for refs/heads/main.
remote: - Changes must be made through a pull request.
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to 'https://github.com/radieu/p24-infra'

Affected workflows: Credential Rotation (rotate-credentials.py → sops_update_key), n8n Workflow Backup, n8n Workflow Snapshot, Rotation Log Housekeeping, wasabi-sops-update.yml — any workflow that runs on the default ref (main) and does git commit && git push back to it.

Root cause pattern: main is a protected branch requiring PRs. Any GH Actions job that commits and pushes directly to the checked-out main ref is rejected with GH006, regardless of the token used (scoped GITHUB_TOKEN, GH_PAT, or GH_TOKEN) unless that identity is on the branch-protection bypass allowlist. The push failure is often logged only as a [warn] (best-effort), so the job may still exit 0 while the commit is silently dropped.

Why this matters — silent data loss, not just a red run: for credential rotation the sequence is create new token → gh_secret_set (✓) → sops_update_key push (✗ GH006) → delete old token. Net result: GH Secrets/provider hold the new value, but the SOPS repo keeps the OLD (now-deleted, dead) value. secrets-sync would then deploy the dead credential. The weekly scheduled rotation hits this every week.

Fix (design): push bot commits to a chore/auto-rotate-<date> branch and open a PR to main (the branch-protection-compatible path) instead of pushing to the checked-out ref — see the update-vps-h1-sops.yml pattern (branch + gh pr create). Alternative: add the automation identity to the branch-protection bypass allowlist (repo-admin decision).

Workaround (manual rotation): dispatch the workflow with --ref <feature-branch> so the push lands on a branch, then open + merge a PR (used for #2701 → PR #3354).

Tracked in:

• #3357 — credential-rotation sops_update_key push to protected main (full analysis + fix design)
• #3640 — n8n Workflow Backup commit step rejected by GH006 (same pattern, different workflow)
• #3350 — stale Wasabi GH Secret causing InvalidAccessKeyId (the co-occurring failed=1 in the Credential Rotation run — a separate root cause, not the push failure)

History:

• 2026-07-08 (#3357): Credential Rotation rotated 3 CF tokens; provider + GH Secret updated but SOPS push to main rejected → SOPS retained dead values.
• 2026-07-09 (#3632): daily failed-jobs audit re-surfaced the same Credential Rotation failure (Wasabi InvalidAccessKeyId set failed=1 → exit 1; CF SOPS pushes rejected GH006 as before). Both root causes already tracked (#3357 + #3350) → closed as duplicate.

Prevention: treat GH006 in any auto-commit workflow as this group — do NOT re-file per-workflow duplicates; route to #3357 (systemic fix) and the workflow-specific tracker. Do not git push to main from CI without a bypass-allowlisted identity.

---

Group P — sops --encrypt input path does not match a .sops.yaml creation rule

Symptom:

error loading config: no matching creation rules found
##[error]Process completed with exit code 1.

The job builds the updated plaintext fine (e.g. logs Updated keys: {...}) and fails on the very next sops --encrypt call.

Affected workflows: wasabi-sops-update.yml (and any workflow that re-encrypts a SOPS file from a staging plaintext).

Root cause pattern: SOPS matches .sops.yaml’s creation_rules[].path_regex against the input file path passed to --encrypt, not against the --output destination. The catch-all rule is (^|[/\\])secrets[/\\].*\.env\.sops$, so an input written to /tmp/monitoring-updated.env matches no rule → “no matching creation rules found”, even though --output points at secrets/monitoring.env.sops.

Fix (#3880, commit 6957ebf0): write the staging plaintext to a path that matches the rule — secrets/monitoring-tmp.env.sops — and pass that as the encrypt input (sops --encrypt --output secrets/monitoring.env.sops secrets/monitoring-tmp.env.sops). The secrets/*-tmp.env.sops name follows the CLAUDE.md §Secrets convention and is rm’d before the commit step. Precursors on the same workflow: #3281 (dispatch ref dev→main + path_regex) and #3332 (unparseable YAML).

Secret-free reproduction:

printf 'K=x\n' > /tmp/foo.env             # sops --encrypt /tmp/foo.env            -> "no matching creation rules found" (exit 1)
printf 'K=x\n' > secrets/foo-tmp.env.sops # sops --encrypt secrets/foo-tmp.env.sops -> succeeds (exit 0); rm the tmp file after

Prevention: when re-encrypting from a staging file, the encrypt input path (not just --output) must satisfy a .sops.yaml path_regex. Stage under secrets/*-tmp.env.sops.

History:

• 2026-07-11 21:57 UTC: run 29169690473 (manual workflow_dispatch) failed with this error on the pre-fix workflow. Fix 6957ebf0 (#3880) merged 22:12 UTC — ~15 min later.
• 2026-07-12: daily gha-failed-jobs-audit.py re-surfaced the now-stale 21:57 run as #3983 (root cause already fixed by #3880; no further code action) — closed as stale, like 3632.

---

Group Q — docker-compose up container-name conflict on a container the project does not own

Symptom:

Creating mailgun-v42-prod ... error
ERROR: for mailgun  Cannot create container for service mailgun: Conflict. The container name
"/mailgun-v42-prod" is already in use by container "45a2873db599…". You have to remove
(or rename) that container to be able to reuse that name.
Encountered errors while bringing up the project.
##[error]Process completed with exit code 1.

Affected workflows: secrets-sync.yml — sync-pinbox24-backends (Restart mailgun-prod). Any step that runs docker-compose up -d --force-recreate <svc> against a compose file with a fixed container_name is vulnerable: the W3 (backend/s3/reso/socket) and W4 restart steps in the same workflow share the shape.

Root cause pattern: --force-recreate only recreates containers the invoking compose project owns — matched via the com.docker.compose.project / .service labels, which are derived from the directory name compose was invoked from. When the name is held by a container created out-of-band (a hotfix run, a separate GitLab-CI build-project directory, or a plain docker run), compose finds nothing to recreate, falls through to create, and Docker rejects the duplicate container_name.

The exit-1 is the visible half. The dangerous half is silent: the env file the step just synced never reaches the running container, so the stack keeps authenticating with the pre-rotation credential — the exact failure mode of #3688 (mailgun) and #3958 (v32-prod backend), where docker ps, PM2 status, and health checks all look green while every request hangs to a 504.

Fix (#4290): in-step recovery, gated on the conflict signature so unrelated compose failures still abort untouched:

1. Capture docker-compose up -d --no-deps --force-recreate <svc> output; on success, done.
2. On failure, match the output for is already in use by container. No match → exit 1 unchanged.
3. Match → docker stop <name> + docker rename <name> <name>-stale-$(date +%Y%m%d%H%M%S) (rename, never remove — keeps a zero-risk rollback), then retry the compose up once.
4. Retry fails → rename back, docker start, exit 1.

After one successful recovery the new container is compose-owned, so the path stops triggering. The renamed *-stale-* container is left stopped for inspection; removing it is a follow-up sys-admin action.

Diagnosing which project owns the name (on the host):

docker inspect mailgun-v42-prod --format '{{index .Config.Labels "com.docker.compose.project"}}'
# empty or != the directory the workflow cd's into  →  Group Q

Prevention: any new deploy/restart step that force-recreates a fixed-container_name service should carry the same recovery guard, or drop container_name from the compose file so compose can name containers itself. Related playbooks: mailgun-mongodb-stale-credential-hang.md §Fix (the manual form of this procedure), v32-prod-mongoose-authfail-storm.md §Variant (#3958 — same conflict silently skipping the v32-prod backend), bms1-container-audit.md.

---

Group R — gh issue create --label <name> aborts when the label does not exist (exit 1)

Symptom:

could not add label: 'drift-detected' not found
##[error]Process completed with exit code 1.

Affected workflows: ansible-drift.yml (Ansible drift detection — “Open drift issue” step). Any workflow whose reporting step calls gh issue create --label "<x>,<y>" / gh issue comment against a label that was never created in the repo is vulnerable.

Root cause pattern: gh issue create --label X is fail-closed — if label X does not exist in the repo, the command exits 1 (it does NOT auto-create the label). The drift workflow reports drift by opening/commenting a drift-detected,bug-labelled issue, but drift-detected was never created in radieu/p24-infra (only sops-drift existed). So the moment the check found real drift and tried to report it, the reporting step itself died — turning the run RED and swallowing the drift signal it was built to surface. This is the same “the reporter is the thing that breaks” shape as Group A’s exit $RC trap.

Here the drift being reported was legitimate: the vps-i1 --check had failed=1 because the claude-runner role’s p24-server-heartbeat cron task references undefined supabase_url / supabase_service_key vars (Error while resolving value for 'job': 'supabase_url' is undefined). That ansible bug is a separate defect tracked on its own — it is exactly the kind of drift the check exists to report; it must not also crash the reporter.

Fix (#4377):

1. Created the missing drift-detected label in the repo (gh label create drift-detected --color d93f0b --force).
2. Made the “Open drift issue” step self-heal: it now runs an idempotent gh label create drift-detected … --force 2>/dev/null || true before any gh issue list/create/comment, so a deleted/renamed label can never re-break the reporter.

Prevention: any workflow step that applies a label via gh must ensure the label exists first (gh label create … --force is idempotent) — never assume a repo label is present.

---

Group S — shared-$HOME race on ~/.ssh/id_bms1 between concurrent bms-1 sync jobs

Symptom:

Warning: Identity file /home/claude-runner/.ssh/id_bms1 not accessible: No such file or directory.
root@94.23.26.113: Permission denied (publickey).
lost connection
##[error]Process completed with exit code 1.

Fires in a bms-1 SSH/SCP step (Deploy W4 persistent-patches to bms-1, Ship to bms-1, any W3/W4 step) even though the same step’s own printf … | base64 -d > ~/.ssh/id_bms1 ran two lines earlier. Intermittent — only when 2+ bms-1 jobs overlap.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — the concurrent bms-1 jobs sync-pinbox24-w3, sync-pinbox24-w4, sync-pinbox24-backends, sync-bms-1 (mezmo). All wrote their SSH key to the same path ~/.ssh/id_bms1.

Root cause pattern: every secrets-deploy runner instance on bms-4 runs as the same claude-runner user and shares one $HOME (the same shared-$HOME substrate as Groups K & L). Each bms-1 step did touch ~/.ssh/id_bms1 … > ~/.ssh/id_bms1, used it with ssh/scp -i ~/.ssh/id_bms1, and — critically — each job’s Scrub plaintext secrets step (if: always()) ran rm -f ~/.ssh/id_bms1. When job A’s scrub deletes the shared key mid-flight while job B’s SSH step is still running, B’s -i ~/.ssh/id_bms1 resolves to a now-missing file → “Identity file … not accessible: No such file or directory” → “Permission denied (publickey)” → exit 1. It is a pure delete-out-from-under race, so it looks intermittent and hits whichever bms-1 job happens to still be mid-SSH when a sibling finishes.

Distinguishing signature (Group S vs Groups K/L):

Observation	Group S — id_bms1 delete race	Group K — drift sudoers	Group L — sops install race
Error text	id_bms1 not accessible: No such file + Permission denied (publickey)	EACCES … stat '/root/.local/bin/git'	… /sops: Text file busy
Exit code	1	1	126
Trigger	2+ bms-1 jobs overlap; sibling scrub rms the shared key	any job needing lost sudo	2+ jobs overlap on Install-sops
Fix	per-job $RUNNER_TEMP/id_bms1 (#4376)	restore sudoers (#3291/#3344)	atomic/idempotent install (#3625)

Fix (#4376, merged 2026-07-20 11:53 UTC): migrated every bms-1 SSH-key write, -i reference, and scrub in secrets-sync.yml from the shared ~/.ssh/id_bms1 to the per-job-unique $RUNNER_TEMP/id_bms1 ($RUNNER_TEMP is a fresh dir per job even on a shared-$HOME runner, so concurrent jobs can no longer clobber or delete each other’s key; the rm -f "$RUNNER_TEMP/id_bms1" scrub only removes that job’s own copy). Verify with grep -c 'RUNNER_TEMP/id_bms1' .github/workflows/secrets-sync.yml (61 usages) and grep -nE '~/\.ssh/id_bms1' … (must be zero).

History:

• 2026-07-19: 2 failed runs — 29681365216 (09:19 UTC, sync-pinbox24-w3) and 29691164407 (14:37 UTC, sync-pinbox24-w4 / Deploy W4 persistent-patches). Both on the pre-fix shared-path workflow. Fixed by #4376 the next morning.
• 2026-07-20: daily gha-failed-jobs-audit.py re-surfaced the two now-stale runs as #4381 (root cause already fixed by #4376; no further code action) — closed as stale, like 3632.

Prevention: never write/read/rm a per-run credential file at a shared path under $HOME from parallel jobs on a shared-$HOME self-hosted runner. Use $RUNNER_TEMP (per-job) for any transient key/secret material. This is the same lesson as Groups K & L — treat $HOME as shared, $RUNNER_TEMP as private.

---

Group T — scp: <file>.js: Is a directory — Docker bind-mount placeholder on bms-1

Symptom:

scp: /home/gitlab-runner/builds/.../p24-v-3.2/persistent-patches//crudService.js: Is a directory
scp: /home/gitlab-runner/builds/.../p24-v-3.2/persistent-patches//filesUpload.js: Is a directory
scp: .../mongoose.js: Is a directory
scp: .../pushNotification.helper.js: Is a directory
scp: .../twilioService.js: Is a directory
##[error]Process completed with exit code 1.

Fires in Deploy W3 persistent-patches to bms-1 (or the W4 / s3-v2 equivalents). Not intermittent — once it starts, every subsequent run of the job fails identically until the server state is cleared. Distinguishes from Group S by the error text: Group S fails to connect (Permission denied (publickey)), Group T connects fine and fails on the destination path.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — sync-pinbox24-w3, sync-pinbox24-w4, and the Deploy s3-v2-v42-prod patches step.

Root cause pattern: two mechanisms compound.

1. persistent-patches/ lives inside a GitLab runner build directory (/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/ for W3, /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ for W4). The .js patch files are untracked there, so a GitLab CI checkout/clean wipes them — the identical failure class as the 2026-07-09 W3 incident that emptied backend-environment.env / s3-environment.env.
2. Docker auto-creates a directory at any bind-mount source path that does not exist. The next v32-prod / v42-prod start therefore replaces each wiped file with an empty directory of the same name, and scp cannot overwrite a directory.

The tell that it is the Docker mechanism and not something else: the set of failing filenames matches exactly the bind-mount list in infra-src/pinbox24/w3/docker-compose.yml. In the 2026-07-30 occurrence the 5 failures were precisely the files mounted into v32-prod (lines 10–15); controller.js and local.js, mounted only into s3-v32-prod, were untouched because that service had not been restarted in the window.

Fix (#4652): each patch-deploy step now runs a pre-scp guard over SSH that rmdirs any *.js path in the target dir that is a directory:

for stale in "$PP"/*.js; do
  [ -d "$stale" ] || continue
  if rmdir "$stale" 2>/dev/null; then echo "removed stale Docker-created directory: $stale"
  else echo "::error::$stale is a non-empty directory — manual cleanup required on bms-1"; exit 1; fi
done

rmdir rather than rm -rf is deliberate: Docker’s placeholders are always empty, so rmdir succeeds on exactly the intended case and refuses to destroy real content on a production path — a non-empty directory aborts the step loudly instead. The job is now self-healing: the following Deploy W3/W4 containers step recreates the containers so they pick up the restored files (a running container keeps the old mount inode until it is recreated).

History:

• 2026-07-21 19:04 — last green run before the break.
• 2026-07-29 — a W3 GitLab pipeline ran on bms-1; docs/priorities.md records chown -R gitlab-runner:gitlab-runner being applied to persistent-patches/ because the files were “root-owned from Docker” — i.e. the placeholders were already being created around then.
• 2026-07-29 18:03 → 2026-07-30 07:29 — 6 consecutive failed runs, filed as #4652 with root-cause group unknown. Fixed by #4652.

Prevention: treat any bind-mount source path inside a CI build directory as volatile. If a deploy step copies files that Docker also mounts, guard for the placeholder-directory state before copying — a missing bind-mount source never fails loudly, it silently becomes a directory.

---

Group U — Vercel 403 ENV_ALREADY_EXISTS on env-var create (duplicate create rejected)

Symptom:

HTTP 403 on POST /v9/projects/prj_***/env: {"error":{"code":"ENV_ALREADY_EXISTS",
"message":"Another Environment Variable with the same Name and Environment exists in your
project. Remove it or choose a different Name or Environment.","key":"MONGODB_URI"}}
urllib.error.HTTPError: HTTP Error 403: Forbidden
##[error]Process completed with exit code 1.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — sync-et-lager / Sync et-lager secrets to Vercel.

Root cause pattern: not a credential problem — the bare HTTP Error 403: Forbidden in the traceback reads like a revoked token, but the response body names the real cause. #2838 had changed this job to POST-then-DELETE ordering (create the replacement first, then retire the old entry) so that a failed write could never leave the project with no value for the key — et-lager.env.sops holds a single key (MONGODB_URI), so an empty window there means total DB failure for et-lager. That ordering depends on Vercel accepting a duplicate key+target create. Vercel now rejects it outright, and the guard only recognised 409:

if e.code != 409 or not stale_ids:
    raise

so the delete-then-create fallback never fired. The sibling sync-et-operational-platform job kept passing in the same runs because it still uses plain delete-then-create — proof the token and project access were fine.

Second-order bug found alongside it: api() did print(f"… {e.read().decode()}") and then raise. e.read() is a one-shot stream, so the print consumed the only copy of the body and any caller was structurally unable to inspect Vercel’s error code.

Fix (#4652): the job now PATCHes the existing entry in place (PATCH /v9/projects/{prj}/env/{id} with only {"value": …}, leaving key/target/type untouched) and POSTs only when the key is genuinely absent. PATCH is a single atomic update with no window at all, so it satisfies #2838’s requirement strictly better than either ordering while sidestepping the duplicate-create rejection entirely. Retained as fallbacks: delete-then-create if the PATCH is rejected, and a re-GET-then-PATCH recovery if a concurrent run creates the key between the pre-loop GET and the POST. api() now stashes the body on the exception (e.p24_body) before printing.

Conflict detection matches on the body code, not the status alone, so a genuine 403 (revoked token, lost project access) still fails loudly rather than being mistaken for a conflict.

PATCH precedent in this repo: rotate-openai-etop-key.yml PATCHes /v9/projects/{prj}/env/{id} for an encrypted-type var. Caveat: PATCH does not work for type: sensitive vars — see the wasabi-admin-sync.yml comment “Delete existing (can’t change sensitive type via PATCH)”. et-lager and et-op both use encrypted.

History:

• 2026-07-21 19:04 — last green sync-et-lager run. The job triggers on every push to main (no path filter), so once Vercel tightened duplicate handling every push failed it.
• 2026-07-29 18:03 → 2026-07-30 07:29 — 6 consecutive failed runs, same 6 runs as Group T (two independent bugs in one workflow, filed together as #4652). Fixed by #4652.

Prevention: when a sync job writes to a third-party API, branch on the body error code, not the HTTP status alone — and never consume an error stream in a log line before the handler that needs it. Prefer an idempotent update verb (PATCH/PUT) over create-then-delete or delete-then-create ordering games; both orderings have a failure window, an atomic update has none.

---

Group V — OpenAI Admin API 404 Not Found on a non-existent endpoint path

Symptom:

rotate	OpenAI GET /v1/organization/api_keys?limit=100 → 404:
rotate	urllib.error.HTTPError: HTTP Error 404: Not Found
rotate	##[error]Process completed with exit code 1.

(The error body is empty — a 404 on the path, not a JSON error object.)

Affected workflows: rotate-openai-etop-key (#5320; failed all 3 dispatches — 2026-06-30 ×2, 2026-08-04).

Root cause: The workflow calls OpenAI Admin API paths that do not exist. The auth is fine (a bad OPENAI_ADMIN_KEY would 401, not 404) — the paths themselves are wrong:

Workflow used	Reality
GET /v1/organization/api_keys (line 51)	no such endpoint → 404 (the observed failure)
POST /v1/organization/projects/{id}/api_keys (line 66)	no create-project-API-key endpoint exists — would fail next even after fixing line 51
DELETE /v1/organization/api_keys/{id} (line 94)	no such endpoint

The correct OpenAI Admin API shape is project-scoped:

• List project keys: GET /v1/organization/projects/{project_id}/api_keys
• Delete a project key: DELETE /v1/organization/projects/{project_id}/api_keys/{key_id}
• Admin (org-management) keys: GET|POST|DELETE /v1/organization/admin_api_keys
• Programmatic creation of a usable key is only possible via a service account: POST /v1/organization/projects/{project_id}/service_accounts returns api_key.value (an sk-svcacct-… key). Plain human project API keys cannot be minted through the API.

So this workflow’s whole premise (create a plain project key via API, capture its value) is unworkable without switching to service-account keys — a security/ownership design change, not a one-line path fix.

Fix (#5320): the workflow was a one-shot rotation for the #2269 exposure (“Triggered manually only. After successful run, delete this workflow file.”) that never once succeeded. It was retired (deleted) rather than rewritten — a security-sensitive, prod-mutating, untestable rewrite would give false confidence. The residual OPENAI_API_KEY rotation for et-op was escalated to a human (verify via the OpenAI dashboard + rotate manually per docs/playbooks/static-api-key-incident-rotation.md, then update the Vercel OPENAI_API_KEY env var for the et-op project).

Prevention: before shipping an Admin-API automation, confirm each path against the current OpenAI API reference — org key management is project-scoped (/organization/projects/{id}/api_keys), and usable keys are minted via service accounts, never a top-level /organization/api_keys create.

---

Group W — startup_failure (0 jobs, no logs) on push to a non-default branch

Symptom:

gh run view <id>            → "This run likely failed because of a workflow file issue"
gh run view <id> --log-failed → "failed to get run log: log not found"
gh api .../runs/<id>/jobs --jq .total_count → 0
gh api .../runs/<id>       --jq .event      → push   (head_branch: dev / a feature branch)

The run’s .name shows the file path (.github/workflows/foo.yml) instead of the workflow’s name: field — GitHub never parsed far enough to read name:. Run conclusion is failure but zero jobs were created, so there is no --log-failed output.

Affected workflows (observed 2026-08-04): Wasabi SOPS update post-rotation (wasabi-sops-update.yml, on: workflow_dispatch only) and apply-supabase-migrations.yml (on: push with a paths: filter). 4 runs, all on dev / feature branches (#5528).

Root cause: These are GitHub startup failures — the run is created but no job ever starts. They fire on push to non-default branches for workflows that should not have matched the push at all (a workflow_dispatch-only file, or a path-filtered one whose paths weren’t touched). The workflow files themselves are valid (python3 -c 'import yaml; yaml.safe_load(...)' passes; the on: block is clean) — so this is a GitHub-side push-evaluation artifact on stale/legacy branches (dev is “historical reference only” per CLAUDE.md), not a real job failure and not a broken workflow. The check for whether it’s the same-old artifact vs. genuine breakage is: does the file actionlint clean, and is it on the default branch?

Not to be confused with Group P: Group P (no matching creation rules found) is a real job failure inside wasabi-sops-update.yml’s SOPS step (already fixed by #3880). Group W is the opposite — the job never runs, there is no log, and the workflow logic is irrelevant.

Fix (#5528): scripts/gha-failed-jobs-audit.py now detects startup_failures by job count (run_job_count(run_id) == 0) and:

• skips them on non-default branches (transient push-artifacts — the noise class that filed #5528),
• on the default branch (AUDIT_DEFAULT_BRANCH, default main) files an accurate Group W issue (validate with actionlint, check reusable-workflow uses: refs) instead of the un-actionable “unknown — log analysis required” template.

Prevention: never classify a 0-job run from --log-failed — there is nothing to read. If a startup_failure recurs on main, lint the workflow file; off-main startup_failures on legacy branches are safe to ignore/close.

---

Group X — Password Rotation Check exit 1 is a true-positive alert, not a bug (+ inline-Python Discord 403)

Symptom:

Check overdue credential rotations  Discord notification failed: HTTP Error 403: Forbidden
Check overdue credential rotations  ##[error]1 CRITICAL credential(s) have overdue rotation. Immediate action required.
Check overdue credential rotations  ##[error]Process completed with exit code 1.

Affected workflow: Password Rotation Check (pwd-rotation-check.yml), scheduled Mondays 08:00 UTC.

Two independent things happen in one failed run — do not conflate them:

1. The exit 1 is intentional (working as designed). The final step Fail job if critical credentials overdue (lines 120-130) exits 1 on purpose when dev_r_services has a criticality=critical credential whose next_due < today. The job failure is the alarm — it is a true positive, not a workflow bug. Do not “fix” the workflow to stop it failing. The remediation is to rotate the flagged credential, which is credential work (secret-manager role), not a dev/CI fix.

   • 2026-08-10 (#6073): the flagged critical was GH_PAT (last_rotated 2026-05-08, 90d → next_due 2026-08-06, genuinely overdue). It was already tracked by the open alert issue #5822 (CredentialRotationOverdue: GH_PAT) plus the retire-and-migrate plan #5841/#5842 (move GH_PAT off a static PAT onto GitHub App installation tokens). So the failed-gh-actions issue was a duplicate signal of an already-owned rotation — cross-reference #5822 and close, do not open a second rotation issue.
   • Before escalating a flagged credential as human-action: check its tier in docs/playbooks/secret-rotation-access-matrix.md. GitHub PATs (GH_TOKEN, GITHUB_PAT_*, GH_PAT) are Tier 2 — dispatch to the secret-manager queue, never label human-action.
   • False-positive check (stale data, cf. #209): if the flagged credential was actually rotated but its next_due in dev_r_services was never advanced, the fix is a data update (bump last_rotated/next_due), not a rotation. Verify last_rotated + rotation_freq against reality before acting.

2. The Discord notification failed: HTTP Error 403: Forbidden is a real, separate bug — but it is caught and non-fatal (the try/except at lines 109-113 only prints), so it does not cause the exit 1. Root cause is the **same Cloudflare-edge UA block as 2666: the step’s inline Python posts to the webhook with urllib.request and no User-Agent header, so it goes out as Python-urllib/3.x, which Discord’s Cloudflare edge 403s before the request reaches the webhook. This is distinct from Group H (dead webhook → 404 / Unknown Webhook code 10015) — a 403 on a webhook that still exists is the UA block, not a deleted webhook.

Fix (#6073): add an explicit User-Agent header to the urllib.request.Request in the Format and post Discord report step (matches the fix already shipped in agent-push-error.py, compliance-audit-due-check.py, wasabi-replication-repair.py, discord-provisioning.py). The exit-1 half needs no workflow change — it correctly surfaced a genuine overdue rotation.

Audit-noise fix (#6997): even with the exit-1 correctly understood as a true positive, the daily gha-failed-jobs-audit.py kept re-filing a [failed-gh-actions] investigation issue for it on every Monday run while a critical stayed overdue (#6073, then #6997) — duplicate noise, since the genuine overdue rotation is already owned by the dedicated CredentialRotationOverdue alert issue (e.g. #5822). Added "Password Rotation Check" to SKIP_WORKFLOWS in scripts/gha-failed-jobs-audit.py, same treatment as Group Y’s Lint PR closing keywords (#6589) and Group I’s Enforce-gitflow gate. The rotation itself is still surfaced by its own alert; the workflow’s exit-1 needs no change.

Prevention: any inline-Python-in-YAML step that POSTs to a Cloudflare-fronted endpoint (Discord, the meta-dispatcher Worker) must send an explicit User-Agent; the default Python-urllib/3.x is edge-blocked with 403. When a “rotation check” / “compliance check” workflow exit-1s, first ask is it alerting correctly? before treating the red run as broken CI.

---

Group Y — Lint PR closing keywords red run is a blocking PR-quality gate firing as designed, not a bug

Symptom:

Fail on full-close keyword + unchecked acceptance items  Violation on issue number(s): 6583
Fail on full-close keyword + unchecked acceptance items  ##[error]This PR uses a full closing keyword while an acceptance checkbox is still unchecked.
Fail on full-close keyword + unchecked acceptance items  ##[error]Process completed with exit code 1.

Affected workflow: Lint PR closing keywords (lint-pr-closing-keywords.yml) — specifically its second job, lint-implements-unchecked-items (“Fail on full-close keyword + unchecked acceptance items”).

This is the guardrail working, not an infra fault. Job 2 is a blocking check (#6413) for the recurring #4677 class: it deliberately exit 1s when a PR body pairs a full closing keyword (Implements:/Closes:/Fixes:/Resolves: #N) with an unchecked acceptance checkbox (- [ ]) for the same issue — which would otherwise auto-close still-unfinished work on merge. The red X is the alarm: the author must either check the remaining box(es) or switch to the Partially implements: #N opt-out before the PR can merge. Do not “fix” the workflow to stop it failing.

Job 1 (Warn on native-linker vs partial-scope contradiction) in the same workflow is non-blocking and always succeeds; only Job 2 turns the whole run red.

Why the daily audit used to file a bug issue for this (#6589): gha-failed-jobs-audit.py groups failed runs by workflow name and had no entry for this one in SKIP_WORKFLOWS, so every correctly-blocked PR produced a false-positive [failed-gh-actions] investigation issue. Both jobs already file their own bug issue + Discord ping on a genuine internal step error (their if: failure() notify steps run before the intentional exit 1), so the audit added only noise.

Fix (#6589): added "Lint PR closing keywords" to SKIP_WORKFLOWS in scripts/gha-failed-jobs-audit.py (same treatment as Group I’s Enforce-gitflow gate). Genuine faults remain covered by the workflow’s own notify steps. When a red run appears on a PR, the correct action is to fix the PR body (check the box or use Partially implements: #N), not to touch the workflow.

---

Group Z — trivy Permission denied (exit 126) on the shared ~/.local/bin/trivy-bin/trivy — parallel-install race

Symptom:

/opt/gh-runner-3/_work/_actions/aquasecurity/trivy-action/v0.36.0/entrypoint.sh: line 88: /home/claude-runner/.local/bin/trivy-bin/trivy: Permission denied
##[error]Process completed with exit code 126.

One or more Trivy CVE scan (<image>) matrix jobs in Image CVE Scan fail on exec of the trivy binary. Same exit code (126) and same shared-$HOME mechanism as Group L (sops Text file busy), but the error text is Permission denied rather than Text file busy — the losing job execs the binary in the window after another job has created/truncated it but before chmod +x lands.

Affected workflow: Image CVE Scan (image-scan.yml) — its 22-image matrix runs runs-on: [self-hosted, bms4] with fail-fast: false, so up to 3 jobs run at once across bms-4’s 3 p24-infra runners.

Root cause pattern: bms-4’s gh-runner, gh-runner-2, gh-runner-3 all run as the same claude-runner user and share one $HOME. aquasecurity/trivy-action invokes aquasecurity/setup-trivy, which installs to the fixed path $HOME/.local/bin/trivy-bin/trivy and does not expose a per-job install dir through trivy-action (only version/cache/token are forwarded). Concurrent matrix jobs therefore all write the same binary; the entrypoint resolves trivy from PATH (TRIVY_CMD="${TRIVY_CMD:-trivy}") and execs it mid-write → exit 126. Pure concurrency race, so it looks intermittent (the shared binary is left executable after the last writer, which is why a post-mortem ls shows it -rwxr-xr-x).

Fix (#6835): install trivy ourselves into a per-job $RUNNER_TEMP/trivy-bin (fresh dir per job even on a shared $HOME, same remedy as Group S) via the pinned contrib/install.sh, prepend it to $GITHUB_PATH, and set skip-setup-trivy: "true" on trivy-action so it reuses that binary from PATH instead of the shared install. No two jobs ever touch the same file.

Distinguishing signature (Group Z vs Group L):

Observation	Group Z — trivy install race	Group L — sops install race
Exit code	126	126
Error text	…/trivy-bin/trivy: Permission denied	…/sops: Text file busy
Workflow	Image CVE Scan	secrets-sync
Fix	per-job $RUNNER_TEMP + skip-setup-trivy (#6835)	atomic/idempotent in-place install (#3625)

Prevention: same lesson as Groups K/L/S — never let parallel jobs install/exec a tool binary at a shared $HOME path. Use a per-job $RUNNER_TEMP dir (when the install path is not overridable, as with trivy-action) or an atomic download-to-temp-then-mv (when you control the install step).

---

Group AA — deploy-vps-scripts-transient-ssh: transient SSH/git failure in Deploy VPS scripts

Taxonomy key: deploy-vps-scripts-transient-ssh

First seen: 2026-10-05 (runs 37298220732 and 37366608495, issue #7504)

Affected workflow: Deploy VPS scripts (deploy-vps-scripts.yml) — SSH-deploys /opt/p24-infra to vps-i1 (217.154.82.162) and bms-4 (54.36.123.110) on every push to main that touches scripts/**.

Distinguishing signatures

Pattern A — vps-i1 or bms-4 git lock (step fails in <10 s)

The SSH step completes in under 10 seconds (typical network round-trip is 3–4 s; 5 s = almost no work done). Log shows one of:

Another git process seems to be running in this repository
fatal: Unable to create '/opt/p24-infra/.git/index.lock': File exists

Root cause: another git process (a worker’s freshness gate, a prior deploy run that was interrupted, or a concurrent git fetch) left .git/index.lock. The deploy’s git fetch or git reset --hard fails immediately.

Pattern B — bms-4 ensure-dispatcher-cloudflared-pin.sh download failure

The bms-4 SSH step fails after ~20–30 s with:

[cloudflared-pin] ERROR: download failed from https://github.com/cloudflare/cloudflared/releases/...

or:

[cloudflared-pin] ERROR: checksum mismatch after download (got ..., expected ...)

Root cause: the pinned cloudflared-2026.5.1 binary at /opt/p24-infra/bin/cloudflared-2026.5.1 was missing or corrupted (not a git-tracked file, so git reset --hard does not restore it), and the GitHub Releases download failed due to transient network or rate-limiting. The script was added to the bms-4 deploy step in issue #7898 specifically to self-heal this binary on every deploy cycle.

Pattern C — GitHub Actions auto-cancel (conclusion=cancelled surfacing as failure)

A run is cancelled by GitHub before any job starts (0 jobs executed, empty steps array). This happens when a deploy is queued during a GitHub Actions capacity event and then auto-cancelled after many hours in the queue. The audit script sees conclusion=failure (GH CLI normalises cancelled to failure in run listings), so it files an issue — but no action is needed since the next push to main triggers a fresh deploy.

Distinguishing Group AA from other groups

Feature	Group AA-A (git lock)	Group AA-B (cloudflared download)	Group AA-C (auto-cancel)
Step duration	<10 s	20–30 s	N/A (no steps ran)
Fails on host	vps-i1 or bms-4	bms-4 only	—
Log keyword	index.lock / Another git process	[cloudflared-pin] ERROR	empty steps array
Retry self-heals?	yes — lock expires when the other process finishes	yes — next push retriggers download	yes — next push triggers a new run
Action required	none (transient)	none if next run succeeds; investigate network if recurring	none

Fix / recovery

All patterns: verify the next triggered run (push to main) succeeds. These are all self-healing. If runs continue to fail:

• Pattern A: SSH into the host and run rm -f /opt/p24-infra/.git/index.lock, then re-trigger via gh workflow run deploy-vps-scripts.yml --repo radieu/p24-infra.
• Pattern B: check egress from bms-4 to github.com; verify scripts/ensure-dispatcher-cloudflared-pin.sh --check exits 0 on the host after a successful run. If the binary is genuinely missing, run the script manually on bms-4 as root.
• Pattern C: no action — the cancel was a GH platform event.

Structural fix (issue #7504)

Added concurrency: group: deploy-vps-scripts, cancel-in-progress: false to deploy-vps-scripts.yml. This queues overlapping pushes instead of cancelling them, eliminating the main trigger for Pattern A (concurrent deploy runs racing on the same git lock). Pattern B is mitigated by ensure-dispatcher-cloudflared-pin.sh running on every deploy, so a missing binary is restored within one deploy cycle rather than persisting until a manual restore (#7898).

---

Group AB — per-session SSH signing key missing after Claude Code session cleanup

Taxonomy key: n8n-backup-commit-signing-failure

First seen: 2026-10-09 (daily recurrence through 2026-10-14, issue #7697)

Affected workflows: n8n Workflow Backup (n8n-backup.yml) step “Commit if workflows changed”, n8n Workflow Snapshot (n8n-workflow-snapshot.yml) step “Commit if changed”. Any other workflow that runs git commit on a bms-4 self-hosted runner as the claude-runner OS user is susceptible.

Distinguishing signature

error: Couldn't load public key /home/claude-runner/.local/share/p24-worker/signing-keys/signing-key-NNNN-XXXXX: No such file or directory?
fatal: failed to write commit object

The claude-runner OS user on bms-4 configures SSH commit signing in ~/.gitconfig with a per-session key file path. When a Claude Code session ends, the key file is cleaned up. The global ~/.gitconfig still references the now-missing key. Any workflow that runs git commit as the same OS user inherits this broken signing config.

Fix

Add git config commit.gpgsign false immediately before the git commit in the failing step:

- name: Commit if workflows changed
  run: |
    git config commit.gpgsign false   # override inherited per-session signing config
    git commit -m "chore: ..."

This is a local config write that overrides the user-level signing config for that step only. Disabling signing is correct for automation steps — per-session ephemeral keys provide no meaningful audit trail.

Fixed: #7697 (PR #8032 — git config commit.gpgsign false added to n8n-backup.yml and n8n-workflow-snapshot.yml).

Detection signal for future occurrences: Couldn't load public key + signing-key-NNNN-XXXXX

• fatal: failed to write commit object in any commit step of a workflow running on bms-4 as claude-runner.

---

Aggregated statistics (2026-06-30 session)

Group	Workflows affected	Count	Status
A — Python test regression	Python Tests, P24-Infra Self Test, n8n Backup	~10	FIXED (PR #2193)
B — Missing SSH backup key	Grafana Backup	1+	OPEN
C — Expired n8n API key	n8n Snapshot	1+	OPEN
D — Stale Supabase DB password	DB Maintenance	1+	OPEN
E — BOM in HTTP header	compliance-audit-due-check	1+	OPEN
F — vps-i1 unstaged changes	Prometheus AI Triage	1+	OPEN
G1 — GH App missing (Dependabot)	Dispatch to Worker Queue	2	OPEN (design fix needed)
G2 — GH_TOKEN GraphQL permissions	n8n Snapshot, Infra Task Request	1+	OPEN
H — Dead Discord webhook	Grafana Backup + many	ongoing	OPEN (tracked in priorities)
I — Gitflow block (by design)	Enforce gitflow	1	N/A — working as intended
J — Branch YAML issue	db-maintenance-check, CF security-check	2	RESOLVED (branch merged/deleted)
K — Queued + job-level EACCES (drift sudoers)	secrets-sync, Self Test, gitflow, dispatch	4	RESOLVED (2026-07-08 #3324); drift sudoers tracked 3344
L — Text file busy (exit 126) sops parallel-install race	secrets-sync	20 (last 25h)	FIXED (#3625 — atomic/idempotent install)
M — GH API rate-limit 403 in sync-gh-secrets	secrets-sync	6 (5 in 25h through 2026-07-13)	FIXED (#3697 retry/backoff + #4089 wait-until-reset); transient/self-healing
N — receiver exit-2 canary invalid dotenv line: {	SOPS Sync Receiver	1	RESOLVED (2026-07-09); code fixed 3412 (PR #3531), drift reconciled #3609
O — GH006 bot push to protected main	Credential Rotation, n8n Backup/Snapshot, Rotation Log Housekeeping, wasabi-sops-update	3+	OPEN — systemic fix tracked #3357; n8n variant #3640
P — sops --encrypt input path misses .sops.yaml rule	wasabi-sops-update	1 (2026-07-11)	FIXED (#3880 6957ebf0 — stage encrypt input under secrets/*-tmp.env.sops); #3983 re-surface closed stale
Q — docker-compose container-name conflict on a container the project does not own	secrets-sync (sync-pinbox24-backends / mailgun-prod)	1 (2026-07-19)	FIXED (#4290 — gated in-step stop+rename+retry recovery)
R — gh issue create --label aborts on missing label	Ansible drift detection	1 (2026-07-20)	FIXED (#4377 — created drift-detected label + idempotent gh label create --force in reporter); underlying vps-i1 heartbeat-cron supabase_url undefined tracked separately
S — shared-$HOME race on ~/.ssh/id_bms1 (sibling scrub rms the key mid-SSH)	secrets-sync (bms-1 jobs: w3, w4, backends, mezmo)	2 (2026-07-19)	FIXED (#4376 — per-job $RUNNER_TEMP/id_bms1); #4381 re-surface closed stale
V — OpenAI Admin API 404 on non-existent endpoint path	rotate-openai-etop-key	3 (2026-06-30 ×2, 2026-08-04)	RESOLVED (#5320 — broken one-shot retired/deleted; residual manual OPENAI_API_KEY rotation escalated to human)
W — startup_failure (0 jobs, no logs) on push to non-default branch	wasabi-sops-update, apply-supabase-migrations	4 (2026-08-04)	FIXED (#5528 — audit skips off-main startup_failures by job-count; no more “unknown” noise issues)
X — rotation-check exit 1 = true-positive alert (overdue credential) + inline-Python Discord 403 (missing UA)	Password Rotation Check	1 (2026-08-10)	exit-1 WORKING AS INTENDED (GH_PAT overdue, already tracked 5842); Discord 403 FIXED (#6073 — explicit User-Agent, same class as 2666)
Y — Lint PR closing keywords red = blocking PR-quality gate firing by design (#4677 class)	Lint PR closing keywords	4 (last 25h, 2026-08-20)	N/A — working as intended; FIXED audit noise (#6589 — added to SKIP_WORKFLOWS, same as Group I)
Z — trivy Permission denied (exit 126) shared-$HOME parallel-install race	Image CVE Scan	1 (2026-09-02)	FIXED (#6835 — per-job $RUNNER_TEMP install + skip-setup-trivy; same class as Groups L/S)
AA — transient SSH/git failure (git lock, cloudflared download, GH auto-cancel)	Deploy VPS scripts	3 (2026-10-05 ×2 + 2026-10-13)	MITIGATED (#7504 — concurrency:queue; #7898 — cloudflared pin self-heal on every deploy)
AB — per-session SSH signing key missing after Claude Code session cleanup	n8n Workflow Backup, n8n Workflow Snapshot	6+ (2026-10-09–14, daily)	FIXED (#7697 — git config commit.gpgsign false before commit in both workflows)

---

Cross-session comparison — recurring patterns

Pattern	Seen in	Verdict
Python test mock mismatch after refactor	2026-06-30	Structural: tests don’t track refactors fast enough
Dead Discord webhook	Every session	Long-standing: webhook needs replacement
Expired API key (n8n, backup SSH)	Recurring	No credential rotation automation in place
vps-i1 dirty working tree	Recurring	Agents leave unstaged changes; no gitignore for agent output
Dependabot + GH App token	Recurring	Workflow needs Dependabot exemption condition
BOM in credential file	2026-06-30	Windows dev machine write path issue (see SOPS playbook)
“Queued workflows” misread as runner outage	2026-07-08 (#3324)	Check runner API status + whether any job hit in_progress before restarting (Group K)