GH Actions Failed Jobs — Recurring Failure Taxonomy

Living document. Update after each “failed jobs” review session. Last reviewed: 2026-08-05 (added Group W — startup_failure runs, 0 jobs / no logs, GitHub emitting conclusion: failure on push to non-default branches for workflow_dispatch-only / path-filtered workflows like wasabi-sops-update.yml; the audit now skips these off-main and no longer files “unknown — log analysis required” noise, #5528) Previously: 2026-07-20 (added Group S — shared-$HOME race on ~/.ssh/id_bms1 between the concurrent bms-1 sync jobs in secrets-sync.yml; sibling job’s if: always() scrub rms the shared key mid-SSH, #4381 stale re-surface, already fixed by #4376) Previously: 2026-07-20 (added Group R — gh issue create --label aborts when the label is missing, ansible-drift.yml reporter swallowed its own drift signal, #4377) Previously: 2026-07-19 (added Group Q — compose container-name conflict on a container the compose project does not own, sync-pinbox24-backends/mailgun-prod, #4290; previously listed as a still-open failure under Group L) Previously: 2026-07-12 (added Group P — sops --encrypt input path misses .sops.yaml creation rule, 3880; added Group O — bot auto-commit push rejected by main branch protection GH006, 3640; Group N is sops-sync-receiver exit-2 canary invalid dotenv input line: {, #3633; Group M is GH API rate-limit 403 in sync-gh-secrets, #3697; Group L is Text file busy sops install race, #3625)

How to use this playbook

Open GH Actions → filter by status: failure
For each failing workflow, match against a Group below
Apply the listed fix; record any new root cause at the bottom

Quick lookup:

gh run list --status failure --limit 30
gh run view <id> --log-failed

Group A — Python test regression in `test_sync_n8n_supabase_creds.py`

Symptom:

FAILED scripts/tests/test_sync_n8n_supabase_creds.py::TestExportCredential::test_happy_path_returns_dict
FAILED scripts/tests/test_sync_n8n_supabase_creds.py::TestExportCredential::test_retries_empty_export_then_succeeds
RuntimeError: exported credential file empty or unreadable

Affected workflows: Python Tests, P24-Infra Self Test, n8n Workflow Backup, any workflow with pytest step

Root cause pattern: The sync-n8n-supabase-creds.py script was refactored but the test mocks were not updated to match the new export_credential API. Mock doesn’t return a non-empty file so retry exhaustion raises RuntimeError.

History:

2026-06-30 morning: ~10 failed runs (both bms-4 and GH-hosted runner)
Fixed by PR #2193 (fix(#2122): update export_credential tests for #2120 single-exec refactor) merged ~10:24

Fix: Update test mocks to match new export_credential signature. Prevention: When refactoring a function that has unit tests, always run pytest -x locally on the test file before pushing.

Group B — Missing SSH key for backup jobs

Symptom:

getKeyFile error: open /.ssh/grafana_backup_key: no such file or directory
ssh: handshake failed: ssh: unable to authenticate, attempted methods [none], no supported methods remain
Process completed with exit code 1.

Affected workflows: Grafana Backup, n8n Backup (any using appleboy/drone-ssh)

Root cause pattern: The GH Secret GRAFANA_BACKUP_SSH_KEY (or equivalent) is missing or empty. The workflow writes it to ~/.ssh/grafana_backup_key but there is nothing to write.

History:

2026-06-30: Grafana Backup failing (run 28426550041)

Fix:

Check the GH Secret exists: gh secret list | grep GRAFANA_BACKUP_SSH_KEY
If missing: generate or retrieve the SSH private key for backup user on vps-i1, re-add via gh secret set GRAFANA_BACKUP_SSH_KEY
The secret must be the raw private key content (PEM format, no passphrase)

Prevention: After any credential rotation affecting backup SSH keys, immediately re-run the affected workflow manually to confirm.

Group C — Expired/rotated n8n API key

Symptom:

curl: (22) The requested URL returned error: 401
# n8n API endpoint returns 401 Unauthorized

Affected workflows: n8n Workflow Snapshot, n8n Backup

Root cause pattern: BMS4_N8N_API_KEY GH Secret is stale — n8n API key was regenerated in n8n Settings but the GH Secret was not updated.

History:

2026-06-30: n8n workflow snapshot failing (run 28428432130)

Fix:

SSH into bms-4: ssh root@54.36.123.110
In n8n: Settings → API → regenerate key (or check if existing key works)
Update GH Secret: gh secret set BMS4_N8N_API_KEY --body "<new-key>"

Prevention: When rotating n8n API key, add a checklist step to update all GH Secrets that reference it.

Group D — Stale Supabase DB password (direct postgres connection)

Symptom:

FATAL: password authentication failed for user "postgres"
curl: (22) The requested URL returned error: 404  # on Discord notification step

Affected workflows: DB Maintenance (db-maintenance.yml)

Root cause pattern: SUPABASE_DB_PASSWORD GH Secret contains an old password. Supabase postgres password was rotated (e.g. during sb_secret key migration) without updating this secret.

History:

2026-06-30: DB Maintenance failing (run 28425367096)

Fix:

Get current DB password from Supabase dashboard → Project Settings → Database → Connection string
Update: gh secret set SUPABASE_DB_PASSWORD --body "<new-password>"
Re-run workflow manually to confirm

Note: The SUPABASE_SERVICE_KEY → sb_secret migration (PR 896, 2026-06-21) may have also rotated the direct postgres password. Always check both after any Supabase credential rotation.

Group E — BOM character in HTTP header (UnicodeEncodeError)

Symptom:

UnicodeEncodeError: 'latin-1' codec can't encode character '' in position 7: ordinal not in range(256)

Affected workflows: compliance-audit-due-check

Root cause pattern: A file being read and used as an HTTP Authorization header contains a UTF-8 BOM (). This happens when:

A secret file was written with BOM on Windows (Out-File, Set-Content, > redirection)
A SOPS file was decrypted with BOM-adding Windows tools

History:

2026-06-30: compliance-audit-due-check failing (run 28434831724)

Fix:

Identify which file/secret is being read as the auth header in the script
Strip BOM: sed -i 's/^\xEF\xBB\xBF//' <file> or re-write with UTF8Encoding($false) on Windows
If the secret was written to GH Secrets with BOM, delete and re-add the secret

Prevention: Always use [System.IO.File]::WriteAllText($path, $content, [System.Text.UTF8Encoding]::new($false)) on Windows. See docs/playbooks/sops-windows-crlf.md.

Group F — vps-i1 unstaged changes blocking git pull

Symptom:

error: cannot pull with rebase: You have unstaged changes.
error: Please commit or stash them.
Process completed with exit code 128.

Affected workflows: Prometheus Alerts → AI Sysadmin Triage, any workflow that SSHes into vps-i1 and runs git pull

Root cause pattern: An agent or manual operation left uncommitted changes in /opt/p24-infra on vps-i1. The git pull --ff-only or git pull --rebase in the workflow step then fails.

History:

2026-06-30: Prometheus AI Triage failing (run 28436140120)

Fix:

ssh root@217.154.82.162
cd /opt/p24-infra
git status           # identify the dirty files
git stash            # stash if changes are temp / exploratory
# OR
git diff             # review changes
git add -A && git commit -m "chore: save agent workdir state"

Prevention:

Agents on vps-i1 should never modify tracked files in /opt/p24-infra directly
If a script must write config, write to /opt/p24-infra/.local/ (gitignored) or a temp file

Group G — GH Token / App credential issues

G1 — GitHub App `app-id` missing (Dependabot context)

Symptom:

throw new Error("Input required and not supplied: app-id");
Error: Input required and not supplied: app-id
Secret source: Dependabot

Affected workflows: Dispatch to Worker Queue (dispatch-to-queue.yml)

Root cause: Dependabot PRs run with a restricted token that does not have access to repository Actions secrets. The actions/create-github-app-token@v1 step requires APP_ID + PRIVATE_KEY which are not available.

Fix: Add a condition to skip dispatch when triggered by Dependabot:

if: github.actor != 'dependabot[bot]'

Or alternatively, grant Dependabot access to these specific secrets in repo Settings → Secrets.

History: 2026-06-30 (run 28425183531)

G2 — GH_TOKEN insufficient for GraphQL labels

Symptom:

error fetching labels: GraphQL: Resource not accessible by integration (repository.labels)

Affected workflows: n8n workflow snapshot (error-handler step), Infra Task Request — enqueue

Root cause: The GITHUB_TOKEN on bms-4 runner does not have issues: write permission, or the PAT used lacks repo-level label access. The error occurs in the error-handler step that tries to add a bug label.

Fix: The workflow needs permissions: issues: write in the job definition, OR the GH_TOKEN secret needs broader scope.

Group H — Dead Discord webhook

Symptom:

{"message": "Unknown Webhook", "code": 10015}
# OR
curl: (22) The requested URL returned error: 404

Affected workflows: Grafana Backup, any workflow sending Discord alerts via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL or DISCORD_WEBHOOK_URL

Root cause: The Discord webhook URL stored in GH Secrets has been deleted or expired in the Discord server. Error notifications from all infrastructure jobs silently drop.

Status: Already tracked in docs/priorities.md P2 section.

Fix:

Create new Discord webhook in target channel → Integrations → Webhooks → New Webhook → Copy URL
Update GH Secret: gh secret set DISCORD_WEBHOOK_URL --body "<new-url>"
And also: gh secret set P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL --body "<new-url>"
Test with: curl -X POST "$URL" -H "Content-Type: application/json" -d '{"content":"test"}'

Group I — Gitflow enforcement block (OBSOLETE since #2513)

⚠️ No longer applies. The feat/* → dev → rc/* → main flow was retired on 2026-07-08 (#2513); enforce-gitflow.yml was updated to pass unconditionally. p24-infra feat/*/fix/* branches now PR directly to main — that is the correct, expected flow, not a failure. This section is kept for historical reference; a genuine “PRs to main must come from ‘dev’/‘rc/*’” block should no longer fire. If you see it, the enforcing workflow is itself stale — fix the workflow, do not reroute the PR through the frozen dev branch.

Historical symptom (no longer expected):

BLOCKED: PRs to main must come from 'dev' or 'rc/*'
Got: fix/2007-waha-monitoring-off
Process completed with exit code 1.

Affected workflows: Enforce gitflow — main branch (now passes unconditionally)

Group J — Workflow YAML file issues on branch

Symptom:

X This run likely failed because of a workflow file issue.
(no log output available)

Affected workflows: db-maintenance-check, cloudflare-security-check

Root cause pattern: A push on a branch containing a broken workflow YAML file causes GitHub to report “workflow file issue” for ALL workflows triggered by that push. The branch hotfix/delete-waha-cf-worker caused this on 2026-06-30.

Fix:

Identify the bad branch: gh run view <id> --json headBranch
Check the workflow YAML on that branch for syntax errors
If the branch is merged/deleted, the issue auto-resolves

Note: These failures do NOT indicate the workflow itself is broken on main/dev.

Group K — Runs stuck `queued` + job-level `EACCES` on the self-hosted runner (NOT a runner outage)

Symptom:

# Multiple GH Actions runs sit in `queued` for 20–30+ min, then one job that DID start logs:
Error: EACCES: permission denied, stat '/root/.local/bin/git'

Affected workflows: Sync secrets to VPSes, P24-Infra Self Test, Enforce gitflow, Dispatch to Worker Queue — anything routed to the bms-4 / secrets-deploy self-hosted runner.

Root cause pattern: The runner process is healthy — it never crashed. Jobs stall because a job-level step needs elevated/sudo access the claude-runner user no longer has (sudoers drift): the sops/git install/stat step trips EACCES: permission denied, stat '/root/.local/bin/git'. The queued backlog is a side effect of secrets-deploy failover waiting on the specific labelled runner, not evidence the runner is offline. Tracked separately as the drift-sudoers gap: **#3291 / 3344 (claude-runner not in sudoers for the sops install step).

Why this matters — do NOT misdiagnose as a runner crash: the natural first read of “many runs queued 25min” is “runner is down → svc.sh restart” (see gh-runner-restart-offline.md). That restart is wasted here — the runner was never down. Confirm liveness BEFORE restarting:

# Runner alive? (registration Online/Idle → it is NOT a runner-outage case)
gh api repos/radieu/p24-infra/actions/runners --jq '.runners[] | {name, status, busy}'
# A recent job that reached "in_progress" (even if it later failed) also proves the runner is live.
gh run list --repo radieu/p24-infra --limit 10 --json databaseId,status,conclusion,name

Distinguishing signature (Group K vs. a real runner outage):

Observation	Group K — job-level drift sudoers	Real runner outage (`gh-runner-restart-offline.md`)
Runner `status` via API	`online`	`offline`
Any job reaches `in_progress`?	yes (then fails on EACCES)	no — all stay `queued`
Error text	`EACCES … stat '/root/.local/bin/git'`	`Failed to create a session` / `Unauthorized` / none
Fix	restore `claude-runner` sudoers (#3291/#3344) — NOT a restart	`systemctl restart` or re-register

Fix: restore the claude-runner sudoers entry for the sops install step (see #3291 / #3344). Do not restart the runner. Once sudoers is fixed, requeued/re-run jobs proceed.

History:

2026-07-08 (#3324): 4 workflows queued >25min after 03:58 UTC; initially reported as “runners offline”. Owner confirmed the runner was healthy throughout — root cause was job-level drift sudoers (EACCES). secrets-sync partially completed (sync-pinbox24-backends OK, W4 up HTTP 200); old Wasabi key deleted manually. Side gaps split out: drift sudoers (#3291/#3344) and a stale sync-bms-4 SSH key (#3345, since closed).

Prevention: before treating “queued workflows” as a runner outage, check runner API status and whether any job reached in_progress. If a job started and failed on EACCES, it is a job-level permission gap — not a runner crash.

Group L — `Text file busy` (exit 126) on the shared `~/.local/bin/sops` — parallel-install race

Symptom:

/opt/gh-runner-3/_work/_temp/<uuid>.sh: line 6: /home/claude-runner/.local/bin/sops: Text file busy
##[error]Process completed with exit code 126.

Several secrets-sync jobs fail simultaneously in their Install sops step with exit 126. Distinct from Group K (that is EACCES/exit 1 from sudoers drift; this is ETXTBSY/exit 126).

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — any run where 2+ of its 13 self-hosted, secrets-deploy jobs execute concurrently.

Root cause pattern: All secrets-deploy runner instances on bms-4 (gh-runner, gh-runner-3, …) run as the same claude-runner user and therefore share one $HOME — so every job’s Install-sops step wrote the same file ~/.local/bin/sops. When job A’s curl -o ~/.local/bin/sops is still writing the binary while job B tries to ~/.local/bin/sops --version, the kernel returns ETXTBSY (cannot exec a file open for writing) → exit 126. It is a pure concurrency race: it fires only when jobs overlap, so it looks intermittent.

Fix (#3625): made the install idempotent and atomic in all 13 blocks — skip the download if the pinned version (3.9.1) is already on PATH; otherwise curl to a per-job mktemp file, chmod, then mv -f it into ~/.local/bin/sops. The shared path is never held open for writing during an exec, so ETXTBSY cannot occur, and concurrent installs converge on an identical binary.

Distinguishing signature (Group L vs Group K):

Observation	Group L — sops install race	Group K — drift sudoers
Exit code	126	1
Error text	`… /sops: Text file busy`	`EACCES … stat '/root/.local/bin/git'`
Trigger	2+ jobs overlapping	any job needing sudo the user lost
Fix	atomic/idempotent install (#3625)	restore sudoers (#3291/#3344)

Prevention: never write a shared-$HOME tool binary in place from parallel jobs. Download to a temp file and atomically mv into place, and short-circuit when the pinned version already exists.

Note — separate, still-open failures in the same run (not fixed here; track individually):

sync-pinbox24-backends exit 1 — mailgun-v42-prod container name conflict on docker compose up. Now classified as Group Q and fixed with in-step conflict recovery (#4290).
sync-gh-secrets exit 1 — failure while processing P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (transient GH API rate-limit — now classified as Group M and hardened with retry/backoff, #3697).

Group M — GH API rate-limit exhaustion in `sync-gh-secrets` (HTTP 403, transient)

Symptom:

  P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
failed to fetch public key: HTTP 403: API rate limit exceeded for user ID 11704852 …
  (https://api.github.com/repos/radieu/p24-infra/actions/secrets/public-key)
##[error]Process completed with exit code 1.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — the sync-gh-secrets job. Any workflow that calls gh secret set in a loop is vulnerable.

Root cause pattern: gh secret set must first GET .../actions/secrets/public-key to encrypt each value. That request counts against the shared GH_PAT account’s 5000/hr core rate budget (user ID 11704852). Many agents, dispatchers, and the daily audits all authenticate as that same user, so the budget can be momentarily exhausted. When the public-key fetch lands during exhaustion it returns HTTP 403 rate-limit exceeded and — before #3697 — failed the entire sync on the first key. This is transient and self-healing: the budget resets hourly, and prior/subsequent runs succeed.

Distinguishing signature: error text is HTTP 403: API rate limit exceeded for user ID … on the .../actions/secrets/public-key endpoint. NOT EACCES (Group K), NOT Text file busy/exit 126 (Group L). It is a GitHub-API quota issue, not a runner or sudoers problem.

Diagnosis / confirm transient:

gh run list --repo radieu/p24-infra --workflow secrets-sync.yml --limit 8 \
  --json databaseId,status,conclusion,createdAt      # neighbours succeed → transient
gh api rate_limit --jq .resources.core               # remaining/reset for the current token

Fix / recovery:

Re-run just the affected job (idempotent — re-writes the same SOPS values): gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=gh-secrets
If the core budget is still near-zero, wait for reset before retrying.

Prevention (#3697 + #4089):

#3697 — wrapped gh secret set in a retry-with-backoff helper (set_secret_with_retry, 5 attempts, 15s→30s→60s→120s ≈ 4 min total window) in the sync-gh-secrets job, so a single transient 403/network blip no longer fails the whole sync.
#4089 — added wait_for_core_budget: called once at job start (pre-flight) and again from set_secret_with_retry whenever the captured error matches API rate limit exceeded|HTTP 403. It reads gh api rate_limit, and if core.remaining < 50 it sleeps until core.reset + 10s (capped at 3600s). Handles sustained exhaustion — a multi-minute drain where the ~4 min retry backoff window from #3697 wasn’t long enough. timeout-minutes bumped 10 → 65 so the wait can complete inside the job.

History:

2026-07-11 (#3697): 1 failed run (29131871037, 00:06 UTC) — 403 on the first key (P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL). Confirmed transient (neighbouring runs OK); a target=gh-secrets re-run (29148035117) succeeded. Added retry/backoff hardening.
2026-07-12 → 2026-07-13 (#4089): 5 failed runs in 25h, e.g. 29241072075 at 10:01:44 UTC — 403 on the same key across all 5 retry attempts. The retry backoff (max 120s between attempts) landed inside the same exhausted hourly window every time, so retries burned without progress. Fixed by pre-flight wait_for_core_budget + rate-limit-aware retry that sleeps until reset.

Group N — sops-sync-receiver exit 2: `canary decrypt failed … invalid dotenv input line: {`

Symptom:

[SOPS-RECEIVER] sops replaced key <CREDENTIAL>
[SOPS-RECEIVER] ERROR exit=2 canary decrypt failed after encrypt — NOT committing: invalid dotenv input line: {
##[error]Process completed with exit code 2.

Affected workflows: SOPS Sync Receiver (sops-sync-receiver.yml → scripts/sops-sync-receiver.py)

Root cause pattern: A rotation producer built the payload with a non-scalar value — either a nested JSON object ({"value": "..."}) or a double-encoded JSON string of that object ('{"value": "..."}') — instead of a plain string. update_sops() then serialized the dotenv line KEY={"value": "..."}, which begins with {; SOPS’s dotenv parser rejects it at the post-encrypt canary decrypt, so the receiver aborts with exit 2 BEFORE committing. This is the safety fence working as designed — the SOPS file is left intact, no corruption ships. The new credential is simply never persisted (the rotation must be re-run once the producer is fixed).

Not a runner/secret problem: the receiver did its job. The bug is upstream in whatever built the payload. Do not “fix” by editing SOPS by hand.

Fix (already shipped — verify it is present before re-deriving):

_coerce_string_value() unwraps both a dict value and a str value that (stripped) starts with { and JSON-decodes to a dict — extracting the scalar via the value/new_value field.
validate_payload() rejects any value still {-prefixed after coercion as an explicit exit-1 validation error, so this class never again reaches the SOPS writer as a confusing exit-2 canary failure.
Covered by scripts/tests/test_sops_sync_receiver.py. If a NEW variant slips through, add the offending shape as a test case and extend _coerce_string_value().

Drift check after any occurrence: the rotate script verifies + applies the new credential to the live service before dispatching to the receiver, so an exit-2 abort can leave live ≠ SOPS drift (new value live, SOPS still old). Confirm the live service health and, if a real rotation (non-empty LOG_ID / a dev_r_rotation_log row) was in flight, re-run the rotation or reconcile SOPS. A log_id=none + workflow_dispatch run with no dev_r_rotation_log row is a manual repro/test, not a tracked production rotation — usually no drift.

History:

2026-07-08: W3_APP_MONGODB_PASSWORD rotation payload malformed → exit-2 canary abort (run 28951846700). Real live/SOPS drift for w3_app did occur in this window and caused a v32-prod auth-fail crash-loop, reconciled by #3609 (rotate w3_app password to match). The receiver code path was hardened by #3171 (dict unwrap) and **#3412 / PR 3531 (double- encoded JSON-string unwrap + exit-1 reject; 18 tests). Failure-notification dedup: #3413.
2026-07-09: daily gha-failed-jobs-audit.py re-surfaced the now-stale 28951846700 run as #3633 (root cause already fixed the previous night; no further code action).

Prevention: rotation producers must send plain-string scalars in new_value/new_values. When adding a new scripts/rotate/*.sh producer, assert the payload value is a JSON string (not an object) before dispatch.

Group O — Bot auto-commit `git push` rejected by `main` branch protection (GH006)

Symptom:

[warn] git push failed: remote: error: GH006: Protected branch update failed for refs/heads/main.
remote: - Changes must be made through a pull request.
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to 'https://github.com/radieu/p24-infra'

Affected workflows: Credential Rotation (rotate-credentials.py → sops_update_key), n8n Workflow Backup, n8n Workflow Snapshot, Rotation Log Housekeeping, wasabi-sops-update.yml — any workflow that runs on the default ref (main) and does git commit && git push back to it.

Root cause pattern: main is a protected branch requiring PRs. Any GH Actions job that commits and pushes directly to the checked-out main ref is rejected with GH006, regardless of the token used (scoped GITHUB_TOKEN, GH_PAT, or GH_TOKEN) unless that identity is on the branch-protection bypass allowlist. The push failure is often logged only as a [warn] (best-effort), so the job may still exit 0 while the commit is silently dropped.

Why this matters — silent data loss, not just a red run: for credential rotation the sequence is create new token → gh_secret_set (✓) → sops_update_key push (✗ GH006) → delete old token. Net result: GH Secrets/provider hold the new value, but the SOPS repo keeps the OLD (now-deleted, dead) value. secrets-sync would then deploy the dead credential. The weekly scheduled rotation hits this every week.

Fix (design): push bot commits to a chore/auto-rotate-<date> branch and open a PR to main (the branch-protection-compatible path) instead of pushing to the checked-out ref — see the update-vps-h1-sops.yml pattern (branch + gh pr create). Alternative: add the automation identity to the branch-protection bypass allowlist (repo-admin decision).

Workaround (manual rotation): dispatch the workflow with --ref <feature-branch> so the push lands on a branch, then open + merge a PR (used for #2701 → PR #3354).

Tracked in:

#3357 — credential-rotation sops_update_key push to protected main (full analysis + fix design)
#3640 — n8n Workflow Backup commit step rejected by GH006 (same pattern, different workflow)
#3350 — stale Wasabi GH Secret causing InvalidAccessKeyId (the co-occurring failed=1 in the Credential Rotation run — a separate root cause, not the push failure)

History:

2026-07-08 (#3357): Credential Rotation rotated 3 CF tokens; provider + GH Secret updated but SOPS push to main rejected → SOPS retained dead values.
2026-07-09 (#3632): daily failed-jobs audit re-surfaced the same Credential Rotation failure (Wasabi InvalidAccessKeyId set failed=1 → exit 1; CF SOPS pushes rejected GH006 as before). Both root causes already tracked (#3357 + #3350) → closed as duplicate.

Prevention: treat GH006 in any auto-commit workflow as this group — do NOT re-file per-workflow duplicates; route to #3357 (systemic fix) and the workflow-specific tracker. Do not git push to main from CI without a bypass-allowlisted identity.

Group P — `sops --encrypt` input path does not match a `.sops.yaml` creation rule

Symptom:

error loading config: no matching creation rules found
##[error]Process completed with exit code 1.

The job builds the updated plaintext fine (e.g. logs Updated keys: {...}) and fails on the very next sops --encrypt call.

Affected workflows: wasabi-sops-update.yml (and any workflow that re-encrypts a SOPS file from a staging plaintext).

Root cause pattern: SOPS matches .sops.yaml’s creation_rules[].path_regex against the input file path passed to --encrypt, not against the --output destination. The catch-all rule is (^|[/\\])secrets[/\\].*\.env\.sops$, so an input written to /tmp/monitoring-updated.env matches no rule → “no matching creation rules found”, even though --output points at secrets/monitoring.env.sops.

Fix (#3880, commit 6957ebf0): write the staging plaintext to a path that matches the rule — secrets/monitoring-tmp.env.sops — and pass that as the encrypt input (sops --encrypt --output secrets/monitoring.env.sops secrets/monitoring-tmp.env.sops). The secrets/*-tmp.env.sops name follows the CLAUDE.md §Secrets convention and is rm’d before the commit step. Precursors on the same workflow: #3281 (dispatch ref dev→main + path_regex) and #3332 (unparseable YAML).

Secret-free reproduction:

printf 'K=x\n' > /tmp/foo.env             # sops --encrypt /tmp/foo.env            -> "no matching creation rules found" (exit 1)
printf 'K=x\n' > secrets/foo-tmp.env.sops # sops --encrypt secrets/foo-tmp.env.sops -> succeeds (exit 0); rm the tmp file after

Prevention: when re-encrypting from a staging file, the encrypt input path (not just --output) must satisfy a .sops.yaml path_regex. Stage under secrets/*-tmp.env.sops.

History:

2026-07-11 21:57 UTC: run 29169690473 (manual workflow_dispatch) failed with this error on the pre-fix workflow. Fix 6957ebf0 (#3880) merged 22:12 UTC — ~15 min later.
2026-07-12: daily gha-failed-jobs-audit.py re-surfaced the now-stale 21:57 run as #3983 (root cause already fixed by #3880; no further code action) — closed as stale, like 3632.

Group Q — `docker-compose up` container-name conflict on a container the project does not own

Symptom:

Creating mailgun-v42-prod ... error
ERROR: for mailgun  Cannot create container for service mailgun: Conflict. The container name
"/mailgun-v42-prod" is already in use by container "45a2873db599…". You have to remove
(or rename) that container to be able to reuse that name.
Encountered errors while bringing up the project.
##[error]Process completed with exit code 1.

Affected workflows: secrets-sync.yml — sync-pinbox24-backends (Restart mailgun-prod). Any step that runs docker-compose up -d --force-recreate <svc> against a compose file with a fixed container_name is vulnerable: the W3 (backend/s3/reso/socket) and W4 restart steps in the same workflow share the shape.

Root cause pattern: --force-recreate only recreates containers the invoking compose project owns — matched via the com.docker.compose.project / .service labels, which are derived from the directory name compose was invoked from. When the name is held by a container created out-of-band (a hotfix run, a separate GitLab-CI build-project directory, or a plain docker run), compose finds nothing to recreate, falls through to create, and Docker rejects the duplicate container_name.

The exit-1 is the visible half. The dangerous half is silent: the env file the step just synced never reaches the running container, so the stack keeps authenticating with the pre-rotation credential — the exact failure mode of #3688 (mailgun) and #3958 (v32-prod backend), where docker ps, PM2 status, and health checks all look green while every request hangs to a 504.

Fix (#4290): in-step recovery, gated on the conflict signature so unrelated compose failures still abort untouched:

Capture docker-compose up -d --no-deps --force-recreate <svc> output; on success, done.
On failure, match the output for is already in use by container. No match → exit 1 unchanged.
Match → docker stop <name> + docker rename <name> <name>-stale-$(date +%Y%m%d%H%M%S) (rename, never remove — keeps a zero-risk rollback), then retry the compose up once.
Retry fails → rename back, docker start, exit 1.

After one successful recovery the new container is compose-owned, so the path stops triggering. The renamed *-stale-* container is left stopped for inspection; removing it is a follow-up sys-admin action.

Diagnosing which project owns the name (on the host):

docker inspect mailgun-v42-prod --format '{{index .Config.Labels "com.docker.compose.project"}}'
# empty or != the directory the workflow cd's into  →  Group Q

Prevention: any new deploy/restart step that force-recreates a fixed-container_name service should carry the same recovery guard, or drop container_name from the compose file so compose can name containers itself. Related playbooks: mailgun-mongodb-stale-credential-hang.md §Fix (the manual form of this procedure), v32-prod-mongoose-authfail-storm.md §Variant (#3958 — same conflict silently skipping the v32-prod backend), bms1-container-audit.md.

Group R — `gh issue create --label <name>` aborts when the label does not exist (exit 1)

Symptom:

could not add label: 'drift-detected' not found
##[error]Process completed with exit code 1.

Affected workflows: ansible-drift.yml (Ansible drift detection — “Open drift issue” step). Any workflow whose reporting step calls gh issue create --label "<x>,<y>" / gh issue comment against a label that was never created in the repo is vulnerable.

Root cause pattern: gh issue create --label X is fail-closed — if label X does not exist in the repo, the command exits 1 (it does NOT auto-create the label). The drift workflow reports drift by opening/commenting a drift-detected,bug-labelled issue, but drift-detected was never created in radieu/p24-infra (only sops-drift existed). So the moment the check found real drift and tried to report it, the reporting step itself died — turning the run RED and swallowing the drift signal it was built to surface. This is the same “the reporter is the thing that breaks” shape as Group A’s exit $RC trap.

Here the drift being reported was legitimate: the vps-i1 --check had failed=1 because the claude-runner role’s p24-server-heartbeat cron task references undefined supabase_url / supabase_service_key vars (Error while resolving value for 'job': 'supabase_url' is undefined). That ansible bug is a separate defect tracked on its own — it is exactly the kind of drift the check exists to report; it must not also crash the reporter.

Fix (#4377):

Created the missing drift-detected label in the repo (gh label create drift-detected --color d93f0b --force).
Made the “Open drift issue” step self-heal: it now runs an idempotent gh label create drift-detected … --force 2>/dev/null || true before any gh issue list/create/comment, so a deleted/renamed label can never re-break the reporter.

Prevention: any workflow step that applies a label via gh must ensure the label exists first (gh label create … --force is idempotent) — never assume a repo label is present.

Group S — shared-`$HOME` race on `~/.ssh/id_bms1` between concurrent bms-1 sync jobs

Symptom:

Warning: Identity file /home/claude-runner/.ssh/id_bms1 not accessible: No such file or directory.
root@94.23.26.113: Permission denied (publickey).
lost connection
##[error]Process completed with exit code 1.

Fires in a bms-1 SSH/SCP step (Deploy W4 persistent-patches to bms-1, Ship to bms-1, any W3/W4 step) even though the same step’s own printf … | base64 -d > ~/.ssh/id_bms1 ran two lines earlier. Intermittent — only when 2+ bms-1 jobs overlap.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — the concurrent bms-1 jobs sync-pinbox24-w3, sync-pinbox24-w4, sync-pinbox24-backends, sync-bms-1 (mezmo). All wrote their SSH key to the same path ~/.ssh/id_bms1.

Root cause pattern: every secrets-deploy runner instance on bms-4 runs as the same claude-runner user and shares one $HOME (the same shared-$HOME substrate as Groups K & L). Each bms-1 step did touch ~/.ssh/id_bms1 … > ~/.ssh/id_bms1, used it with ssh/scp -i ~/.ssh/id_bms1, and — critically — each job’s Scrub plaintext secrets step (if: always()) ran rm -f ~/.ssh/id_bms1. When job A’s scrub deletes the shared key mid-flight while job B’s SSH step is still running, B’s -i ~/.ssh/id_bms1 resolves to a now-missing file → “Identity file … not accessible: No such file or directory” → “Permission denied (publickey)” → exit 1. It is a pure delete-out-from-under race, so it looks intermittent and hits whichever bms-1 job happens to still be mid-SSH when a sibling finishes.

Distinguishing signature (Group S vs Groups K/L):

Observation	Group S — id_bms1 delete race	Group K — drift sudoers	Group L — sops install race
Error text	`id_bms1 not accessible: No such file` + `Permission denied (publickey)`	`EACCES … stat '/root/.local/bin/git'`	`… /sops: Text file busy`
Exit code	1	1	126
Trigger	2+ bms-1 jobs overlap; sibling scrub `rm`s the shared key	any job needing lost sudo	2+ jobs overlap on Install-sops
Fix	per-job `$RUNNER_TEMP/id_bms1` (#4376)	restore sudoers (#3291/#3344)	atomic/idempotent install (#3625)

Fix (#4376, merged 2026-07-20 11:53 UTC): migrated every bms-1 SSH-key write, -i reference, and scrub in secrets-sync.yml from the shared ~/.ssh/id_bms1 to the per-job-unique $RUNNER_TEMP/id_bms1 ($RUNNER_TEMP is a fresh dir per job even on a shared-$HOME runner, so concurrent jobs can no longer clobber or delete each other’s key; the rm -f "$RUNNER_TEMP/id_bms1" scrub only removes that job’s own copy). Verify with grep -c 'RUNNER_TEMP/id_bms1' .github/workflows/secrets-sync.yml (61 usages) and grep -nE '~/\.ssh/id_bms1' … (must be zero).

History:

2026-07-19: 2 failed runs — 29681365216 (09:19 UTC, sync-pinbox24-w3) and 29691164407 (14:37 UTC, sync-pinbox24-w4 / Deploy W4 persistent-patches). Both on the pre-fix shared-path workflow. Fixed by #4376 the next morning.
2026-07-20: daily gha-failed-jobs-audit.py re-surfaced the two now-stale runs as #4381 (root cause already fixed by #4376; no further code action) — closed as stale, like 3632.

Prevention: never write/read/rm a per-run credential file at a shared path under $HOME from parallel jobs on a shared-$HOME self-hosted runner. Use $RUNNER_TEMP (per-job) for any transient key/secret material. This is the same lesson as Groups K & L — treat $HOME as shared, $RUNNER_TEMP as private.

Group T — `scp: <file>.js: Is a directory` — Docker bind-mount placeholder on bms-1

Symptom:

scp: /home/gitlab-runner/builds/.../p24-v-3.2/persistent-patches//crudService.js: Is a directory
scp: /home/gitlab-runner/builds/.../p24-v-3.2/persistent-patches//filesUpload.js: Is a directory
scp: .../mongoose.js: Is a directory
scp: .../pushNotification.helper.js: Is a directory
scp: .../twilioService.js: Is a directory
##[error]Process completed with exit code 1.

Fires in Deploy W3 persistent-patches to bms-1 (or the W4 / s3-v2 equivalents). Not intermittent — once it starts, every subsequent run of the job fails identically until the server state is cleared. Distinguishes from Group S by the error text: Group S fails to connect (Permission denied (publickey)), Group T connects fine and fails on the destination path.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — sync-pinbox24-w3, sync-pinbox24-w4, and the Deploy s3-v2-v42-prod patches step.

Root cause pattern: two mechanisms compound.

persistent-patches/ lives inside a GitLab runner build directory (/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/ for W3, /root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/ for W4). The .js patch files are untracked there, so a GitLab CI checkout/clean wipes them — the identical failure class as the 2026-07-09 W3 incident that emptied backend-environment.env / s3-environment.env.
Docker auto-creates a directory at any bind-mount source path that does not exist. The next v32-prod / v42-prod start therefore replaces each wiped file with an empty directory of the same name, and scp cannot overwrite a directory.

The tell that it is the Docker mechanism and not something else: the set of failing filenames matches exactly the bind-mount list in infra-src/pinbox24/w3/docker-compose.yml. In the 2026-07-30 occurrence the 5 failures were precisely the files mounted into v32-prod (lines 10–15); controller.js and local.js, mounted only into s3-v32-prod, were untouched because that service had not been restarted in the window.

Fix (#4652): each patch-deploy step now runs a pre-scp guard over SSH that rmdirs any *.js path in the target dir that is a directory:

for stale in "$PP"/*.js; do
  [ -d "$stale" ] || continue
  if rmdir "$stale" 2>/dev/null; then echo "removed stale Docker-created directory: $stale"
  else echo "::error::$stale is a non-empty directory — manual cleanup required on bms-1"; exit 1; fi
done

rmdir rather than rm -rf is deliberate: Docker’s placeholders are always empty, so rmdir succeeds on exactly the intended case and refuses to destroy real content on a production path — a non-empty directory aborts the step loudly instead. The job is now self-healing: the following Deploy W3/W4 containers step recreates the containers so they pick up the restored files (a running container keeps the old mount inode until it is recreated).

History:

2026-07-21 19:04 — last green run before the break.
2026-07-29 — a W3 GitLab pipeline ran on bms-1; docs/priorities.md records chown -R gitlab-runner:gitlab-runner being applied to persistent-patches/ because the files were “root-owned from Docker” — i.e. the placeholders were already being created around then.
2026-07-29 18:03 → 2026-07-30 07:29 — 6 consecutive failed runs, filed as #4652 with root-cause group unknown. Fixed by #4652.

Prevention: treat any bind-mount source path inside a CI build directory as volatile. If a deploy step copies files that Docker also mounts, guard for the placeholder-directory state before copying — a missing bind-mount source never fails loudly, it silently becomes a directory.

Group U — Vercel `403 ENV_ALREADY_EXISTS` on env-var create (duplicate create rejected)

Symptom:

HTTP 403 on POST /v9/projects/prj_***/env: {"error":{"code":"ENV_ALREADY_EXISTS",
"message":"Another Environment Variable with the same Name and Environment exists in your
project. Remove it or choose a different Name or Environment.","key":"MONGODB_URI"}}
urllib.error.HTTPError: HTTP Error 403: Forbidden
##[error]Process completed with exit code 1.

Affected workflows: Sync secrets to VPSes (secrets-sync.yml) — sync-et-lager / Sync et-lager secrets to Vercel.

Root cause pattern: not a credential problem — the bare HTTP Error 403: Forbidden in the traceback reads like a revoked token, but the response body names the real cause. #2838 had changed this job to POST-then-DELETE ordering (create the replacement first, then retire the old entry) so that a failed write could never leave the project with no value for the key — et-lager.env.sops holds a single key (MONGODB_URI), so an empty window there means total DB failure for et-lager. That ordering depends on Vercel accepting a duplicate key+target create. Vercel now rejects it outright, and the guard only recognised 409:

if e.code != 409 or not stale_ids:
    raise

so the delete-then-create fallback never fired. The sibling sync-et-operational-platform job kept passing in the same runs because it still uses plain delete-then-create — proof the token and project access were fine.

Second-order bug found alongside it: api() did print(f"… {e.read().decode()}") and then raise. e.read() is a one-shot stream, so the print consumed the only copy of the body and any caller was structurally unable to inspect Vercel’s error code.

Fix (#4652): the job now PATCHes the existing entry in place (PATCH /v9/projects/{prj}/env/{id} with only {"value": …}, leaving key/target/type untouched) and POSTs only when the key is genuinely absent. PATCH is a single atomic update with no window at all, so it satisfies #2838’s requirement strictly better than either ordering while sidestepping the duplicate-create rejection entirely. Retained as fallbacks: delete-then-create if the PATCH is rejected, and a re-GET-then-PATCH recovery if a concurrent run creates the key between the pre-loop GET and the POST. api() now stashes the body on the exception (e.p24_body) before printing.

Conflict detection matches on the body code, not the status alone, so a genuine 403 (revoked token, lost project access) still fails loudly rather than being mistaken for a conflict.

PATCH precedent in this repo: rotate-openai-etop-key.yml PATCHes /v9/projects/{prj}/env/{id} for an encrypted-type var. Caveat: PATCH does not work for type: sensitive vars — see the wasabi-admin-sync.yml comment “Delete existing (can’t change sensitive type via PATCH)”. et-lager and et-op both use encrypted.

History:

2026-07-21 19:04 — last green sync-et-lager run. The job triggers on every push to main (no path filter), so once Vercel tightened duplicate handling every push failed it.
2026-07-29 18:03 → 2026-07-30 07:29 — 6 consecutive failed runs, same 6 runs as Group T (two independent bugs in one workflow, filed together as #4652). Fixed by #4652.

Prevention: when a sync job writes to a third-party API, branch on the body error code, not the HTTP status alone — and never consume an error stream in a log line before the handler that needs it. Prefer an idempotent update verb (PATCH/PUT) over create-then-delete or delete-then-create ordering games; both orderings have a failure window, an atomic update has none.

Group V — OpenAI Admin API `404 Not Found` on a non-existent endpoint path

Symptom:

rotate	OpenAI GET /v1/organization/api_keys?limit=100 → 404:
rotate	urllib.error.HTTPError: HTTP Error 404: Not Found
rotate	##[error]Process completed with exit code 1.

(The error body is empty — a 404 on the path, not a JSON error object.)

Affected workflows: rotate-openai-etop-key (#5320; failed all 3 dispatches — 2026-06-30 ×2, 2026-08-04).

Root cause: The workflow calls OpenAI Admin API paths that do not exist. The auth is fine (a bad OPENAI_ADMIN_KEY would 401, not 404) — the paths themselves are wrong:

Workflow used	Reality
`GET /v1/organization/api_keys` (line 51)	no such endpoint → 404 (the observed failure)
`POST /v1/organization/projects/{id}/api_keys` (line 66)	no create-project-API-key endpoint exists — would fail next even after fixing line 51
`DELETE /v1/organization/api_keys/{id}` (line 94)	no such endpoint

The correct OpenAI Admin API shape is project-scoped:

List project keys: GET /v1/organization/projects/{project_id}/api_keys
Delete a project key: DELETE /v1/organization/projects/{project_id}/api_keys/{key_id}
Admin (org-management) keys: GET|POST|DELETE /v1/organization/admin_api_keys
Programmatic creation of a usable key is only possible via a service account: POST /v1/organization/projects/{project_id}/service_accounts returns api_key.value (an sk-svcacct-… key). Plain human project API keys cannot be minted through the API.

So this workflow’s whole premise (create a plain project key via API, capture its value) is unworkable without switching to service-account keys — a security/ownership design change, not a one-line path fix.

Fix (#5320): the workflow was a one-shot rotation for the #2269 exposure (“Triggered manually only. After successful run, delete this workflow file.”) that never once succeeded. It was retired (deleted) rather than rewritten — a security-sensitive, prod-mutating, untestable rewrite would give false confidence. The residual OPENAI_API_KEY rotation for et-op was escalated to a human (verify via the OpenAI dashboard + rotate manually per docs/playbooks/static-api-key-incident-rotation.md, then update the Vercel OPENAI_API_KEY env var for the et-op project).

Prevention: before shipping an Admin-API automation, confirm each path against the current OpenAI API reference — org key management is project-scoped (/organization/projects/{id}/api_keys), and usable keys are minted via service accounts, never a top-level /organization/api_keys create.

Group W — `startup_failure` (0 jobs, no logs) on `push` to a non-default branch

Symptom:

gh run view <id>            → "This run likely failed because of a workflow file issue"
gh run view <id> --log-failed → "failed to get run log: log not found"
gh api .../runs/<id>/jobs --jq .total_count → 0
gh api .../runs/<id>       --jq .event      → push   (head_branch: dev / a feature branch)

The run’s .name shows the file path (.github/workflows/foo.yml) instead of the workflow’s name: field — GitHub never parsed far enough to read name:. Run conclusion is failure but zero jobs were created, so there is no --log-failed output.

Affected workflows (observed 2026-08-04): Wasabi SOPS update post-rotation (wasabi-sops-update.yml, on: workflow_dispatch only) and apply-supabase-migrations.yml (on: push with a paths: filter). 4 runs, all on dev / feature branches (#5528).

Root cause: These are GitHub startup failures — the run is created but no job ever starts. They fire on push to non-default branches for workflows that should not have matched the push at all (a workflow_dispatch-only file, or a path-filtered one whose paths weren’t touched). The workflow files themselves are valid (python3 -c 'import yaml; yaml.safe_load(...)' passes; the on: block is clean) — so this is a GitHub-side push-evaluation artifact on stale/legacy branches (dev is “historical reference only” per CLAUDE.md), not a real job failure and not a broken workflow. The check for whether it’s the same-old artifact vs. genuine breakage is: does the file actionlint clean, and is it on the default branch?

Not to be confused with Group P: Group P (no matching creation rules found) is a real job failure inside wasabi-sops-update.yml’s SOPS step (already fixed by #3880). Group W is the opposite — the job never runs, there is no log, and the workflow logic is irrelevant.

Fix (#5528): scripts/gha-failed-jobs-audit.py now detects startup_failures by job count (run_job_count(run_id) == 0) and:

skips them on non-default branches (transient push-artifacts — the noise class that filed #5528),
on the default branch (AUDIT_DEFAULT_BRANCH, default main) files an accurate Group W issue (validate with actionlint, check reusable-workflow uses: refs) instead of the un-actionable “unknown — log analysis required” template.

Prevention: never classify a 0-job run from --log-failed — there is nothing to read. If a startup_failure recurs on main, lint the workflow file; off-main startup_failures on legacy branches are safe to ignore/close.

Group X — `Password Rotation Check` exit 1 is a true-positive alert, not a bug (+ inline-Python Discord 403)

Symptom:

Check overdue credential rotations  Discord notification failed: HTTP Error 403: Forbidden
Check overdue credential rotations  ##[error]1 CRITICAL credential(s) have overdue rotation. Immediate action required.
Check overdue credential rotations  ##[error]Process completed with exit code 1.

Affected workflow: Password Rotation Check (pwd-rotation-check.yml), scheduled Mondays 08:00 UTC.

Two independent things happen in one failed run — do not conflate them:

The exit 1 is intentional (working as designed). The final step Fail job if critical credentials overdue (lines 120-130) exits 1 on purpose when dev_r_services has a criticality=critical credential whose next_due < today. The job failure is the alarm — it is a true positive, not a workflow bug. Do not “fix” the workflow to stop it failing. The remediation is to rotate the flagged credential, which is credential work (secret-manager role), not a dev/CI fix.
- 2026-08-10 (#6073): the flagged critical was GH_PAT (last_rotated 2026-05-08, 90d → next_due 2026-08-06, genuinely overdue). It was already tracked by the open alert issue #5822 (CredentialRotationOverdue: GH_PAT) plus the retire-and-migrate plan #5841/#5842 (move GH_PAT off a static PAT onto GitHub App installation tokens). So the failed-gh-actions issue was a duplicate signal of an already-owned rotation — cross-reference #5822 and close, do not open a second rotation issue.
- Before escalating a flagged credential as human-action: check its tier in docs/playbooks/secret-rotation-access-matrix.md. GitHub PATs (GH_TOKEN, GITHUB_PAT_*, GH_PAT) are Tier 2 — dispatch to the secret-manager queue, never label human-action.
- False-positive check (stale data, cf. #209): if the flagged credential was actually rotated but its next_due in dev_r_services was never advanced, the fix is a data update (bump last_rotated/next_due), not a rotation. Verify last_rotated + rotation_freq against reality before acting.
The Discord notification failed: HTTP Error 403: Forbidden is a real, separate bug — but it is caught and non-fatal (the try/except at lines 109-113 only prints), so it does not cause the exit 1. Root cause is the **same Cloudflare-edge UA block as 2666: the step’s inline Python posts to the webhook with urllib.request and no User-Agent header, so it goes out as Python-urllib/3.x, which Discord’s Cloudflare edge 403s before the request reaches the webhook. This is distinct from Group H (dead webhook → 404 / Unknown Webhook code 10015) — a 403 on a webhook that still exists is the UA block, not a deleted webhook.

Fix (#6073): add an explicit User-Agent header to the urllib.request.Request in the Format and post Discord report step (matches the fix already shipped in agent-push-error.py, compliance-audit-due-check.py, wasabi-replication-repair.py, discord-provisioning.py). The exit-1 half needs no workflow change — it correctly surfaced a genuine overdue rotation.

Prevention: any inline-Python-in-YAML step that POSTs to a Cloudflare-fronted endpoint (Discord, the meta-dispatcher Worker) must send an explicit User-Agent; the default Python-urllib/3.x is edge-blocked with 403. When a “rotation check” / “compliance check” workflow exit-1s, first ask is it alerting correctly? before treating the red run as broken CI.

Aggregated statistics (2026-06-30 session)

Group	Workflows affected	Count	Status
A — Python test regression	Python Tests, P24-Infra Self Test, n8n Backup	~10	FIXED (PR #2193)
B — Missing SSH backup key	Grafana Backup	1+	OPEN
C — Expired n8n API key	n8n Snapshot	1+	OPEN
D — Stale Supabase DB password	DB Maintenance	1+	OPEN
E — BOM in HTTP header	compliance-audit-due-check	1+	OPEN
F — vps-i1 unstaged changes	Prometheus AI Triage	1+	OPEN
G1 — GH App missing (Dependabot)	Dispatch to Worker Queue	2	OPEN (design fix needed)
G2 — GH_TOKEN GraphQL permissions	n8n Snapshot, Infra Task Request	1+	OPEN
H — Dead Discord webhook	Grafana Backup + many	ongoing	OPEN (tracked in priorities)
I — Gitflow block (by design)	Enforce gitflow	1	N/A — working as intended
J — Branch YAML issue	db-maintenance-check, CF security-check	2	RESOLVED (branch merged/deleted)
K — Queued + job-level EACCES (drift sudoers)	secrets-sync, Self Test, gitflow, dispatch	4	RESOLVED (2026-07-08 #3324); drift sudoers tracked 3344
L — `Text file busy` (exit 126) sops parallel-install race	secrets-sync	20 (last 25h)	FIXED (#3625 — atomic/idempotent install)
M — GH API rate-limit 403 in sync-gh-secrets	secrets-sync	6 (5 in 25h through 2026-07-13)	FIXED (#3697 retry/backoff + #4089 wait-until-reset); transient/self-healing
N — receiver exit-2 canary `invalid dotenv line: {`	SOPS Sync Receiver	1	RESOLVED (2026-07-09); code fixed 3412 (PR #3531), drift reconciled #3609
O — GH006 bot push to protected main	Credential Rotation, n8n Backup/Snapshot, Rotation Log Housekeeping, wasabi-sops-update	3+	OPEN — systemic fix tracked #3357; n8n variant #3640
P — `sops --encrypt` input path misses `.sops.yaml` rule	wasabi-sops-update	1 (2026-07-11)	FIXED (#3880 `6957ebf0` — stage encrypt input under `secrets/*-tmp.env.sops`); #3983 re-surface closed stale
Q — docker-compose container-name conflict on a container the project does not own	secrets-sync (sync-pinbox24-backends / mailgun-prod)	1 (2026-07-19)	FIXED (#4290 — gated in-step stop+rename+retry recovery)
R — `gh issue create --label` aborts on missing label	Ansible drift detection	1 (2026-07-20)	FIXED (#4377 — created `drift-detected` label + idempotent `gh label create --force` in reporter); underlying vps-i1 heartbeat-cron `supabase_url` undefined tracked separately
S — shared-`$HOME` race on `~/.ssh/id_bms1` (sibling scrub `rm`s the key mid-SSH)	secrets-sync (bms-1 jobs: w3, w4, backends, mezmo)	2 (2026-07-19)	FIXED (#4376 — per-job `$RUNNER_TEMP/id_bms1`); #4381 re-surface closed stale
V — OpenAI Admin API 404 on non-existent endpoint path	rotate-openai-etop-key	3 (2026-06-30 ×2, 2026-08-04)	RESOLVED (#5320 — broken one-shot retired/deleted; residual manual OPENAI_API_KEY rotation escalated to human)
W — `startup_failure` (0 jobs, no logs) on push to non-default branch	wasabi-sops-update, apply-supabase-migrations	4 (2026-08-04)	FIXED (#5528 — audit skips off-`main` startup_failures by job-count; no more “unknown” noise issues)
X — rotation-check exit 1 = true-positive alert (overdue credential) + inline-Python Discord 403 (missing UA)	Password Rotation Check	1 (2026-08-10)	exit-1 WORKING AS INTENDED (GH_PAT overdue, already tracked 5842); Discord 403 FIXED (#6073 — explicit User-Agent, same class as 2666)

Cross-session comparison — recurring patterns

Pattern	Seen in	Verdict
Python test mock mismatch after refactor	2026-06-30	Structural: tests don’t track refactors fast enough
Dead Discord webhook	Every session	Long-standing: webhook needs replacement
Expired API key (n8n, backup SSH)	Recurring	No credential rotation automation in place
vps-i1 dirty working tree	Recurring	Agents leave unstaged changes; no gitignore for agent output
Dependabot + GH App token	Recurring	Workflow needs Dependabot exemption condition
BOM in credential file	2026-06-30	Windows dev machine write path issue (see SOPS playbook)
“Queued workflows” misread as runner outage	2026-07-08 (#3324)	Check runner API `status` + whether any job hit `in_progress` before restarting (Group K)

p24-infra Docs

Explorer

gh-actions-failed-jobs-taxonomy

GH Actions Failed Jobs — Recurring Failure Taxonomy

How to use this playbook

Group A — Python test regression in `test_sync_n8n_supabase_creds.py`

Group B — Missing SSH key for backup jobs

Group C — Expired/rotated n8n API key

Group D — Stale Supabase DB password (direct postgres connection)

Group E — BOM character in HTTP header (UnicodeEncodeError)

Group F — vps-i1 unstaged changes blocking git pull

Group G — GH Token / App credential issues

G1 — GitHub App `app-id` missing (Dependabot context)

G2 — GH_TOKEN insufficient for GraphQL labels

Group H — Dead Discord webhook

Group I — Gitflow enforcement block (OBSOLETE since #2513)

Group J — Workflow YAML file issues on branch

Group K — Runs stuck `queued` + job-level `EACCES` on the self-hosted runner (NOT a runner outage)

Group L — `Text file busy` (exit 126) on the shared `~/.local/bin/sops` — parallel-install race

Group M — GH API rate-limit exhaustion in `sync-gh-secrets` (HTTP 403, transient)

Group N — sops-sync-receiver exit 2: `canary decrypt failed … invalid dotenv input line: {`

Group O — Bot auto-commit `git push` rejected by `main` branch protection (GH006)

Group P — `sops --encrypt` input path does not match a `.sops.yaml` creation rule

Group Q — `docker-compose up` container-name conflict on a container the project does not own

Group R — `gh issue create --label <name>` aborts when the label does not exist (exit 1)

Group S — shared-`$HOME` race on `~/.ssh/id_bms1` between concurrent bms-1 sync jobs

Group T — `scp: <file>.js: Is a directory` — Docker bind-mount placeholder on bms-1

Group U — Vercel `403 ENV_ALREADY_EXISTS` on env-var create (duplicate create rejected)

Group V — OpenAI Admin API `404 Not Found` on a non-existent endpoint path

Group W — `startup_failure` (0 jobs, no logs) on `push` to a non-default branch

Group X — `Password Rotation Check` exit 1 is a true-positive alert, not a bug (+ inline-Python Discord 403)

Aggregated statistics (2026-06-30 session)

Cross-session comparison — recurring patterns

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

gh-actions-failed-jobs-taxonomy

GH Actions Failed Jobs — Recurring Failure Taxonomy

How to use this playbook

Group A — Python test regression in test_sync_n8n_supabase_creds.py

Group B — Missing SSH key for backup jobs

Group C — Expired/rotated n8n API key

Group D — Stale Supabase DB password (direct postgres connection)

Group E — BOM character in HTTP header (UnicodeEncodeError)

Group F — vps-i1 unstaged changes blocking git pull

Group G — GH Token / App credential issues

G1 — GitHub App app-id missing (Dependabot context)

G2 — GH_TOKEN insufficient for GraphQL labels

Group H — Dead Discord webhook

Group I — Gitflow enforcement block (OBSOLETE since #2513)

Group J — Workflow YAML file issues on branch

Group K — Runs stuck queued + job-level EACCES on the self-hosted runner (NOT a runner outage)

Group L — Text file busy (exit 126) on the shared ~/.local/bin/sops — parallel-install race

Group M — GH API rate-limit exhaustion in sync-gh-secrets (HTTP 403, transient)

Group N — sops-sync-receiver exit 2: canary decrypt failed … invalid dotenv input line: {

Group O — Bot auto-commit git push rejected by main branch protection (GH006)

Group P — sops --encrypt input path does not match a .sops.yaml creation rule

Group Q — docker-compose up container-name conflict on a container the project does not own

Group R — gh issue create --label <name> aborts when the label does not exist (exit 1)

Group S — shared-$HOME race on ~/.ssh/id_bms1 between concurrent bms-1 sync jobs

Group T — scp: <file>.js: Is a directory — Docker bind-mount placeholder on bms-1

Group U — Vercel 403 ENV_ALREADY_EXISTS on env-var create (duplicate create rejected)

Group V — OpenAI Admin API 404 Not Found on a non-existent endpoint path

Group W — startup_failure (0 jobs, no logs) on push to a non-default branch

Group X — Password Rotation Check exit 1 is a true-positive alert, not a bug (+ inline-Python Discord 403)

Aggregated statistics (2026-06-30 session)

Cross-session comparison — recurring patterns

Graph View

Table of Contents

Backlinks

Group A — Python test regression in `test_sync_n8n_supabase_creds.py`

G1 — GitHub App `app-id` missing (Dependabot context)

Group K — Runs stuck `queued` + job-level `EACCES` on the self-hosted runner (NOT a runner outage)

Group L — `Text file busy` (exit 126) on the shared `~/.local/bin/sops` — parallel-install race

Group M — GH API rate-limit exhaustion in `sync-gh-secrets` (HTTP 403, transient)

Group N — sops-sync-receiver exit 2: `canary decrypt failed … invalid dotenv input line: {`

Group O — Bot auto-commit `git push` rejected by `main` branch protection (GH006)

Group P — `sops --encrypt` input path does not match a `.sops.yaml` creation rule

Group Q — `docker-compose up` container-name conflict on a container the project does not own

Group R — `gh issue create --label <name>` aborts when the label does not exist (exit 1)

Group S — shared-`$HOME` race on `~/.ssh/id_bms1` between concurrent bms-1 sync jobs

Group T — `scp: <file>.js: Is a directory` — Docker bind-mount placeholder on bms-1

Group U — Vercel `403 ENV_ALREADY_EXISTS` on env-var create (duplicate create rejected)

Group V — OpenAI Admin API `404 Not Found` on a non-existent endpoint path

Group W — `startup_failure` (0 jobs, no logs) on `push` to a non-default branch

Group X — `Password Rotation Check` exit 1 is a true-positive alert, not a bug (+ inline-Python Discord 403)