p24-infra Docs

gh-secret-sops-drift

4 min read

Playbook — GitHub Actions Secret drifted from SOPS

Symptom class: a credential works for server-side scripts but fails inside GitHub Actions. Canonical instance: P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL returning {"message": "Unknown Webhook", "code": 10015} only when a GH Actions workflow posts to it (issue #2150).

Why it happens

Secrets have three distribution targets, kept in sync by secrets-sync.yml:

TargetConsumed bySynced by
secrets/*.env.sopssource of truthcommitted to git
on-server /opt/p24-infra/<host>/.envshell scripts, cron, containerssecrets-sync.yml (SOPS → SSH)
GitHub Actions Secret store${{ secrets.<KEY> }} in workflowssync-gh-secrets job (proposed — #2158)

The third target has no automated sync yet. When a value is rotated into SOPS (and pushed to servers), the GH Actions Secret was a separately, manually-maintained copy — so it silently went stale. A workflow reading the stale secret kept using the old (now-deleted) value while scripts reading on-server .env were already healthy. That asymmetry — “fine on the server, dead in CI” — is the fingerprint of this drift.

Detect

  1. Confirm which copy is dead. Never print the value — test liveness silently:

    # SOPS copy (source of truth)
WH=$(sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops \
  | grep "^P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL=" | cut -d= -f2- | tr -d '"')
curl -s -o /tmp/b.json -w "%{http_code}\n" "$WH"   # 200 = alive, 404 + 10015 = dead
grep -q 10015 /tmp/b.json && echo DEAD || echo alive
rm -f /tmp/b.json; unset WH

  2. Compare the GH Secret’s updatedAt against the SOPS rotation date (names/timestamps only, never values):

    gh secret list --repo radieu/p24-infra | grep P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL

    A GH Secret updatedAt older than the SOPS value’s last rotation = drift.

Fix

Immediate (resolves the live incident)

Re-set the GH Secret from the known-good SOPS value — piped via stdin so the value never appears on the command line or in logs:

VAL=$(sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops \
  | grep "^P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL=" | cut -d= -f2- | tr -d '"' \
  | sed 's/^\xEF\xBB\xBF//' | tr -d '\r\n')
printf '%s' "$VAL" | gh secret set P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL --repo radieu/p24-infra
unset VAL

Requires a token with secrets:write scope (classic repo PAT or fine-grained Secrets: write). In CI this is the GH_PAT secret.

Durable (prevents recurrence) — proposed in #2158

The recommended fix is a sync-gh-secrets job in .github/workflows/secrets-sync.yml that pushes a defined allowlist (CI_GH_SECRET_KEYS) from monitoring.env.sops into the GH Actions Secret store — making SOPS the single source of truth for CI secrets too. It runs on any push that changes secrets/*.env.sops, or on demand:

gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=gh-secrets

This requires a commit under .github/workflows/, which needs a token with the workflow scope — the autonomous worker PAT does not have it, so the job is tracked as a human-action in #2158 (full YAML patch included there).

When you add a new SOPS key that a GitHub Actions workflow must read, append it to CI_GH_SECRET_KEYS in that job so it stays in sync. The allowlist is explicit by design — the sync can never clobber a GH Secret that legitimately differs from SOPS (e.g. a CI-only deploy token).

Verify

# 1. GH Secret updatedAt advanced
gh secret list --repo radieu/p24-infra | grep P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL
# 2. Re-run an affected workflow's failure path and confirm the Discord embed lands:
gh workflow run prometheus-alerts-ai-triage.yml --repo radieu/p24-infra
  • standards/common/error-notification.md — the Discord + GH-issue error standard
  • docs/playbooks/static-api-key-incident-rotation.md — full credential rotation/distribution
  • scripts/rotate-credentials.py — gh_secret_set() / sops_update_key() helpers

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="gh-secret-sops",
    result="success",  # "success" | "failed" | "skipped"
    detail="GH Secret — SOPS drift detected and resolved — secret resynced from SOPS to GH",
    env="local",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'gh-secret-sops', 'success', 'GH Secret — SOPS drift detected and resolved — secret resynced from SOPS to GH', 'local')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks