Credential Rotation Policy
Owner: p24-infra · Review: every 90 days or after any rotation incident
Audit trail: docs/secrets-rotation-log.md + GH issue lifecycle (created → closed)
1. Triggers and response SLA
| Trigger | Condition | SLA | Severity |
|---|---|---|---|
| Exposure | Secret value visible in chat, logs, git, or any unsecured channel | 1 hour | critical |
| Scheduled | next_due date reached in dev_r_services | 24 hours | per key criticality |
| Preventive | Before sharing access, before offboarding, before key handoff | same day | per key criticality |
2. Rotation categories
| Type | rotation_type | Who runs it | Trigger |
|---|---|---|---|
| Automated | auto | rotate-credentials.py via GH Actions | Monday 06:00 UTC or manual workflow dispatch |
| Playwright | playwright | Claude Code + Playwright MCP; human pastes value via $env:NEW_VALUE | Manual — create GH issue with label human-action |
| Manual | manual | Human at provider dashboard; Claude distributes to SOPS + GH Secrets + VPS | Manual — create GH issue with label human-action |
| Not applicable | NULL | Non-rotatable credential or non-credential service element (container, server) | N/A |
2a. Decision tree
Secret rotation triggered
│
├─ rotation_type = 'auto' in dev_r_services?
│ ├── YES → run credential-rotation.yml (workflow_dispatch, force_all=true for exposure)
│ │ Script rotates key + updates GH Secret + restarts service + closes issue
│ └── NO → rotation_type = 'playwright'?
│ ├── YES → follow docs/playbooks/playwright-rotation-template.md
│ └── NO → rotation_type = 'manual': create GH issue (see §4 format) + rotate manually following §3
│
└─ After rotation (all paths)
├── Update docs/secrets-rotation-log.md (desktop sessions) OR dev_r_rotation_log in Supabase (VPS workers)
├── Canary decrypt SOPS if SOPS was touched
├── Update dev_r_services.last_rotated + next_due
└── Verify old key is revoked at provider
3. Which playbook to follow
| Scenario | Playbook |
|---|---|
| Exposure — triage scope | docs/playbooks/static-api-key-incident-rotation.md |
| Scheduled — auto keys | scripts/rotate-credentials.py (GH Actions, Monday 06:00 UTC) |
| Scheduled — manual keys | docs/playbooks/credential-rotation-180d.md |
| Manual per-service | docs/playbooks/mezmo-key-rotation.md etc. |
| SOPS file editing (any key) | docs/playbooks/sops-edit-operations.md |
| Windows CRLF / BOM issue | docs/playbooks/sops-windows-crlf.md |
| Master key compromise | docs/playbooks/master-keys-bootstrap.md |
4. Human-action issue format (MANDATORY)
Every human-action rotation must produce a GH issue using this template.
Issue is the audit record — do not skip it, even for urgent exposure rotations.
Created automatically by: scripts/rotate-credentials.py for scheduled rotations.
Created manually by: Claude or the operator for exposure/preventive rotations.
Labels
| Condition | Labels |
|---|---|
| Exposure or criticality = critical/high | human-action + critical |
| Scheduled + criticality = medium | human-action + patch |
| Scheduled + criticality = low | human-action |
Title format
sec: rotate KEY_NAME — [exposure|scheduled|preventive] YYYY-MM-DD
Body template
## Credential Rotation Required
**Rotation requested:** YYYY-MM-DDTHH:MM:SSZ (UTC)
**Trigger:** exposure | scheduled | preventive
**Criticality:** critical | high | medium | low
**Overdue by:** N days (0 for exposure/preventive)
| Field | Value |
|---|---|
| Credential | `KEY_NAME` |
| SOPS file | `secrets/xxx.env.sops` (or `none — GH Secret only`) |
| Rotation frequency | 180d |
| Last rotated | YYYY-MM-DD (or `never`) |
| Provider dashboard | URL |
## Rotation checklist
- [ ] Generate new credential at provider dashboard
- [ ] Update SOPS file (`docs/playbooks/sops-edit-operations.md`)
- [ ] Canary decrypt:
`sops --decrypt --input-type dotenv --output-type dotenv secrets/xxx.env.sops | Out-Null`
- [ ] Update GH Secret:
`gh secret set KEY_NAME --body "<new>" --repo radieu/p24-infra`
- [ ] Restart affected services (list which ones)
- [ ] Revoke old credential at provider
- [ ] Append row to `docs/secrets-rotation-log.md`
- [ ] Verify `dev_r_services.last_rotated` + `next_due` updated (run SQL from per-key playbook §N)
- [ ] Close this issue
See `docs/playbooks/credential-rotation-180d.md §<KEY_NAME>` for per-key procedure.5. Audit trail — secrets-rotation-log.md standard
Every rotation must append a row to docs/secrets-rotation-log.md before the issue is closed.
| YYYY-MM-DD HH:MM UTC | #ISSUE | KEY_NAME | reason: exposure/scheduled/preventive | rotator (radieu/claude/AI-Dev-IO1) | confirmed in: SOPS / GH Secret / vps-i1 live |HH:MM is mandatory for exposure incidents (1-hour SLA). Use 00:00 for scheduled/bootstrap. Use — for GH Issue if no issue was created.
Exception — VPS worker rotations: Workers (AI-Dev-IO1, AI-Dev-BMS4-1) write to dev_r_rotation_log in Supabase instead. Desktop Claude sessions should summarise these in the markdown file when confirmed complete.
Do not close the GH issue until the log row exists.
5a. Machine audit trail — infra_operations (MANDATORY)
In addition to the human-readable secrets-rotation-log.md row (§5), every rotation MUST emit a
machine-readable log_op() entry to the infra_operations table. This feeds the
infra-ops-audit-v1 Grafana dashboard and the 8-week rolling audit archive. Reference the
key NAME only — never the value. Full reference: docs/infra-operations-audit-operations.md.
# VPS worker (bash) — call AFTER the new key is verified working and all consumers updated:
source /opt/p24-infra/scripts/lib/log_op.sh
log_op "claude" "credential_rotation" "KEY_NAME" "success" \
"Scheduled 180d rotation; SOPS + GH Secret + live envs updated; old key revoked" "vps-i1" 2730
# result: "failed" if rotation aborted, "skipped" if intentionally deferred.# Python scripts / audit-engine:
import sys; sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
log_op(actor="claude", op_type="credential_rotation", resource="KEY_NAME",
result="success", detail="Exposure rotation #NNNN; all consumers updated", env="vps-i1", gh_issue=2730)continue-on-error semantics: log_op is fail-open — a logging failure never blocks or reverts
the rotation. Do NOT close the GH issue on a logging failure; retry the log entry instead.
6. Post-rotation verification
After rotating any credential:
-
SOPS canary (if SOPS was touched):
sops --decrypt --input-type dotenv --output-type dotenv secrets\<file>.env.sops | Out-Null if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" } -
Service smoke test — confirm the service using the key is still healthy:
- Prometheus target still
UP - n8n workflow still runs
scripts/mezmo-manage.py usagereturns data
- Prometheus target still
-
Old key revoked at provider — confirm in provider audit log or by testing the old key (expect 401/403).
-
GH issue closed with a closing comment:
Rotated on YYYY-MM-DD HH:MM UTC by <who>. Old key revoked. Services confirmed healthy. Log updated.
7. Auto-rotation coverage (current)
Keys with auto_rotate=true in dev_r_services — rotated by credential-rotation.yml:
| Key | Frequency | Script |
|---|---|---|
GRAFANA_ADMIN_PASSWORD | 180d | bcrypt hash + Caddyfile + GH Secret |
SUPABASE_GRAFANA_PASSWORD | 180d | ALTER ROLE via Management API |
OPENAI_MONITORING_TOKEN | 90d | OpenAI admin API create + delete stale |
CLOUDFLARE_TOKEN_ZINTEGROWANA | 180d | CF API roll via Global Key |
EMAIL_SENDER_API_KEY | 180d | CF Worker secret + GH Secret |
MYSQL_PASSWORD | 180d | ALTER USER + traccar restart |
WASABI_KEYS_P24_INFRA | 180d | IAM create + delete + all consumers restarted |
BMS4_N8N_API_KEY | 90d | n8n REST API + GH Secret |
RADEK_BRAND_RESEND_API_KEY | 180d | Resend API create + delete stale + GH Secret |
8. Human-action only — no API available (reference)
These keys cannot be auto-rotated. auto_rotate=false in dev_r_services.
A GH issue is created automatically when next_due is reached.
Note (2026-06-27): Several items previously classified as human-action-only were successfully automated via Playwright MCP browser automation in a Claude Code session. See §8a for the updated classification.
| Key | Reason no API | Issue auto-created? |
|---|---|---|
SUPABASE_SERVICE_ROLE_KEY | sb_secret value masked in Management API response | YES (scheduled) |
GH_TOKEN / PATs | GitHub PAT creation requires browser login | YES |
CLOUDFLARE_GLOBAL_API_KEY | CF Global Key has no rotation API | YES |
DISCORD_BOT_TOKEN | Discord Developer Portal is UI-only | YES |
grog_api_key | Groq Console has no key management API | YES |
p24_infra_mezmo_service_key | Mezmo REST API: POST /v1/config/keys returns 400 | YES |
| Server root passwords | passwd requires interactive TTY | YES |
| MongoDB passwords | mongosh session required | YES |
| LinkedIn OAuth token | Requires browser OAuth flow | YES (monthly alert) |
| Discord webhooks | Discord API requires MANAGE_WEBHOOKS bot scope | YES |
| OVH consumer keys | OVH OAuth requires browser authorization | YES |
WASABI_ADMIN_ACCESS_KEY | Wasabi Console UI only — no IAM self-rotation API | YES |
CF_GLOBAL_API_KEY | CF dashboard only — Global Key has no rotation endpoint | YES |
AGE_KEY_GHA | Requires age-keygen + sops updatekeys + manual GH Secret update | NO (manual procedure in master-keys-bootstrap.md) |
8a. Playwright-automatable (previously human-action)
These were listed as human-action but were successfully automated on 2026-06-27 using Playwright MCP browser automation from a Claude Code session. They do not need a human present but require the Playwright MCP server to be running in the session.
| Key | How automated | Procedure |
|---|---|---|
Telegram bot tokens (TELEGRAM_BOT_TOKEN, TELEGRAM_P24_CLAUDE_BOT_API_KEY) | Navigate web.telegram.org/k/#@BotFather → /mybots → select bot → API Token → Revoke | Capture new token from BotFather message; store silently to SOPS + GH Secret |
OPENAI_ADMIN_KEY | Navigate platform.openai.com/api-keys → delete old row → Create new secret key | Capture key from “Save your key” dialog; store silently to SOPS + GH Secret |
SSH root key (VPS_ROOT_SSH_KEY) | ssh-keygen locally → ssh-copy-id style append → verify → swap id_ed25519 → remove old | All via PowerShell + SSH; no browser needed. Update GH Secret via gh secret set |
SSH claude-admin key (VPS_SSH_PRIVATE_KEY) | Same as root key pattern; vps-i1 claude-admin only | Update GH Secret VPS_SSH_PRIVATE_KEY via gh secret set |
Claude Max OAuth (claude-runner on vps-i1 + bms-4) | auth_v4.py sentinel pattern + Playwright Google OAuth flow | See docs/playbooks/claude-runner-oauth-rotation.md |
Prerequisite for Playwright automation: Google account radieu@gmail.com must be logged in
to the Playwright browser profile. If Google session has expired, a Google login (email + password
- 2FA push to phone) is required first — typically takes ~2 min.
9. Delivering new values to SOPS (replaces .env.local workflow)
.env.local was removed 2026-06-27. All credentials flow through SOPS only.
Option A — Single key (preferred)
- Human pastes new value in terminal ONLY:
$env:NEW_VALUE = "paste-here" - Claude runs decrypt → regex replace → re-encrypt → canary → delete temp (see
playwright-rotation-template.mdfor the full PowerShell pattern) - Claude uses
$env:NEW_VALUEfor GH Secret update, then clears it
Option B — Multi-key edit
- Claude decrypts SOPS file to
secrets/*-edit.env.sops(temp file insidesecrets/to match.sops.yamlpath_regex) - Human opens the temp file in VSCode and edits
- Claude re-encrypts + canary + deletes temp
Note on sops —set
sops --set uses JSON path notation designed for YAML/JSON files. It does NOT work for dotenv-format SOPS files. Do not use it.
10. Exposure incident checklist (quick reference)
When .env.local, a chat session, or a log file contains a secret value:
[ ] 1. Identify all exposed keys (search transcript for sk-, ghp_, sb_secret_, cfut_, etc.)
[ ] 2. Create one GH issue per key (label: human-action + critical, trigger=exposure)
OR run: python scripts/create-exposure-issues.py (if implemented)
[ ] 3. Trigger auto-rotation immediately:
gh workflow run credential-rotation.yml --field force_all=true --field dry_run=false
[ ] 4. Rotate all human-action keys within 1 hour (follow §4 checklist per key)
[ ] 5. Revoke old values at provider for each key
[ ] 6. Append rows to docs/secrets-rotation-log.md (reason: exposure, date: today)
[ ] 7. Close all GH issues with closing comment
[ ] 8. Investigate how the exposure occurred → fix root cause → commit playbook update