Playbooks Index

This is the flat index for docs/playbooks/ — 323 operational runbooks covering credential rotation, server operations, the Pinbox24 W3/W4 stack, monitoring, the queue dispatcher, and more. Unlike the rest of this Obsidian vault (see [[index]], the vault home, and its 18 numbered perspective hubs), docs/playbooks/ is deliberately flat and prefix-named rather than wikilinked — this file, not [[wikilinks]] between individual playbooks, is what makes it navigable. It is the target of the [[playbooks/README]] link from perspective hubs that point into this directory.

For architecture/ops/design knowledge, start at [[index]] instead and follow a perspective hub’s Key Documents table. For an operational runbook or incident fix, find the relevant category below and open the playbook directly. See [[playbooks/obsidian-vault-navigation]] for the full two-tier navigation method.

Sub-directory Indexes

docs/playbooks/ has two subdirectories, each already covered by its own local index — go there directly for anything n8n- or BrandPilot-specific rather than searching the flat categories below, which only cover the top-level files.

Index	What it covers
README.md	Single entry point for all n8n-related playbooks (26 files, issue #2641 Task 5 — n8n playbook restructure), grouped by purpose so the right one is one lookup away.
index.md	Ops playbooks for the BrandPilot app (5 files) — components (Next.js/Vercel, n8n AI campaign workflow, Supabase, LinkedIn OAuth) mapped to their health signals and per-incident playbooks.

Secrets & Credential Rotation — Policy & Standards

Playbook	What it covers
credential-automation-registry.md	Single source of truth for which services Claude can manage keys/tokens autonomously and how.
credential-rotation-180d.md	Trigger: `credential-rotation.yml` (Monday 06:00 UTC) opens a GitHub issue with label `human-action` when `next_due < today` for any credential with `auto_rotate=false`. This playbook covers the manual rotation pr…
credential-rotation-manual-static.md	Trigger conditions:
credential-rotation-policy.md	Owner: p24-infra · Review: every 90 days or after any rotation incident
encryption-key-rotation.md	Covers: `SUPABASE_JWT_SECRET` · `IAM_ENC_KEY` · `N8N_ENCRYPTION_KEY` · `WEBHOOK_SECRET`
env-key-extraction-safety.md	Origin: incident #3494 (2026-07-09). A Tier-1 MongoDB `rs0` admin credential was exposed in
env-locking.md	Table: `p24_env_locks` (Supabase)
git-remote-inline-token-leak.md	Trigger: A git remote URL on any checkout contains an inline credential —
lap1-worker-credentials.md	The 2026-07-06 secrets-sync audit found that role credential files (`role-sys-admin`,
master-keys-bootstrap.md	Owner: p24-infra · Updated: 2026-06-27
per-service-credential-isolation-plan.md	Issue: #2416
playwright-rotation-template.md	Use this playbook when a credential’s provider requires browser-based login or MFA
prevent-env-local-read.md	Issue: #1500
ps-argv-secret-exposure.md	Status: live · Origin: discovered 2026-08-18 during #5030 live validation, root-cause fix pr #6401, rotation incident pr #6402
rotate-script-security-review.md	Use this checklist when reviewing any new or modified rotation script.
rotation-scripts-audit-2026-07-08.md	Issue: #3416
secret-manager.md	Status: Single source of truth for all p24-infra secret operations.
secret-manager-request.md	Who reads this: any agent in any repo that needs to add, rotate, delete, or retrieve a credential.
secret-manager-rotation-log.md	Table: `dev_r_rotation_log` (Supabase `mwkqmgadqnkkihjdeqsi`)
secret-repo-access-matrix.md	Who can request what. Read by the secret-manager worker before any cross-repo secret operation.
secret-rotation-access-matrix.md	Last updated: 2026-08-04
secrets-design-standard.md	Czytaj ten dokument PRZED zaprojektowaniem każdego rozwiązania które dotyka credentials.
secrets-sync-shared-home-ssh-key-race.md	Applies to: `.github/workflows/secrets-sync.yml`, any workflow with multiple
shell-init-credential-leak.md	Status: STABLE
stale-env-local-after-sops-rotation.md	A local dev server (any repo using the `secrets/<name>.env.sops` → `.env.local`
static-api-key-incident-rotation.md	STOP. Read this playbook before doing anything.

SOPS Operations

Playbook	What it covers
sops-audit-report.md	Generated: 2026-07-05
sops-autonomous-reset.md	Autor: secret-manager agent
sops-checkout-freshness.md	Issue: #4453 ·
sops-credential-liveness.md	Applies to: `p24_sops_credential_valid{key="..."}` (Prometheus, `monitoring/prometheus/rules/credentials.yml`)
sops-edit-operations.md	SOPS `path_regex` in `.sops.yaml` matches against the input file path when encrypting,
sops-environment-bootstrap.md	Owner: p24-infra · Updated: 2026-07-05
sops-parallel-reset-plan.md	Autor: secret-manager agent
sops-powershell-decrypt.md	Running `sops --decrypt secrets/X.env.sops` in PowerShell (or even Bash on this Windows machine
sops-rotation-classifier-patterns.md	Investigation output for #2407 (“Investigate SOPS write/rotation batch formatting issues on
sops-windows-crlf.md	`.gitattributes` enforces `text eol=lf` on `secrets/*.env.sops`. Git normalizes any
sops-windows-patterns.md	Single source of truth for SOPS file operations on Windows.

Per-Service Credential/Token Rotation

Playbook	What it covers
anthropic-admin-api-key.md	Key name: `ANTHROPIC_ADMIN_API_KEY`
anthropic-api-key-rotation.md	Service: Anthropic Claude API (`https://console.anthropic.com`)
atrax-credential-rotation.md	Covers: `ATRAX_AUTH_STRING` · `ATRAX_USERNAME` · `ATRAX_PASSWORD` · `ATRAX_CLIENT_ID` · `ATRAX_CLIENT_SECRET`
aws-stale-key-account-identification.md	A stale or leaked AWS access key (`AKIA...`) needs to be revoked, but it’s unclear which
brand-credential-rotation.md	Trigger: Telegram alert from n8n `Brand — Credential Health Check` (every Monday 08:30 Warsaw) OR `BrandCredentialFailed` / `BrandCredentialRotationOverdue` Prometheus alert fires in Grafana.
clickup-api-key-rotation.md	Service: ClickUp (`https://app.clickup.com`) — project management
clickup-credential-rotation.md	Covers: `clickup_clickup_api_key`
cloudflare-credential-rotation.md	Covers: CF_GLOBAL_API_KEY · CLOUDFLARE_TOKEN_ZINTEGROWANA · CF_EDIT_ALL_ZONES_API_TOKEN
cloudflare-token-rotation-cleanup.md	After any `credential-rotation.yml` run that rotated `CLOUDFLARE_TOKEN_ZINTEGROWANA`
convertapi-token-management.md	Autor: secret-manager agent
discord-bot-decommission.md	Scope: Retire `DISCORD_BOT_TOKEN` and the legacy Discord-poll path
discord-bot-token-rotation.md	Superseded — the canonical, more complete playbook is
discord-credential-rotation.md	Covers: `DISCORD_BOT_TOKEN` · `discord_radieu_password` · `DISCORD_WEBHOOK_URL`
gh-pat-rotation.md	Trigger: `GithubPatExpiringSoon` or `GithubPatExpired` Prometheus alert, or Discord embed from `gh-pat-expiry-check.yml`.
github-pat-rotation.md	Service: GitHub (`https://github.com`)
gitlab-token-playwright-broker.md	Issue: #5298 · Design: `docs/adr/003-sops-secret-access-broker.md`
gitlab-token-rotation.md	\| Prefix \| Type \| Rotation method \|
gmail-tools-oauth-secret-sync.md	Any gmail-tools script (`gmail-query.js`, `send-digest.js`, `gmail-mutate.js`, or any other script
google-oauth-credential-rotation.md	Covers: `GOOGLE_CLIENT_ID/SECRET` · `GOOGLE_OAUTH_WEB_CLIENT_ID/SECRET` · `GOOGLE_ADS_*` · `GCP_SA_KEY_JSON`
groq-credential-rotation.md	Covers: `GROQ_API_KEY`
heygen-api-key-rotation.md	Service: HeyGen (`https://app.heygen.com`) — AI video generation
hstgr-n8n-mcp-token-rotation.md	Service: n8n on bms-4 (`https://n8n.bms-4.infra.zintegrowana.online`)
ionos-api-token-rotation.md	Service: IONOS Developer Portal (`https://developer.hosting.ionos.de`)
linkedin-credential-rotation.md	Covers: `LINKEDIN_ACCESS_TOKEN` · `LINKEDIN_CLIENT_ID` · `LINKEDIN_CLIENT_SECRET`
mailgun-api-key-rotation.md	- `MAILGUN_API_KEY` — used by `infra-src/report-scheduler/report_scheduler.py` to send daily/weekly reports
mailgun-credential-rotation.md	Covers: `MAILGUN_API_KEY` · `MAILGUN_ADMIN_API_KEY` · `SMTP_PASSWORD` · `PINBOX24_SMTP_PASSWORD`
mezmo-key-rotation.md	Related: `docs/mezmo-operations.md`, `docs/secrets-rotation-log.md`, issues #1897 / #1904 / #1911
mezmo-logdna-key-deploy.md	Status: ACTIVE (created 2026-08-04 during #5453 diagnosis)
mongodb-credential-rotation.md	Rotate MongoDB credentials when any of the following occur:
mongodb-exporter-uri-special-chars.md	`mongodb_up{cluster_role="mongod"} 0` in Prometheus. Container logs show:
n8n-bms4-redis-password-rotation.md	Scope: rotating the self-hosted bms-4 Redis password (`REDIS_PASSWORD`, SOPS
n8n-cloud-api-key-rotation.md	Issue: #2802 (part of #2620) · Design: `docs/adr/003-sops-secret-access-broker.md`
nexcon-api-key-rotation.md	Service: Nexcon — SIM card provider for GPS trackers
openai-key-management.md	Owner: p24-infra · Updated: 2026-08-04
pinbox24-credential-rotation.md	Last updated: 2026-07-06
radekkonarski-brand-wasabi-credentials.md	Trigger: Any Wasabi operation in the brand repo fails with `InvalidAccessKeyId`
resend-api-key-rotation.md	Service: Resend (`https://resend.com`) — transactional email
rotate-discord-bot-token.md	RETIRED — 2026-08-07 (#5868)
sentry-token-rotation.md	Service: Sentry (sentry.io)
ssh-key-rotation.md	Trigger: Routine 90–180 day rotation, suspected key compromise, or new workstation setup.
supabase-access-token-rotation.md	Service: Supabase management API (`https://supabase.com`)
supabase-key-rotation.md	Status: Verified 2026-07-05 (issue #2882 — secrets-manager role)
supabase-service-key-rotation.md	Service: Supabase project `mwkqmgadqnkkihjdeqsi`
telegram-bot-token-rotation.md	Service: Telegram BotFather (`https://t.me/BotFather`)
traccar-admin-key-rotation.md	Service: Traccar 6.x GPS tracking server (vps-i1)
v42-prod-credential-rotation.md	Scope: Rotating the credentials that both Pinbox24 backend containers on bms-1 use:
vercel-token-rotation.md	`VERCEL_TOKEN` is a Vercel API token scoped to the `devpinbox24-6490` team. It is used by:
w3-mongodb-credential-rotation.md	Created: 2026-07-02
w3-w4-password-set-secret-manager.md	Read `docs/playbooks/w4-auth-password-model.md` first for the verified W4 auth model.
w3-w4-rotation-orchestration.md	Related: issue #3451, #3256 (orchestrator split), #3449 (W3 secrets table cleanup), #3467
w4-mongodb-credential-rotation.md	Created: 2026-07-02
wasabi-iam-rotator.md	Service: `infra-src/wasabi-iam-rotator/`
wasabi-key-rotation.md	- Alert: `BackupExporterDown` + error `InvalidAccessKeyId` in backup-exporter logs

MongoDB / rs0

Playbook	What it covers
mongodb-admin-password-recovery.md	Applies to: `MONGODB_RS0_ADMIN_PASSWORD` in `secrets/monitoring.env.sops`
mongodb-compound-wildcard-indexes.md	Issue: #1198
mongodb-restore-drill-troubleshooting.md	Reference for diagnosing failures in `scripts/mongodb-restore-drill-cron.sh` and
mongodb-rs0-full-restore.md	Full disaster-recovery runbook for MongoDB replica set `rs0` in the scenario where
mongodb-rs0-heartbeat-drop-block.md	- bms-3 logs: `Heartbeat failed after max retries, NetworkInterfaceExceededTimeLimit, target=bms-2:27017`
mongodb-rs-heartbeat-block.md	A MongoDB `rs0` replica-set member cannot exchange heartbeats with the PRIMARY and/or
mongodb-slow-queries-index.md	Issue: #1198
mongodb-user-isolation.md	Created: 2026-07-05 (feat/#2728)
mongotimeout-false-positive.md	Trigger: GitHub issues titled `[MongoTimeoutError] <host>` (label `bug`) that recur / keep
mysql-root-password-reset.md	`rotate-credentials.py` MYSQL_PASSWORD rotation fails with:

Pinbox24 W3/W4 Stack

Playbook	What it covers
pdf-gen-v42-prod-missing-chromium-libs.md	Status: ✅ DURABLE FIX LANDED 2026-08-09 (#4849, closed) — this is no longer a live-patch-only
pinbox24-bms1-image-backup.md	Status: ACTIVE
pinbox24-bms1-manual-deploy.md	Last updated: 2026-06-29
pinbox24-bms1-redis-wrongpass.md	Trigger: `v42-prod` (w4.pinbox24.com) shows elevated PM2 restarts in Grafana, `WRONGPASS` errors in PM2 logs, backend returns 500/login failures.
pinbox24-container-runtime-hotfix.md	Last updated: 2026-07-02
pinbox24-docker-compose-staging-prod-collision.md	Services: `pinbox24-ms-s3-v2`, `p24-ms-mailgun` on bms-1 (`94.23.26.113`)
pinbox24-do-spaces-investigation-bms1.md	Created: 2026-06-29
pinbox24-mailgun-deploy-script-bug.md	Service: `p24-ms-mailgun` on bms-1 (`94.23.26.113`)
pinbox24-mailgun-duplicate-check-fix.md	Incident date: 2026-07-02
pinbox24-mailgun-s3v2-stabilization.md	Last updated: 2026-07-02
pinbox24-ms-deploy.md	Created: 2026-06-29
pinbox24-nginx-client-max-body-size.md	Status: ACTIVE
pinbox24-nginx-emergency-path-block.md	Status: REUSABLE RUNBOOK — the #5093 block itself was removed 2026-08-02 once the
pinbox24-nginx-roundrobin-s3.md	Created: 2026-07-07
pinbox24-no-logs-bms1.md	Trigger: P1 GitHub issue titled `[P1] Pinbox24 no logs - bms-1` (auto-generated by the
pinbox24-no-logs-false-positive.md	Trigger: P1 GitHub issue titled `[P1] Pinbox24 no logs - bms-1` (auto-generated by the
pinbox24-no-logs-mezmo-dual-agent.md	Symptom: P1 alert `[P1] Pinbox24 no logs - bms-1` fires and re-fires (~every 30 min via
pinbox24-office-membership-grant.md	Applies to: giving an existing `pinbox.profiles` document (W3 `w3_db` or W4 `w4_db`) read access
pinbox24-s3-mongo-uri-log-redaction.md	Applies to: bms-1 (`94.23.26.113`) · Pinbox24 w3 S3 microservice container `s3-v32-prod`
pinbox24-s3v2-master-branch-autodeploy.md	Discovered: 2026-08-06, while executing the async-bugs half of p24-infra#2048. Merging a
pinbox24-s3-wasabi-bms1.md	Applies to: bms-1 (`94.23.26.113`) Ă‚Â· Pinbox24 production (w3 = v3.2, w4 = v4.2)
pinbox24-w3-operations.md	Last updated: 2026-07-06 (session 53)
pinbox24-w3-tasklogs-retention-cleanup.md	Type: Maintenance / Data Cleanup
pinbox24-w3-w4-health-verification.md	Created: 2026-08-03 (issue #5180) — consolidates verification steps that were previously
pinbox24-w3-w4-outage-diagnosis.md	Created: 2026-06-27
pinbox24-w3-w4-security-policy.md	Applies to: bms-1 (`94.23.26.113`) — W3 (v32-prod, s3-v32-prod) and W4 (p24-back-ts, s3-w4)
pinbox24-w4-deleted-records-cleanup.md	Typ: Maintenance / Data Cleanup
pinbox24-w4-holidays-apply-process-activate-trigger-bugs.md	Incident date: 2026-08-06
pinbox24-w4-user-office-workspace-provisioning.md	Applies to: W4 (`v42-prod`, current production, `api.w4.pinbox24.com`) backend `p24-back-ts`
pinbox24-wkhtml-network-isolation.md	Created: 2026-07-07
reso-import-post-incident-verify.md	Purpose: Prove the W3 RESO Excel import path is really working after an
v32-prod-mongoose-authfail-storm.md	Trigger: Alert `MongooseDisconnect` on container `v32-prod` (e.g. “75 in 5m”), usually alongside
v32-prod-prod-v-3-net-persistence.md	Service: `v32-prod` (W3 backend, `api.w3.pinbox24.com`) on bms-1 (`94.23.26.113`)
v32-reso-socket-stale-image-redis-v5-crashloop.md	Service: `v32-prod-reso` (`w3.reso-integration-addrecords.pinbox24.com`, RESO EUROPA — ~86% of W3
v41-prod-ecr-rebuild.md	Source issue: #3733. Related: `docs/plans/w3-w4-cicd-full-deployment.md` (Gap 2),
v42-prod-gus-wsdl-crash-loop.md	Trigger: PM2 Plus dashboard shows `v42-prod_backend` restart counter climbing rapidly (uptime
v42-prod-memory-leak.md	Trigger: `Pinbox24BackendRestartLoop` Prometheus alert (>5 PM2 restarts in 10m), or
w3-gitlab-ci-image-pipeline.md	Created: 2026-07-11
w3-pm2-jlist-exposure-verify-first.md	Status: investigation complete — verify-first no-op (no rotation, no force-recreate)
w3-redis-v32-missing-container.md	Status: STABLE — executed and verified 2026-07-13 09:16 UTC on bms-1
w3-rollback-2026-07-10.md	Stan aktualny po incydencie 2026-07-09 (postmortem: `docs/incidents/2026-07-09-w3-cascading-outage-postmortem.md`)
w3-s3-hostname-alias-drift.md	- Client report: “can’t upload files” / RESO Excel import fails with `excel file is not found`.
w3-v32-prod-clobbered-by-stage-deploy-recovery.md	Status: STABLE — executed and verified 2026-07-29 17:26–17:29 UTC on bms-1 (incident #4565)
w3-w4-redeploy-idempotency.md	Created: 2026-07-18
w4-auth-password-model.md	Read this before touching any W4 user password. Multiple incidents (#4135, #4144, #4150)
w4-gitlab-ci-image-pipeline.md	Created: 2026-07-11
w4-health-check-egress.md	Issue: #3331 (consolidates #2706, #2708, #3016, #3043, #3167, #3250)
w4-rollback-2026-07-09.md	Aktualny działający stan: tag `w4-working-2026-07-09`
w4-s3v2-mailgun-password-verify-first.md	Status: investigation complete — STOPPED, no rotation performed (dead-credential finding, not a
w4-v42-jwt-exposure-verify-first.md	Status: investigation complete — verify-first no-op (no rotation, no force-recreate)

Queue Dispatcher & Worker Infra

Playbook	What it covers
adding-new-worker.md	Scope: Register a new AI worker (Claude Code agent or GitHub Actions runner) so every
cross-repo-worker-isolation.md	Playbook: What happens when a worker spawned for a foreign repo (e.g. `radieu/et-operational-platform`) runs in the wrong codebase and CLAUDE.md context — and how to fix it.
dev-laptop-claude-remote.md	Łączyć się z Claude Code działającym na dev-laptop (Berlin, 24/7) z telefonu.
dispatcher-keyerror-stall.md	Symptom: `dev_r_worker_queue` accumulates rows with `status=queued` and none move to
dispatcher-metadata-json-crash.md	Symptom: `queue-dispatcher-loop.py` crashes every 2 minutes with:
dispatcher-oom-loop.md	A queue worker gets OOM-killed by its cgroup (`MemoryMax` enforced by `systemd-run --scope`)
dispatch-pipeline-key-sync.md	When to use: CF Worker `p24-meta-dispatcher` returns HTTP 500 after a `secrets-sync.yml` run, or `dispatch-to-queue.yml` fails with auth errors after a SOPS credential rotation.
provision-bms-3-dispatch-node.md	Issue: #2147 · Strategy Phase 2 item [11]
provision-windows-dev-docker-worker.md	Status update (2026-08-11): active guidance again. Briefly marked superseded by
queue-analyst-operations.md	Service: `queue-analyst` (`scripts/queue-analyst.py`)
queue-dispatch-cross-project.md	Service: `p24-meta-dispatcher` Cloudflare Worker
queue-dispatcher-bms4-env-crlf.md	`queue-dispatcher.sh` on bms-4 crashes every run with:
queue-dispatcher-dead-column-cleanup.md	Symptom: `scripts/queue-dispatcher-loop.py` (or another script) reads a
queue-dispatcher-priority.md	Last updated: 2026-07-09
queue-dispatcher-ram-gate-exhaustion.md	Diagnose and clear the failure mode where the worker-queue dispatcher stops
queue-failed-weekly-limit.md	Multiple queue rows with `failure_reason=timeout` and `retry_count=max_retries` (permanently failed),
queue-failure-handling.md	Related issues: #1811, #2081, #2091
queue-stale-duplicate-cleanup.md	The queue dispatcher detects a `stale_duplicate` when a queued job’s GitHub issue
spawn-worker-sudo-scope-failure.md	All workers fail immediately after spawn. Log entries:
stale-agent-worktree-cleanup.md	Created: 2026-07-31
stranded-done-row-redispatch.md	A GitHub issue that shows as “still queued” (sits in the `Queue` / `In Progress`
worker-bash-tool-nonfunctional.md	Issue: #4755 (first seen: #4721, 2026-07-31T00:50 UTC on bms-4)
worker-capacity-management.md	vps-i1 (IONOS, 8 GB RAM) hosts the full monitoring stack (Prometheus, Thanos, Grafana, Loki,
worker-gh-auth-standard.md	Every AI-Dev worker’s `gh` CLI must be authenticated as its own AI-Dev GitHub account —
worker-github-invite-acceptance.md	Run this whenever a new GitHub repo is created and AI workers (`AI-Dev-BMS4-1`,
worker-mcp-free-contexts.md	Issue: #2448
worker-push-workflow-files.md	Trigger: a queue worker’s `git push` is rejected with:
worker-queue-constraint-vs-function-mismatch.md	A `dev_r_worker_queue` slot-reservation function fails with:
worker-queue-operations.md	- Worker stays `running` for >2 hours without completing

Claude Agents & Sessions

Playbook	What it covers
agent-browser-windows-setup.md	`agent-browser` (vercel-labs) is a Rust CLI for
agent-watchdog-alert-response.md	Sibling playbook: `hourly-triage-watchdog.md` covers the
background-agent-transcript-forensics.md	Read this when you need to attribute a specific action (a settings.json edit, a SOPS touch, a
claude-account-depletion-tracking.md	The queue dispatcher balances work across multiple Claude subscription accounts
claude-code-hook-security.md	Written after incident #2620 (2026-07-02): monitoring.env.sops full dump in whatsup-android-chat-puller session.
claude-md-management.md	Scope: All environments where Claude Code runs: local dev workstation (Windows), vps-i1 / AI-Dev-IO1 (AlmaLinux), bms-4 / AI-Dev-BMS4-1 (Ubuntu).
claude-oauth-reauth.md	Category: Agent Operations
claude-runner-oauth-rotation.md	- Scheduled credential rotation (every ~90 days)
claude-session-lifecycle.md	Issue: #2426
concurrent-sops-write-worktree-race.md	Created: 2026-08-03
obsidian-vault-access.md	Scope: Local dev workstation (Windows) only. Skills are project-scoped and travel with the repo, but the live-app capability (`obsidian-cli`) is local-machine-only.
obsidian-vault-navigation.md	Status: STABLE
p24-network-admin.md	Role: p24-network-admin
role-delegation-architecture.md	Read this first when starting any session in the p24-infra ecosystem — before the task-specific
roles-and-skills.md	Applies to: all projects in the p24-infra ecosystem (p24-infra, brandpilot, radekkonarski-personal-brand, et-operational-platform)
session-notes-workflow.md	Issue: #2630
shared-checkout-worktree-isolation.md	Created: 2026-07-08
sync-claude-global-skills.md	- A new project repo needs the portable global Claude Code skills (`/sr`, `/commit`,
worktree-remove-fallthrough-guard.md	Created: 2026-08-09

GitHub / GitLab / CI

Playbook	What it covers
ai-pr-auto-review.md	A `review-pr` job reaches the queue by two complementary paths:
fix-claude-commands-missing.md	- `/sr`, `/commit`, or `/create-pr` not available in Claude Code VS Code extension for a repo
fix-claude-settings-missing.md	- Repo has `.claude/` directory but no `settings.json` (only `settings.local.json` or nothing)
fix-default-branch.md	- `repo-consistency-audit.md` reports `default branch is 'master'`
fix-gitattributes-lf.md	- `repo-consistency-audit.md` reports `.gitattributes MISSING` or `no eol=lf rule`
fix-github-labels.md	- `repo-consistency-audit.md` reports missing labels for a repo
fix-github-milestones.md	- `repo-consistency-audit.md` reports missing standard milestones
gh-actions-failed-jobs-taxonomy.md	Living document. Update after each “failed jobs” review session.
gh-app-token-deploy.md	Status: ACTIVE (created 2026-08-11, #6103 — follow-up from #6096)
gh-runner-assignment-policy.md	vps-i1 (`ionos` label) runs only light, infra-specific jobs.
gh-runner-metrics-approach.md	Self-hosted runners on bms-4 expose only process-level metrics (alive/RAM) via
gh-runner-queue-registration.md	Issue: #1923
gh-runner-restart-offline.md	Trigger: health-check `runner-et` or `runner-kdp` step fails (systemctl is-active returns non-active).
gh-secret-set-powershell-pipe-corruption.md	Symptom: a GH Actions secret set from Windows PowerShell via `$env:VAR \| gh secret set NAME`
gh-secret-sops-drift.md	Symptom class: a credential works for server-side scripts but fails inside GitHub
git-checkout-ownership-repair.md	Status: ACTIVE · first written for [#4462] (bms-4). Applies to any server where a
git-identity-and-roles.md	Canonical identity × role × action matrix — who commits / opens the PR / reviews / approves / merges / deploys, per role and per repo. Links out to the mechanics docs rather than restating them.
github-actions-yaml-block-scalar.md	A GitHub Actions workflow stops firing scheduled runs, `gh workflow run` fails with HTTP 422
github-app-token-integration.md	GitHub App: n8n.io pull request flow
github-unavailability.md	Trigger: GitHub is unreachable, degraded, or access to `radieu/p24-infra` /
gitlab-commit-api-windows-powershell.md	Applies to: any p24-infra session on the Windows dev machine pushing file content or CI
gitlab-runner-bms1-reregister.md	Issue: #2078, #4829
gitlab-runner-root-traverse-acl.md	Status: ACTIVE (verified live on bms-1 `ns367522`, 2026-08-09)
hotfix-cherry-pick-isolated-feature.md	Symptom: A feature or fix is fully merged and tested on `dev` (and usually `staging`), the user
main-dev-backmerge.md	DEPRECATED (#2551, 2026-07-31). This playbook assumes the retired `feat/* → dev →
new-repo-setup.md	Use when creating any new GitHub repository under `radieu/` that becomes part of the
rc-smoke-tests-credential-and-config-debugging.md	Trigger: `rc-smoke-tests.yml` fails, and the failure looks like a broken credential
repo-consistency-audit.md	- After onboarding a new repo (verify `new-repo-setup.md` checklist was fully completed)
vercel-deploy-blocked-git-author-access.md	Trigger: A `gh pr merge` into a project’s `main` (or any Vercel-git-integration-tracked

Servers

Playbook	What it covers
acer-f5573g-ubuntu-2404-install.md	Runbook for installing Ubuntu 24.04 LTS on Radek’s private dev laptop
bms1-blackbox-config-deploy.md	Status: ACTIVE (promoted from DRAFT 2026-07-12)
bms1-container-audit.md	Status: ACTIVE
bms1-dr-plan.md	Disaster-recovery runbook for bms-1 (`p4-ovh-bms-1-ns367522` · `94.23.26.113` ·
bms1-nginx-proxy-missing-cert-symlinks.md	Status: DRAFT — written live during first occurrence, 2026-07-12.
bms1-pm2-log-credential-exposure-and-rotation-gap.md	Status: remediated (server-side) — 2026-08-04. Rotation of the one live exposed credential
bms1-post-restart-recovery.md	Created: 2026-07-06
bms1-ubuntu-2204-upgrade.md	Runbook for the in-place `do-release-upgrade` of bms-1
bms2-mongodb-exporter-decommission.md	Status: ACTIVE — remediation runbook for #5859
bms3-audit-2026-06-23.md	Reference playbook for the full audit of bms-3 (`p4-ovh-bms-3-ns3129867` ·
bms3-ssh-key-auth-only.md	Closes: #2369 · Related: #2367, #2377 / cross-server-ssh-key-standard.md
bms4-claude-runner-ssh-client-config.md	Origin: #6032, discovered 2026-08-09 during a supervised `W3_APP_MONGODB_PASSWORD` rotation —
bms-server-root-ssh-lockout-recovery.md	Trigger: `ssh root@<bms-ip>` fails with `Permission denied (publickey,password)` and a
cross-server-ssh-key-standard.md	Issue: #2377 ·
server-heartbeat.md	A bash script (`scripts/server-heartbeat.sh`) that runs every 2 minutes on each dispatch
server-ssh-connectivity.md	Status (2026-07-08): Headscale VPN fully decommissioned — container, config/data, and Caddy vhost removed from vps-i1; Tailscale disconnected on vps-i1 and windows-dev. Port 22 is open from internet on all servers…
ssh-agent-pattern.md	All Claude agent SSH commands MUST capture output and only surface it on failure.
ssh-pub-sidecar-ci-runner.md	`secrets-sync.yml` `sync-vps-i1` (or any other SSH step) fails with:
vps-h1-cpu-throttle.md	Trigger: Alert `VpsH1ThrottleRisk` — CPU >80% sustained 45+ min
vps-h1-stale-docker-iptables.md	After `docker compose down && docker compose up -d` on vps-h1, the new Docker network
vps-h1-traefik-iptables-stale-rules.md	After a Docker OOM kill or forced host reboot on vps-h1, Traefik is unreachable on ports 80/443 despite all containers showing as `Up`. Symptom: `curl -I https://waha2.vps-h1.infra.zintegrowana.online` times out. Aler…
vps-i1-crash-loop-recovery.md	- A container enters a rapid restart loop (OOM kill cycle, startup crash)
vps-i1-oom-outage.md	A process running under `claude-runner` (UID 1000) on vps-i1 consumes all available RAM,
vps-i1-outage.md	vps-i1 (217.154.82.162, IONOS) stops responding to SSH, ICMP, and HTTPS from both local workstation and external servers (bms-4).
waha-healthcheck-401-gh-secret-drift.md	The `Infrastructure Health Check` workflow reports:
waha-ingestion-failure.md	- Messages pile up in `new` status and never reach `na_zgloszeniu_email`
waha-session-stopped-after-reboot.md	vps-h1 reboots (planned or unplanned) and the WhatsApp `default` session comes up in STOPPED state.
waha-traefik-down-ports-80-443.md	Trigger: `WAHAProbeDown` (critical) fires — `probe_success{job="blackbox_waha"} == 0`,

Monitoring & Alerting

Playbook	What it covers
add-new-monitoring-exporter.md	Purpose: step-by-step pattern for adding the Nth custom exporter to the p24-infra monitoring
grafana-claude-sub-monitoring.md	Status: ACTIVE
hourly-triage-outage.md	Migration note (2026-06-25): the DevOps triage moved hourly → nightly (now runs
hourly-triage-watchdog.md	Migration note (2026-06-25): the DevOps triage moved hourly → nightly — it now runs
incident-management.md	Scope: Pinbox24 W3/W4, MongoDB rs0, n8n, WAHA, monitoring, AI workers
mezmo-exporter-api-errors-triage.md	Alert: `MezmoExporterApiErrors` (`monitoring/prometheus/rules/pinbox24.yml`)
mezmo-terraform-management.md	Status: scaffolding shipped (#4185); native bms-1 no-logs alert codified as DISABLED at source (#4197). `terraform apply` is a sys-admin follow-up.
monitoring-stack-operations.md	Status: ACTIVE
monitoring-standby-vps-h1.md	When `grafana.vps-i1.infra.zintegrowana.online` is unreachable and the CF Worker watchdog
nc-alert-dispatch-outage.md	Failure class: The `nc-alert` pipeline stops processing alert issues. Alert issues accumulate
netdata-health-alarm-tuning-bms1.md	Status: ACTIVE (first written 2026-07-17 for #4242)
oom-auto-remediation.md	The Linux OOM killer fires when available memory drops to zero.
resource-incident-monitoring.md	Issue: #1191 · Prerequisites: #1467 · Status: code merged, activation gated on #1467

Networking

Playbook	What it covers
caddy-basicauth-bcrypt-docker-compose.md	Caddy crash-loops with:
cloudflare-pinbox24-onboarding.md	Issue: #986
cloudflare-waf-hidden-managed-rule.md	Status: Workaround shipped (2026-08-07, p24-infra#5263) — `openai-monitor.py` and
cloudflare-waf-python-urllib-403.md	Symptom: a Python script calling a p24 Cloudflare Worker (`p24-meta-dispatcher`,
email-api-waf-ip-allowlist-403.md	Superseded (2026-08-07, #5263): the “restore the IP in the WAF rule” fix below assumes a
fleet-backup-ssh-cf-tunnel.md	Status: ACTIVE
nginx-proxy-upgrade-jwilder-to-nginxproxy.md	`jwilder/nginx-proxy` (last released ~2021) does not support the
socat-supabase-watchdog.md	Service: `socat-supabase.service` — IPv4→IPv6 proxy on bms-4 `:15432`
socat-supabase-zombie-crash-loop.md	`socat-supabase.service` on bms-4 provides a TCP IPv4→IPv6 proxy for Supabase PostgreSQL

Supabase

Playbook	What it covers
pg-stats-exporter-pooler-tenant-identifier.md	Alert: `SupabasePgStatsScrapeStale` (Prometheus / pg-stats-exporter on vps-i1)
supabase-backup-stale.md	Alert: `SupabaseBackupStale`
supabase-dashboard-playwright-broker.md	Issue: #5911 · Design: `docs/adr/003-sops-secret-access-broker.md` ·
supabase-management-api-sql-windows.md	Use the Supabase Management API for SQL execution when:
supabase-migrations.md	Project (shared DB): `mwkqmgadqnkkihjdeqsi` (`db.mwkqmgadqnkkihjdeqsi.supabase.co`)
supabase-migrations-ci.md	Automated migration apply on push to `main` or `rc/` when `supabase/migrations/*` changes.
supabase-migration-unblock-stuck-ci.md	Symptom: A migration file is merged (e.g. to `dev`/`staging`), the app code that depends on the new
supabase-per-repo-migration-ledger.md	Status: standard as of #5653 (design brainstorm + decisions in #5645 → `et-operational-platform#1500`).
supabase-postgrest-jwt-auth-fix.md	Trigger: CF Worker (or any backend) can’t authenticate to Supabase PostgREST via REST API.
supabase-secrets-management-gaps.md	Date: 2026-06-28
supabase-security-advisor-remediation.md	Owner: dev / sys-security · Related: #4645 #4647 #4663 #4664 #4665 #4666

Cloud Storage (Wasabi / S3)

Playbook	What it covers
s3-v2-bms1-file-service.md	Service: `s3-v2-v42-prod` Docker container on bms-1 (`94.23.26.113`)
s3-v2-container-empty-env-vars.md	Any of:
wasabi-bucket-growth-spike.md	`WasabiBucketGrowthSpike` alert fires when `delta(wasabi_bucket_bytes[24h]) > 15 GB` persists for 2 hours.
wasabi-replication-repair.md	Script: `scripts/wasabi-replication-repair.py`

n8n

Playbook	What it covers
n8n-credential-auto-refresh-github-app.md	Status: ACTIVE (created 2026-08-12, #6220 follow-up)
n8n-workflow-export.md	Service: `scripts/n8n-workflow-export.sh`, run nightly (02:00 UTC) by
n8n-workflow-modification.md	Applies to: n8n Cloud (`p24.app.n8n.cloud`) and bms-4 (`n8n.bms-4.infra.zintegrowana.online`)

External APIs / SaaS

Playbook	What it covers
anthropic-batch-api.md	The Anthropic Batch API (`POST /v1/messages/batches`) processes requests asynchronously at 50% of synchronous API prices. It is designed for non-real-time workloads where latency is not critical.
atrax-daily-workflows-paired-item-and-stale-key.md	Workflows (bms-4 n8n):
atrax-data-freshness-false-positive.md	Trigger: A GitHub issue titled `Atrax data stale: ...` (label `atrax-stale`) is opened by the
atrax-fleet-sync-409-replate.md	Workflow: `atrax, kravag-scheduled-fleet-updates` (id `CCx9UMdphmGficDX`, bms-4 n8n, 3-min schedule)
atrax-request-signing.md	Discovered 2026-08-12, resolves #5921. Applies to any Atrax (`tronik.atrax4.com`) API endpoint
atrax-vehicle-tags-api.md	Discovered 2026-06-30 via browser DevTools inspection.
discord-provisioning.md	Service: Discord Developer Portal (`https://discord.com/developers/applications`)
gcp-programmatic-management.md	Goal: let a p24-infra admin agent manage Google Cloud resources programmatically —
ionos-api-server-management.md	\| Product \| Portal \| API base \| Credentials \| What it can do \|
mailgun-deploy-missing-mongodb-url.md	Investigated: 2026-08-01 (issue #5027)
mailgun-mongodb-stale-credential-hang.md	Incident date: 2026-07-10
mailgun-pipeline-exporter-deploy.md	Status: STABLE (executed successfully 2026-07-11, issue #3688)
mailgun-receiving.md	Mailgun receiving is free — no quota, no cost. Works by configuring MX records for a domain
ovh-api-operations.md	Scope: Autonomous OVH API operations available to the p24-infra Claude role.

Compliance, Process & Standards

Playbook	What it covers
admin-actions-investigation.md	When to use: You need to determine who ran a command, changed a file, installed a package, or modified credentials on any server. Also use for post-incident forensics, compliance audits, or suspicious activity tri…
brand-definition.md	Trigger: A creative role (marketing-director, content-creator, designer, video-producer)
changelog-priorities-pattern.md	Goal: keep `CHANGELOG.md` and `docs/priorities.md` current across every p24 ecosystem project
compliance-audit-policy.md	Owner: Radek Konarski (p24-infra admin)
cross-project-infra-requests.md	Audience: Claude agents in non-p24-infra repos (et-operational-platform, Art-Agency,
dedup-infra-task-operational-vs-tooling.md	Applies to: the pre-dispatch dedup guard — `infra-src/meta-dispatcher/src/dedup.ts`,
dev-r-services-insert.md	PENDING TODO — lap1 worker registration (2026-06-30)
infra-ops-cold-archive.md	Scheduled flush of aged `public.infra_operations` rows to Wasabi cold storage.
infra-task-executor.md	Audience: p24-infra admin Claude (and human admins) processing incoming `infra-task-request`
issue-deferral-future-milestone.md	Deferring a GitHub issue until a specific future date using the `defer-until:` body line,
playbook-lifecycle.md	Rule: If you’re about to do something significant and there’s no playbook, write the playbook FIRST in a draft form, then execute.
priorities-status-sync.md	Service: `nightly-devops-triage` Phase 3.5 (`.claude/commands/nightly-devops-triage.md`)
testing-standards.md	Source of truth for the `dev-tester` role and all test-related decisions in the p24-infra ecosystem.
token-cost-optimization.md	Two cost surfaces are covered here:

Other

Playbook	What it covers
gps-fuel-snapshots-phase1-import.md	Operational runbook for the append-only `p24_gps_fuel_snapshots` branch added to the
report-scheduler-rerun.md	Status: ACTIVE

Cross-references

index — vault home, 18 perspectives
README — Secrets perspective hub (links here)
README — Server Maintenance perspective hub (links here)
README — Roles perspective hub (links here)
README — Project Skills perspective hub (links here)
README — Operations perspective hub
README — Monitoring perspective hub
README — Incident Response perspective hub
README — Networking / DNS perspective hub
README — Automations perspective hub
README — External Services perspective hub
README — Server Security perspective hub
obsidian-vault-navigation — how to navigate this vault (two-tier method)
obsidian-vault-access — Obsidian CLI access playbook for Claude agents
playbook-lifecycle — how to write/modify/retire a playbook

p24-infra Docs

Explorer

README