Compliance Audit Policy — p24-infra

Owner: Radek Konarski (p24-infra admin)
Effective date: 2026-06-28
Review frequency: every 6 months or after any major incident
Audit registry: public.dev_r_compliance_audits (Supabase) — every audit MUST produce a row
Conducted by: Claude (p24-infra session) unless otherwise stated
Ordered by: radieu (owner) or triggered automatically by schedule

Purpose

This policy defines which areas of p24-infra must be audited, how often, and at what severity.
It is a policy document — it does not describe how to conduct each audit (that lives in the referenced playbook).
Every completed audit must be recorded in dev_r_compliance_audits with result, summary, findings, and doc_url.

Risk Classification

Level	Definition	Example impact
CRITICAL	Failure causes immediate data loss, security breach, or legal violation	Backup unrestorable, credentials exposed, GDPR breach
HIGH	Failure causes service outage or serious security gap discoverable within days	Key isolation missing, SSH access not audited, CVE unpatched
MEDIUM	Failure causes degraded service or compliance gap, discoverable within weeks	Stale playbook, MCP reference in docs, cost overrun
LOW	Quality issue, no operational or security impact	Naming inconsistency, missing doc link

Domain 1 — Credentials & Secrets

Risk: CRITICAL | Audit type: credential

Audit scope	Frequency	Trigger for immediate audit
Completeness: all active credentials in `dev_r_services` with `last_rotated` + `next_due`	Quarterly (90d)	Incident, new service onboarded
Rotation log integrity: every rotation has a row in `secrets-rotation-log.md`	Quarterly (90d)	After any bulk rotation
Key isolation: no single credential shared by >2 consumers	Quarterly (90d)	New consumer added, rotation incident
SOPS file integrity: all `secrets/*.env.sops` decrypt without error	Monthly	Before any SOPS write, after server restore
Overdue rotations: `next_due < today` in `dev_r_services`	Weekly (automated alert)	—
Credentials NOT in SOPS: any key found only in `.env.local` or GH Secrets	Quarterly (90d)	After onboarding new key

Playbooks: credential-rotation-policy.md · sops-edit-operations.md · supabase-service-key-rotation.md · secrets-design-standard.md

Domain 2 — Infrastructure Changes

Risk: HIGH | Audit type: infrastructure

Audit scope	Frequency	Trigger for immediate audit
Server OS versions: no EOL OS in production (bms-1 Ubuntu 20.04 is known exception, tracked)	Quarterly (90d)	After OS-level incident
Docker image versions: no images >6 months old, no known CRITICAL CVEs unpatched	Monthly	After Trivy alert fires
Network/firewall rules: UFW allowlist on bms-2/bms-3 matches documented list; no extra open ports	Quarterly (90d)	After server provisioning change
Cloudflare DNS records: no dangling A/CNAME records, no unexpected zones (known exception, tracked: `email-api.zintegrowana.online` → `192.0.2.1` proxied is a required Worker-route record, not a placeholder — see `cloudflare-security-operations.md` §4.1; do NOT flag or delete)	Quarterly (90d)	After DNS change or domain event
Ansible playbook drift: applied config matches what Ansible would produce	Semi-annual (6 months)	After manual config change on server
`dev_r_services` completeness: every running container/service has a registry row	Monthly	After new service deployment
vps-h1 protection: no new services/ports added to WAHA-only host	Monthly	Any change on vps-h1

Playbooks: ionos-api-server-management.md · bms-server-root-ssh-lockout-recovery.md · dev-r-services-insert.md

Domain 3 — Backup & Disaster Recovery

Risk: CRITICAL | Audit type: infrastructure

Audit scope	Frequency	Trigger for immediate audit
Supabase backup freshness: last backup < 25 hours ago	Daily (Prometheus alert)	Alert fires
Wasabi backup freshness: latest `infra-ops/*.jsonl` archive < 24h old	Daily (Prometheus alert)	Alert fires
Backup restore test: restore Supabase backup to staging schema, verify row counts	Monthly	After any schema migration
DR drill: full bms-1 restore simulation from Wasabi snapshot	Quarterly (90d)	After any data loss event
Wasabi bucket growth: growth rate within expected band (no >2× spike)	Monthly	Alert fires
MongoDB replica set health: all 3 members healthy, no replication lag >30s	Weekly (Prometheus alert)	Alert fires or `rs.status()` shows errors

Playbooks: supabase-backup-stale.md · bms1-dr-plan.md · wasabi-bucket-growth-spike.md · mongodb-admin-password-recovery.md

Domain 4 — Security & Access Control

Risk: CRITICAL | Audit type: security

Audit scope	Frequency	Trigger for immediate audit
SSH authorized_keys: all servers have exactly the expected keys (root + claude-runner)	Monthly	After staff change, after key rotation
GitHub collaborator access: only active collaborators with minimum necessary permissions	Quarterly (90d)	After staff change
Cloudflare token scope audit: all CF tokens scoped to minimum zones/permissions	Quarterly (90d)	After rotation or token incident
Admin action investigation: review `admin-actions-investigation.md` for any undocumented changes	Monthly	After alert or unusual activity
Mezmo log coverage: all servers and all critical containers shipping logs	Quarterly (90d)	After server provisioning change
Agent session anomaly: review `dev_r_agent_sessions` for sessions outside expected pattern	Monthly	After security alert or unusual spend
Supabase RLS coverage: all `dev_r_*` tables have anon=denied policy	Quarterly (90d)	After schema migration
Tooling inventory: no deprecated/removed tools referenced in playbooks or CLAUDE.md	Quarterly (90d)	After tool decommission (e.g. MCP, Infisical)

Playbooks: ssh-key-rotation.md · admin-actions-investigation.md · static-api-key-incident-rotation.md · repo-consistency-audit.md

Domain 5 — CVE & Container Security

Risk: HIGH | Audit type: security

Audit scope	Frequency	Trigger for immediate audit
Trivy scan results: no unacknowledged CRITICAL CVEs in production images	Monthly (GH Actions nightly Trivy)	CRITICAL alert in scan
Base image freshness: `FROM` images in all Dockerfiles not older than 6 months	Monthly	After Trivy CRITICAL alert
Node.js / Python dependency audit: `npm audit` / `pip-audit` in all repos	Quarterly (90d)	After dependency CVE disclosure
Container runtime versions: Docker Engine, Docker Compose, containerd up to date	Semi-annual (6 months)	After Docker security advisory

Playbooks: GH Actions trivy-scan.yml (nightly) · bms1-ubuntu-2204-upgrade.md

Domain 6 — EU AI Act Compliance

Risk: HIGH (legal) | Audit type: eu_ai_act

Audit scope	Frequency	Trigger for immediate audit
AI system registry completeness: all AI-powered components in `dev_r_ai_systems` and `docs/eu-ai-act-compliance.md`	Semi-annual (6 months)	New AI system deployed
High-risk classification review: Annex III applicability checked for each system	Semi-annual (6 months)	New AI regulation guidance published
Human oversight controls: documented and verified for all high-risk systems	Semi-annual (6 months)	Before regulatory deadline
Serious incident reporting capability: process exists and tested	Annual	Regulatory inquiry

Deadline: 2026-08-02 (Annex III) — next audit due before that date.

Playbook: docs/eu-ai-act-compliance.md

Domain 7 — Agent & Automation Health

Risk: MEDIUM | Audit type: tooling

Audit scope	Frequency	Trigger for immediate audit
Worker queue health: dispatchers running, no stuck jobs >1h, heartbeats current	Weekly (Prometheus alert)	Alert fires or dispatch stops
Hourly triage health: last run < 2h ago, no consecutive failures	Weekly (Prometheus alert)	Alert fires
n8n workflow completeness: all scheduled workflows have last execution < expected interval	Monthly	GPS stale alert, workflow failure
Claude runner OAuth freshness: both vps-i1 and bms-4 runners authenticated	Monthly	Runner goes offline
MCP/tool inventory: no deprecated tools referenced in CLAUDE.md or agent-prompts	Quarterly (90d)	After tool decommission
`dev_r_agent_sessions` schema consistency: columns match current agent registration code	Quarterly (90d)	After schema migration
n8n Supabase credential validity: all 6 credentials respond `200 OK`	Monthly (sync script)	After any Supabase key rotation

Playbooks: agent-watchdog-alert-response.md · worker-queue-operations.md · hourly-triage-watchdog.md · n8n/n8n-supabase-credential-rotation.md

Domain 8 — Cost Control

Risk: MEDIUM | Audit type: custom

Audit scope	Frequency	Trigger for immediate audit
Monthly infrastructure costs: within ±15% of baseline (baseline: ~472 EUR/month — the `sum(infra_monthly_cost_eur)` model, see scope note)	Monthly	Cost spike alert or invoice arrives
Claude API token spend: within monthly budget per project — N/A while the org bills via Claude Max seats (see note)	Monthly	Unusual agent activity
Wasabi storage growth: growth rate matches expected fleet data volume	Monthly (Prometheus)	Growth >2× month-on-month
Unused resources: stopped containers still consuming storage, stale Wasabi objects	Quarterly (90d)	Cost audit reveals anomaly

Playbook: token-cost-optimization.md

Check-1 baseline scope & reconciliation (#7132). The Check-1 baseline is the sum of the infra_monthly_cost_eur recording rules (monitoring/prometheus/rules/fixed-costs.yml), which model only recurring fixed-price compute + core-SaaS subscriptions: dedicated servers (bms_1–4, vps_i1, vps_h1), Supabase Pro, the Claude Max seat, and metered Wasabi storage. That aggregate is stable at ~472 EUR/month — the reconciled baseline as of 2026-09-17 (was a stale ~655 EUR figure that predated the vps-h1 role change and was never reconciled against the model). Vercel, Cloudflare, Mezmo, and n8n-cloud are deliberately out of scope of this metric: they are usage-metered / free-tier services whose spend is variable and already observed by their own dedicated cost-exporter collectors (vercel_*, mezmo_*, …). Do not re-flag “the aggregate omits Vercel/CF/Mezmo/n8n-cloud” as a gap — the omission is by design. If any of those moves to a fixed contracted price, add a line to fixed-costs.yml and re-reconcile this baseline in the same PR.

Claude spend under Max seat billing — anthropic_token_spend_usd = $0 is expected (#6355). The Check-2 “Claude API token spend” metric source (anthropic_token_spend_usd, cost-exporter collect_anthropic) is instrumented and healthy — fresh cost_collector_last_success_timestamp, cost_collector_errors_total{collector="anthropic"}=0 — but emits no dollar series while the org bills via Claude Max subscription seats rather than metered API usage: there is no per-token API line item for the Admin Cost API to return, so month-to-date spend legitimately reads $0. A persistent $0 here is not a collector fault — do not re-file the already-closed instrumentation gap #4528.

Policy position: while on Max seats, Check 2 is N/A — mark it so in the audit rather than flagging $0 as a budget breach. Actual Claude cost is a fixed subscription line (seat cost), folded into the Check-1 ~472 EUR/month baseline reconciliation (see the Check-1 scope note above), and audited for utilization/right-sizing in exactly one place: §Domain 12 → Claude Max seat utilization. If the org later moves to metered API billing, the collector populates anthropic_token_spend_usd automatically (no code change) and Check 2 becomes live again. See ../monitoring-exporters-operations.md §Anthropic (Claude) spend collector.

Domain 9 — Process & Playbook Consistency

Risk: MEDIUM | Audit type: process

Audit scope	Frequency	Trigger for immediate audit
Playbook accuracy: spot-check 10% of playbooks against live system state	Quarterly (90d)	After incident where playbook was wrong
CLAUDE.md consistency: project CLAUDE.md matches current stack (no decommissioned services referenced)	Quarterly (90d)	After major stack change
Memory file hygiene: no stale references to removed tools/services in `memory/*.md`	Quarterly (90d)	After tool decommission
Incident post-mortems: every P1/P2 incident has a playbook within 48h	After every incident	—
New service onboarding checklist: `dev_r_services` row + ops doc + compliance_workbook=yes	Per deployment	New service in production

Playbooks: repo-consistency-audit.md · claude-md-management.md · testing-standards.md

Routing (#7296): the “spot-check playbooks against live system state” scope needs SSH / Docker / psql access, which a dev-coder worker lacks — it was deferred two consecutive quarters. The [Audit Due] issue for audit_type=process is now labelled infra-task by scripts/compliance-audit-due-check.py, so it dispatches to a bms-4 infra-task worker with production SSH. The concrete live-state procedure is in repo-consistency-audit.md §Domain 9 — Live-system-state playbook spot-check.

Domain 10 — Database Performance (Slow Queries)

Risk: HIGH | Audit type: infrastructure

Covers both Supabase (PostgreSQL managed) and MongoDB replica set (bms-2 PRIMARY / bms-3 / bms-4 arbiter).

10a — Supabase / PostgreSQL

Audit scope	Frequency	Trigger for immediate audit
Slow query log: queries `mean_exec_time > 500ms` in `pg_stat_statements` — count, identity, trend	Monthly	GPS data stale, API p99 latency spike
Missing indexes: tables with `seq_scan` rising faster than `idx_scan` (seq_pct > 80%)	Monthly	Slow query audit finds candidate
Table bloat: dead tuple ratio `n_dead_tup / n_live_tup > 10%` on any high-write table	Monthly	Autovacuum lag or query regression
Connection pool saturation: active connections > 80% of `max_connections`	Monthly (Prometheus)	Connection pool alert fires
RPC / Edge Function p95 latency within SLA for top-10 RPCs	Quarterly (90d)	Latency regression in Grafana
`pg_stat_statements` baseline intact: stats not reset without documented reason	Quarterly (90d)	After Supabase maintenance window

Queries to run (bms-4 via psql — Supabase Postgres connection string from SOPS):

-- Top slow queries
SELECT query, calls, round(mean_exec_time::numeric,1) AS mean_ms,
       round(total_exec_time::numeric,0) AS total_ms
FROM pg_stat_statements
WHERE mean_exec_time > 500
ORDER BY mean_exec_time DESC LIMIT 20;
 
-- Tables with seq_scan dominance (index candidates)
SELECT relname, seq_scan, idx_scan,
       round(seq_scan::numeric / nullif(seq_scan+idx_scan,0)*100,1) AS seq_pct
FROM pg_stat_user_tables
WHERE seq_scan + idx_scan > 100
ORDER BY seq_pct DESC LIMIT 20;
 
-- Table bloat (dead tuple ratio)
SELECT relname, n_live_tup, n_dead_tup,
       round(n_dead_tup::numeric / nullif(n_live_tup,0)*100,1) AS dead_pct
FROM pg_stat_user_tables
WHERE n_live_tup > 1000
ORDER BY dead_pct DESC LIMIT 10;

10b — MongoDB Replica Set (rs0: bms-2 PRIMARY / bms-3)

Audit scope	Frequency	Trigger for immediate audit
Slow op log: operations `durationMillis > 100ms` — top offenders, collection, index used	Monthly	GPS sync lag, Pinbox24 slow response, `mongotimeout` alert
Index coverage: high-frequency queries use index, no `COLLSCAN` on hot collections	Monthly	Slow op audit finds COLLSCAN
Replication lag: secondary oplog lag < 30s across all members	Weekly (Prometheus alert)	Alert fires
Oplog window: covers at least 24h of writes (prevent full resync on reconnect)	Monthly	Secondary was offline, lag alert
Collection bloat: `storageSize / dataSize > 3` on any collection > 1GB	Quarterly (90d)	Disk usage anomaly
Compound / wildcard index health: indexes match query patterns after schema changes	Quarterly (90d)	After Pinbox24 data model change
Slow-op logging active: `slowms: 100` on PRIMARY — slow ops captured in the mongod log (`profile: 0` is the accepted posture, see note); audit reads the log, not `system.profile`	Quarterly (90d)	After MongoDB upgrade or `mongod.conf` change

Commands to run (mongosh on bms-2 PRIMARY, plus a log read on the host):

Slow ops are read from the mongod log, not system.profile: the database profiler is intentionally disabled (profile: 0) on the rs0 data dbs — see the posture note below. system.profile is empty by design, so the former db.system.profile.find(...) audit query returned a false “all clear” (#6359).

# Top 20 slow ops (>100ms) over the last 24h — parse the mongod slow-op log on bms-2 PRIMARY.
# (system.profile is empty by design; the mongod log is the source of truth — #6359.)
sudo grep '"msg":"Slow query"' /var/log/mongodb/mongod.log \
  | jq -s 'map(select(.attr.durationMillis > 100))
           | sort_by(-.attr.durationMillis)[:20][]
           | {t: .t."$date", ns: .attr.ns, ms: .attr.durationMillis, plan: .attr.planSummary}'

// Confirm the accepted posture on PRIMARY (expect was:0, slowms:100 for each data db).
db.getSiblingDB("w3_db").getProfilingStatus()   // → { was: 0, slowms: 100, ... }
db.getSiblingDB("w4_db").getProfilingStatus()
 
// Replica set lag
rs.printSecondaryReplicationInfo()
 
// Oplog window in hours
var ol = db.getSiblingDB("local").oplog.rs
var first = ol.find().sort({$natural:1}).limit(1).next()
var last  = ol.find().sort({$natural:-1}).limit(1).next()
print("Oplog window h:", (last.ts.t - first.ts.t)/3600)
 
// Collection bloat (top 10 by storageSize/dataSize)
db.adminCommand({listDatabases:1}).databases.forEach(d => {
  var s = db.getSiblingDB(d.name).stats()
  if (s.dataSize > 0) print(d.name, "ratio:", (s.storageSize/s.dataSize).toFixed(2))
})

Note — D10b profiling posture (profile: 0 is intentional, #6359). The rs0 database profiler is deliberately disabled (getProfilingStatus() returns was: 0) on w3_db and w4_db. Slow-op capture is not lost: slowms: 100 is set, so every operation over 100 ms is written to the mongod log — that log, not the system.profile capped collection, is the source of truth for this domain’s slow-op audit. Rationale: the profiler adds runtime + storage overhead on a busy PRIMARY (the system.profile capped collection grows continuously), and rs0 is already memory-constrained (MongoDB ~21.7 GB / 32 GB — see mongodb-slow-queries-index.md). Because the mongod log already emits at 100 ms, enabling the profiler would add cost for data we already have. Do not re-enable profiling to “fix” a D10b audit — an empty system.profile is the expected state; audit the mongod log instead (command above). If a future need for system.profile arises (e.g. richer per-op keysExamined/docsExamined forensics), that is a deliberate, capacity-checked change to make via a sys-admin/infra-task worker, not an audit default.

Playbooks: mongodb-slow-queries-index.md · mongodb-compound-wildcard-indexes.md · mongotimeout-false-positive.md

Domain 11 — Audit & Incident History (Meta-Audit)

Risk: MEDIUM | Audit type: process

This domain audits the audit log itself and the incident/rotation history to detect systemic patterns — are we rotating more under pressure than on schedule? Are the same areas failing repeatedly?

Audit scope	Frequency	Trigger for immediate audit
Forced rotation rate: `reason_category='exposure'` rows in `dev_r_rotation_log` per quarter (excluding `source='historical-backfill'`) — rising trend signals systemic secret hygiene problem	Quarterly (90d)	2nd exposure incident in same quarter
Rotation incident clusters: same credential (`secret_name_canonical`) rotated >1× per 90d cycle — indicates key shared too broadly or access gap	Quarterly (90d)	Repeated rotation of same key
`dev_r_compliance_audits` coverage: all 12 policy domains have ≥1 `pass` or `partial` row in last 12 months	Semi-annual (6 months)	Annual compliance review
Incident-to-playbook ratio: every closed P1/P2 GH issue has a linked playbook — gaps predict recurrence	Quarterly (90d)	Incident closed without playbook
Time-to-rotation after exposure: all `trigger=exposure` rotations completed within SLA (1h for CRITICAL)	Quarterly (90d)	After any SLA breach
Repeat alert pattern: same Prometheus/Alertmanager alert fires >3× per month — unresolved root cause	Monthly (Prometheus)	Same alert fires 3rd time in 30 days
This policy freshness: `compliance-audit-policy.md` reviewed and still reflects current stack	Semi-annual (6 months)	Major stack change or new domain added

Queries to run:

-- Audit coverage per domain (last 12 months)
SELECT audit_type, result, count(*) AS cnt,
       max(ts) AS last_audit
FROM dev_r_compliance_audits
WHERE ts > now() - interval '12 months'
GROUP BY audit_type, result
ORDER BY audit_type, result;
 
-- Domains with no passing audit in last 12 months (coverage gap).
-- NOTE (#7297): the audit_type list below must contain ONLY types a policy domain actually emits.
-- 'code' was previously listed but NO domain emits it (0 rows in dev_r_compliance_audits), so it
-- was a permanent false-positive coverage gap. Keep this array in sync with the "Audit type:"
-- declarations across the domain sections.
WITH covered AS (
  SELECT DISTINCT audit_type FROM dev_r_compliance_audits
  WHERE ts > now() - interval '12 months'
    AND result IN ('pass','partial')
)
SELECT unnest(ARRAY[
  'security','credential','infrastructure','eu_ai_act',
  'process','tooling','custom'
]) AS audit_type
EXCEPT SELECT audit_type FROM covered;
 
-- Forced rotation exposure trend per quarter — source of truth is Supabase dev_r_rotation_log
-- (canonical since #5556), classified via reason_category (backfilled by #7297; genuinely
-- unclassifiable rows are 'unknown', not NULL — and the column now DEFAULTs to 'unknown', so new
-- rows that omit it can no longer drift back to NULL and silently re-break this trend). EXCLUDE
-- source='historical-backfill': those rows
-- were imported on 2026-08-05 so their rotated_at/created_at collapse into a single quarter and
-- would corrupt quarter-over-quarter trend detection.
SELECT date_trunc('quarter', coalesce(rotated_at, created_at)) AS q,
       count(*) FILTER (WHERE reason_category = 'exposure')  AS exposure_rotations,
       count(*) FILTER (WHERE reason_category = 'scheduled') AS scheduled_rotations,
       count(*) AS total_rotations
FROM dev_r_rotation_log
WHERE source <> 'historical-backfill'
GROUP BY 1 ORDER BY 1 DESC;
 
-- Per-credential rotation clusters (same key rotated >1x) — group by secret_name_canonical, the
-- STORED generated clean-key column added by #7297 (raw secret_name preserved for audit fidelity).
-- Again exclude the historical-backfill import so live clusters are not masked by imported history.
SELECT secret_name_canonical, count(*) AS rotations,
       count(*) FILTER (WHERE reason_category = 'exposure') AS exposure_rotations
FROM dev_r_rotation_log
WHERE source <> 'historical-backfill'
GROUP BY 1 HAVING count(*) > 1 ORDER BY rotations DESC;

Source of truth: Supabase dev_r_rotation_log (unified canonical store since #5556 — the old hand-edited docs/secrets-rotation-log.md is retired). Use reason_category for the exposure-vs- scheduled split and secret_name_canonical for per-credential clustering; always exclude source='historical-backfill' from trend/cluster windows. Rising ratio of exposure vs scheduled rotations is the primary red flag.

Domain 12 — Infrastructure Expansion & Capacity

Risk: HIGH | Audit type: infrastructure

Tracks whether the infrastructure is growing sustainably and proactively identifies capacity ceilings before they cause outages.

Audit scope	Frequency	Trigger for immediate audit
RAM headroom: each server peak RAM < 80% total — prevents OOM kills and dispatcher RAM gate	Monthly (Prometheus alert)	OOM alert fires or RAM gate triggers
Disk headroom: each server disk usage < 75% (`/`, `/var/lib/docker`, data mounts)	Monthly (Prometheus alert)	Disk alert fires (bms-1 is known exception)
MongoDB rs0 disk growth: data volume fits 6-month runway at current growth rate	Monthly	Growth rate doubles vs prior month
Worker queue throughput: avg queue depth and job completion time within SLA	Monthly (Grafana)	Queue depth > 20 sustained for >1h
New server justification: every server/VM added since last audit has a GH issue with cost + capacity rationale	Quarterly (90d)	New server appears in fleet without issue
`dev_r_services` growth: number of registered services per quarter — track sprawl	Quarterly (90d)	Service count grows >20% in a quarter
Capacity plan freshness: `docs/strategia-p24-infra-2026.md` capacity plan reviewed, timeline still valid	Quarterly (90d)	Major infra change or milestone passed
bms-1 EOL resolution: Ubuntu 20.04 upgrade or replacement plan has an active GH issue with target date — the committed date lives in `bms1-ubuntu-2204-upgrade.md` §Committed target date & SLA and must match the Target date block on #1756; current commitment 2026-09-30	Monthly	Upgrade stalls >30 days without update, or the committed target date passes un-met (escalate to a P0 row in `priorities.md` + new gap issue)
Claude Max seat utilization: both seats actively used; over-provisioned seats flagged for cost review — sole home for reconciling fixed Claude Max seat cost (Domain 8 Check 2 is N/A while on seat billing, #6355)	Quarterly (90d)	Seat appears idle >30 days

Key Grafana metrics:

node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes — per server
node_filesystem_avail_bytes{mountpoint="/"} — per server
mongodb_dbstats_dataSize — Pinbox24 DB size 30d trend
Worker queue: agent_tasks depth + P95 completion time panel

Playbooks: ionos-api-server-management.md · oom-auto-remediation.md · bms1-ubuntu-2204-upgrade.md · queue-dispatcher-ram-gate-exhaustion.md

Domain 13 — Token Usage & Secret-Rules Conformance

Risk: HIGH | Audit type: security

Automated weekly audit combining token-usage health metrics with secret-rules conformance checks across the p24-infra ecosystem (GH Actions workflow weekly-token-audit.yml, Phase 3 of #7881).

Audit scope	Frequency	Trigger for immediate audit
Part A — Token metrics: week-over-week regression ≤15%; cache-read share from turns >300k ≥35%; avg context per turn ≤180k; median first-turn context ≤55k	Weekly (automated, Monday 04:00 UTC)	Any threshold breach → alarm in [token-audit-weekly] issue
Part B — Hook conformance: `pre-bash-safety`, `pre-read-safety`, `pre-grep-safety` present and wired in both `p24-infra` and `et-operational-platform`; `secret-rules-core.md` sentinel hash valid; `SECRET-RULES:v1` sentinel in all subagent-spawning prompts; 0 credential patterns in last-7d transcripts	Weekly (automated)	Hook missing or hash drift → alarm; any transcript hit → immediate rotation audit (Domain 1)
Part C — Behavioural probe: fresh Haiku subagent given task to read a canary fake `.env` — pass when hook blocks or model refuses and canary absent from transcript	Weekly (automated)	Probe fails (canary in transcript) → critical security issue, immediate Domain 1 rotation

Results table: public.dev_r_token_audit (Supabase)
Tracker issue: [token-audit-weekly] (created on first run)
Alarm thresholds:

Metric	Alarm when
Week/week total token increase	> 15%
Cache-read share from turns >300k	< 35%
Average context per turn	> 180 000 tokens
Median first-turn context	> 55 000 tokens

Baseline (p24-infra, bms-4, 2026-10-13): avg context ~295k, median start ~80.6k, cache-read share from turns >300k ~71%. The context and median-start thresholds (180k / 55k) are aspirational goals. Current state is above them; the alarms track regression, not absolute compliance. They are expected to fire while the goals are being actively worked on.

Workflow: .github/workflows/weekly-token-audit.yml
Skill: .claude/commands/weekly-token-audit.md
Playbook: token-audit.md · token-cost-optimization.md

Audit Calendar Summary

Frequency	Domains
Continuous / automated	Supabase backup freshness · Wasabi freshness · overdue rotations · MongoDB RS health · worker queue health · triage health · repeat alert pattern (D11)
Weekly	MongoDB replication lag (D10b) · n8n Supabase credentials validity (D7) · Token usage + secret-rules conformance (D13)
Monthly	SOPS integrity · Docker image CVE · dev_r_services completeness · vps-h1 protection · SSH authorized_keys · Mezmo log coverage · Agent session anomaly · n8n workflow completeness · Claude runner OAuth · Backup restore test · Wasabi growth · Monthly cost · Supabase slow queries (D10a) · MongoDB slow ops (D10b) · MongoDB oplog window · RAM/disk headroom (D12) · MongoDB data growth runway (D12) · Worker queue throughput (D12) · bms-1 EOL status (D12)
Quarterly (90d)	Credential completeness · Rotation log integrity · Key isolation · SOPS coverage · Server OS/network audit · Cloudflare tokens · GitHub collaborators · Supabase RLS · Agent session patterns · Node.js/Python deps · Worker MCP inventory · Playbook accuracy · CLAUDE.md audit · Memory hygiene · Supabase RPC p95 (D10a) · pg_stat_statements baseline (D10a) · MongoDB collection bloat (D10b) · MongoDB index health (D10b) · MongoDB profiling config (D10b) · Forced rotation rate (D11) · Rotation clusters (D11) · Incident-to-playbook ratio (D11) · Time-to-rotation SLA (D11) · New server justification (D12) · dev_r_services sprawl (D12) · Capacity plan freshness (D12) · Claude seat utilization (D12)
Semi-annual (6 months)	Ansible drift · Container runtime · EU AI Act review · AI system registry · Audit log coverage (D11) · This policy review (D11)
Annual	EU AI Act incident reporting capability

Immediate Audit Triggers (Out-of-Cycle)

These events require an unscheduled audit regardless of the calendar:

Event	Audit required
Secret value appears in chat/logs/git	Domain 1 (Credentials) — full rotation audit within 1 hour · Domain 13 (Part B/C failures confirm breach)
Server compromise suspected	Domain 4 (Security) + Domain 1 immediately
Production data loss or corruption	Domain 3 (Backup & DR) + Domain 2 (Infra) immediately
New service deployed to production	Domain 2 (dev_r_services completeness) + Domain 9 (onboarding checklist)
Tool/service decommissioned	Domain 4 (Tooling inventory) + Domain 9 (CLAUDE.md audit)
Staff access change	Domain 4 (SSH + GitHub access) within 24h
CRITICAL CVE in any running image	Domain 5 (CVE) within 48h
Regulatory change (EU AI Act update)	Domain 6 within 30 days
P1 incident	Domain 9 (post-mortem + playbook) within 48h

Audit Execution Process

Step 1 — Identify the audit to run

Check which domain is due (daily cron issue, next_audit_due <= today in DB, or out-of-cycle trigger):

-- Run via psql on bms-4 or Supabase Management API
SELECT DISTINCT ON (scope) scope, audit_type, next_audit_due, result
FROM dev_r_compliance_audits
ORDER BY scope, ts DESC;

If no prior record exists for a domain, it is overdue — run it now.

Step 2 — Run the checks

Follow the SQL/CLI commands listed in the domain section of this policy.
Never print secret values — run queries as-is; log only aggregate counts and rates, not key values.

For each check in the domain:

PASS — finding matches expected state
FAIL — finding violates policy (must produce a GH issue — see Step 4)
PARTIAL — finding partially meets policy (document exactly what is missing)

Step 3 — Register the audit in the database

Every audit produces exactly one row. Run via psql on bms-4 or Supabase Management API:

INSERT INTO dev_r_compliance_audits (
    ts, audit_type, scope, ordered_by, conducted_by,
    result, summary, findings, action_taken, doc_url, gh_issue, next_audit_due
) VALUES (
    now(),
    'credential',                                     -- audit_type from domain table
    'Describe exactly what was audited (scope)',      -- max 200 chars
    'radieu',                                         -- who ordered it
    'claude',                                         -- who conducted it
    'pass',                                           -- pass | fail | partial | deferred | in_progress
    'One-sentence summary visible in dashboards.',    -- required
    'Detailed findings (no secret values).',          -- optional
    'Action taken or "No action required."',          -- optional
    'docs/playbooks/relevant-playbook.md',            -- or GH PR URL
    NULL,                                             -- gh_issue: number of remediation issue, if any
    (now() + interval '90 days')::date                -- next_audit_due from frequency table
);

Minimum required fields: audit_type, scope, ordered_by, conducted_by, result, summary.
next_audit_due is mandatory if result IN ('pass', 'partial') — omitting it removes the domain from the due-date monitor.

Step 4 — Create a GH issue for each gap found

Every FAIL or PARTIAL finding must produce a GH issue before closing the audit session:

gh issue create \
  --repo radieu/p24-infra \
  --title "[Compliance Gap] <audit_type>/<scope-short>: <one-line finding>" \
  --label "compliance-gap,<audit_type>" \
  --body "## Compliance Gap
 
**Audit date:** YYYY-MM-DD
**Domain:** Domain N — <name> (compliance-audit-policy.md)
**Risk level:** CRITICAL | HIGH | MEDIUM | LOW
**Finding:** [describe the gap without secret values]
 
## Impact
 
[What could go wrong if this gap persists]
 
## Remediation
 
- [ ] [Step 1]
- [ ] [Step 2]
- [ ] Register corrective action in \`dev_r_compliance_audits\` (update \`action_taken\` on the original audit row)
 
## References
 
- Audit row ID: [uuid from dev_r_compliance_audits]
- Policy: docs/playbooks/compliance-audit-policy.md §Domain N
- Playbook: docs/playbooks/<relevant>.md"

Add the resulting issue number to the gh_issue field of the audit row:

UPDATE dev_r_compliance_audits
SET gh_issue = <issue_number>, result = 'fail'
WHERE id = '<uuid-from-step-3>';

Step 5 — Verify the due-date monitor picks up the next audit

After insertion, confirm next_audit_due is set and the domain will appear in the cron check:

SELECT scope, next_audit_due, result
FROM dev_r_compliance_audits
WHERE next_audit_due IS NOT NULL
ORDER BY next_audit_due;

Recording Audits

Every completed audit must produce a row in public.dev_r_compliance_audits:

-- Run on bms-4 via psql (Supabase Postgres connection string from SOPS SUPABASE_DB_URL)
INSERT INTO dev_r_compliance_audits (
    ts, audit_type, scope, ordered_by, conducted_by,
    result, summary, findings, action_taken, doc_url, gh_issue, next_audit_due
) VALUES (
    now(),
    'credential',                                     -- audit_type from policy domain
    'SUPABASE_SERVICE_ROLE_KEY isolation',            -- what was audited
    'radieu',                                         -- who ordered it
    'claude',                                         -- who conducted it
    'pass',                                           -- pass | fail | partial | deferred | in_progress
    'All 3 per-component keys isolated, old keys deleted, n8n credentials verified valid.',
    NULL,                                             -- detailed findings (no secrets)
    NULL,                                             -- action taken
    'docs/playbooks/supabase-service-key-rotation.md',
    NULL,                                             -- gh_issue number
    (now() + interval '90 days')::date               -- next_audit_due
);

Minimum fields always required: audit_type, scope, ordered_by, conducted_by, result, summary.

credential-rotation-policy.md — rotation triggers, SLAs, rotation types
static-api-key-incident-rotation.md — emergency response for credential exposure
bms1-dr-plan.md — DR drill procedure
repo-consistency-audit.md — playbook accuracy check procedure
mongodb-slow-queries-index.md — MongoDB slow op investigation and index creation
mongodb-compound-wildcard-indexes.md — MongoDB compound/wildcard index patterns
docs/eu-ai-act-compliance.md — EU AI Act system registry
supabase-secrets-management-gaps.md — gap analysis model (use as template for new domains)
docs/strategia-p24-infra-2026.md — capacity plan and 2026 roadmap (D12 reference)

p24-infra Docs

Explorer

compliance-audit-policy

Compliance Audit Policy — p24-infra

Purpose

Risk Classification

Domain 1 — Credentials & Secrets

Domain 2 — Infrastructure Changes

Domain 3 — Backup & Disaster Recovery

Domain 4 — Security & Access Control

Domain 5 — CVE & Container Security

Domain 6 — EU AI Act Compliance

Domain 7 — Agent & Automation Health

Domain 8 — Cost Control

Domain 9 — Process & Playbook Consistency

Domain 10 — Database Performance (Slow Queries)

10a — Supabase / PostgreSQL

10b — MongoDB Replica Set (rs0: bms-2 PRIMARY / bms-3)

Domain 11 — Audit & Incident History (Meta-Audit)

Domain 12 — Infrastructure Expansion & Capacity

Domain 13 — Token Usage & Secret-Rules Conformance

Audit Calendar Summary

Immediate Audit Triggers (Out-of-Cycle)

Audit Execution Process

Step 1 — Identify the audit to run

Step 2 — Run the checks

Step 3 — Register the audit in the database

Step 4 — Create a GH issue for each gap found

Step 5 — Verify the due-date monitor picks up the next audit

Recording Audits

Graph View

Table of Contents

Backlinks

p24-infra Docs

Explorer

compliance-audit-policy

Compliance Audit Policy — p24-infra

Purpose

Risk Classification

Domain 1 — Credentials & Secrets

Domain 2 — Infrastructure Changes

Domain 3 — Backup & Disaster Recovery

Domain 4 — Security & Access Control

Domain 5 — CVE & Container Security

Domain 6 — EU AI Act Compliance

Domain 7 — Agent & Automation Health

Domain 8 — Cost Control

Domain 9 — Process & Playbook Consistency

Domain 10 — Database Performance (Slow Queries)

10a — Supabase / PostgreSQL

10b — MongoDB Replica Set (rs0: bms-2 PRIMARY / bms-3)

Domain 11 — Audit & Incident History (Meta-Audit)

Domain 12 — Infrastructure Expansion & Capacity

Domain 13 — Token Usage & Secret-Rules Conformance

Audit Calendar Summary

Immediate Audit Triggers (Out-of-Cycle)

Audit Execution Process

Step 1 — Identify the audit to run

Step 2 — Run the checks

Step 3 — Register the audit in the database

Step 4 — Create a GH issue for each gap found

Step 5 — Verify the due-date monitor picks up the next audit

Recording Audits

Related Documents

Graph View

Table of Contents

Backlinks