p24-infra Docs

infra-ops-cold-archive

2 min read

infra-ops cold archive

Scheduled flush of aged public.infra_operations rows to Wasabi cold storage. This is the deployed replacement for the audit-engine infra_ops_archive scheduled action, which never runs because audit-engine is not deployed.

  • Compliance: closes D3 Gap 1 (#4523) — “latest infra-ops/*.jsonl archive < 24h old”.
  • Design: docs/designs/1553-infra-operations-audit-log.md.
  • Policy: docs/playbooks/compliance-audit-policy.md §Domain 3.

What it does

  1. Selects rows from public.infra_operations older than RETENTION_WEEKS (default 8).
  2. Groups them by YYYY-MM and appends to s3://p24-infra/infra-ops/YYYY-MM.jsonl (JSON Lines, append-safe — the object is read, concatenated, re-uploaded).
  3. Deletes the successfully-archived rows from Supabase (batches of 100).
  4. On any month’s upload failure it raises and does not delete those rows (no data loss), then fires a Discord alert + files a [bug] GH issue.

Supabase hot storage keeps ~2 months; Wasabi keeps the long-term EU AI Act Art. 13 traceability trail.

Components

PiecePath
Scheduled workflow.github/workflows/infra-ops-cold-archive.yml
Standalone runnerscripts/infra_ops_cold_archive.py
Archive core (shared with audit-engine)audit-engine/actions/infra_ops_archive.py → archive_infra_operations()
Unit testsaudit-engine/tests/test_actions_infra_ops_archive.py

The runner imports archive_infra_operations() from the audit-engine module so there is a single source of truth; if/when audit-engine is deployed, its scheduler runs the same core via run(action).

Schedule

  • Cron: 0 4 1 * * (monthly, 1st at 04:00 UTC — after the weekly Supabase full dump).
  • Manual: gh workflow run infra-ops-cold-archive.yml --repo radieu/p24-infra (optional input retention_weeks).
  • Runner: [self-hosted, bms4] (IPv6 to Supabase + Wasabi IAM creds via GH Secrets).

Secrets (all existing GH Secrets — none newly requested)

SUPABASE_URL, SUPABASE_SERVICE_KEY, P24_INFRA_WASABI_ACCESS_KEY, P24_INFRA_WASABI_SECRET_KEY, P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL, and the built-in GITHUB_TOKEN (for the failure issue).

Verify a run

# List archive objects (needs Wasabi creds; run from a keyed host)
python3 scripts/wasabi-cli.py ls infra-ops/    # expect YYYY-MM.jsonl files

The daily D3 Prometheus check (latest infra-ops/*.jsonl < 24h old) is informational here — the archive cadence is monthly, so freshness is bounded by the cron, not 24h. Treat a persistently-empty infra-ops/ as the failure signal.

  • Backfill of historical rows: scripts/backfill-infra-ops-from-rotation-log.py
  • D3 Gap 2 (bms-1 DR drill) is tracked separately on #4523 — it needs an SSH-keyed sys-admin session and is not covered by this workflow.

Graph View