p24-infra Docs

supabase-backup-stale

5 min read

Playbook: SupabaseBackupStale — Supabase Daily Backup Stale (>26h)

Alert: SupabaseBackupStale
Exporter: backup-exporter:9220
Severity: Critical
SLO: Backup must succeed daily; alert fires when supabase_backup_db_age_seconds > 93600 (26h)

What triggers this problem

The supabase-backup GitHub Actions workflow runs daily at 03:30 UTC on the self-hosted ionos runner (vps-i1). The workflow writes a metrics status file to Wasabi S3 (p24-infra bucket) on every run. The backup-exporter reads that file every scrape; if the file is absent or too old, the alert fires.

Common root causes (ranked by frequency as of 2026-06):

  1. IONOS runners OOM-killed — most likely. vps-i1 has 8GB RAM shared between the monitoring stack and runner jobs. Heavy jobs (Docker pg_dump, Python tests with large imports) can exhaust RAM, triggering OOM kills on both ionos-localhost and ionos-localhost-2 runner services. Once killed they go offline and all queued jobs stay stuck.
  2. Runners not restarted after OOM — the runner service exits cleanly (code 0) after OOM, so Restart=on-failure does not auto-restart it.
  3. Workflow runs stuck in queue — queued runs do not expire automatically; they wait indefinitely until a runner picks them up.
  4. Historical: GitHub-hosted runners out of minutes — resolved by PR #1045 (2026-06-22) which migrated to self-hosted ionos runner.

How to confirm it (exact commands)

# 1. Check backup-exporter metric directly (from local workstation)
ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 `
  "curl -s http://localhost:9220/metrics | grep supabase_backup"
 
# Expected when healthy (age < 93600):
#   supabase_backup_db_age_seconds 12345.0
# Unhealthy: metric line present but NO VALUE (gauge with no sample)
 
# 2. Check recent workflow runs
gh run list --workflow supabase-backup.yml --limit 10 --repo radieu/p24-infra
 
# 3. Check runner status
gh api repos/radieu/p24-infra/actions/runners --jq '.runners[] | {name: .name, status: .status, busy: .busy}'
 
# 4. Check for OOM in system journal (run on vps-i1)
ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 `
  "journalctl -k --since '24 hours ago' | grep -i 'oom\|out of memory'"

Step-by-step fix

Step 1: Confirm runners are offline

gh api repos/radieu/p24-infra/actions/runners --jq '.runners[] | {name: .name, status: .status}'

If both show "status": "offline" proceed to Step 2.

Step 2: Restart runners on vps-i1

ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 @'
systemctl start actions.runner.radieu-p24-infra.ionos-localhost.service
systemctl start actions.runner.radieu-p24-infra.ionos-localhost-2.service
sleep 5
systemctl is-active actions.runner.radieu-p24-infra.ionos-localhost.service
systemctl is-active actions.runner.radieu-p24-infra.ionos-localhost-2.service
'@

Step 3: Cancel stale queued runs (if any)

# List queued runs
gh run list --workflow supabase-backup.yml --limit 10 --repo radieu/p24-infra
 
# Cancel each queued run ID
gh run cancel <RUN_ID> --repo radieu/p24-infra

Step 4: Trigger a fresh backup

gh workflow run supabase-backup.yml --repo radieu/p24-infra --field backup_type=daily

Step 5: Verify backup succeeds

# Get the new run ID
gh run list --workflow supabase-backup.yml --limit 3 --repo radieu/p24-infra
 
# Watch the run (takes 5-15 minutes)
gh run watch <RUN_ID> --repo radieu/p24-infra
 
# After success, confirm metric is back
ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 `
  "curl -s 'http://localhost:9090/api/v1/query?query=supabase_backup_db_age_seconds'"

If runners die again after restart (recurring OOM)

The runner services have MemoryAccounting=no + Restart=on-failure systemd overrides (applied 2026-06-23). If OOM recurs:

# Check current memory on vps-i1
ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 "free -h && docker stats --no-stream"
 
# Kill any stuck Docker containers from failed jobs
ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 `
  "docker ps -a --filter status=exited -q | xargs docker rm -f"
 
# Verify overrides are still in place
ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 `
  "systemctl show actions.runner.radieu-p24-infra.ionos-localhost.service | grep -E 'MemoryAccounting|Restart='"

If MemoryAccounting=yes (override missing — re-apply):

ssh -i "C:\Users\konar\.ssh\id_ed25519" root@217.154.82.162 @'
mkdir -p /etc/systemd/system/actions.runner.radieu-p24-infra.ionos-localhost.service.d
mkdir -p /etc/systemd/system/actions.runner.radieu-p24-infra.ionos-localhost-2.service.d
cat > /etc/systemd/system/actions.runner.radieu-p24-infra.ionos-localhost.service.d/override.conf << OVERRIDE
[Service]
MemoryAccounting=no
Restart=on-failure
RestartSec=30s
OVERRIDE
cat > /etc/systemd/system/actions.runner.radieu-p24-infra.ionos-localhost-2.service.d/override.conf << OVERRIDE
[Service]
MemoryAccounting=no
Restart=on-failure
RestartSec=30s
OVERRIDE
systemctl daemon-reload
'@

Escalation path

If the workflow still fails after multiple restarts (non-runner cause):

  1. View workflow logs: gh run view --log --job=<JOB_ID> --repo radieu/p24-infra
  2. Check Supabase pooler connectivity from vps-i1: nc -zv aws-0-eu-central-1.pooler.supabase.com 5432
  3. Check Wasabi write access: verify P24_INFRA_WASABI_ACCESS_KEY secret is valid in GitHub Secrets
  4. If pooler is unreachable: check status.supabase.com
  5. Create a human-action GitHub issue if root cause requires account-level intervention

Prevention notes

  • Systemd overrides applied 2026-06-23: MemoryAccounting=no on both runner services prevents memcg-constrained OOM kills; Restart=on-failure handles non-OOM exits.
  • MemoryAccounting=no note: this disables per-service cgroup memory accounting so the system-level OOM killer decides (uses all available RAM + swap before killing), giving the runner more headroom for heavy Docker jobs.
  • Runner isolation: consider running only one runner at a time on vps-i1, or migrating heavy test jobs to bms-4 which has 32GB RAM.
  • Monitoring: the SupabaseBackupStale alert fires after 26h — respond within 2h to avoid a full backup gap day.
  • Wasabi backup is secondary: Supabase managed PITR backups remain active regardless of this workflow. This workflow provides an off-platform offsite copy for disaster recovery.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="other",
    resource="supabase-backup",
    result="success",  # "success" | "failed" | "skipped"
    detail="Supabase backup staleness resolved — backup process manually triggered",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'other', 'supabase-backup', 'success', 'Supabase backup staleness resolved — backup process manually triggered', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks