vps-i1-crash-loop-recovery

Playbook: vps-i1 Crash Loop / SSH Unreachable

What triggers this

• A container enters a rapid restart loop (OOM kill cycle, startup crash)
• Multiple crashlooping containers saturate CPU/RAM → kernel OOM kills sshd
• GitHub Actions runners spawn Python jobs (~2GB each) on top of the monitoring stack → global OOM
• SSH becomes completely unreachable (TCP:22 closed or timeout)
• Server may auto-reboot or remain in a frozen state

Common root causes

Symptom	Root cause	Fix
SSH banner timeout (load >20)	Container crash-looping	Stop the offending container
TCP:22 closed after reboot	Prometheus WAL replay OOM	Prometheus mem_limit too low
SSH drops after ~90s of boot	Prometheus hits mem_limit	See Step 3
Crash loop with no obvious container	Missing env vars at startup	Check .env exists on server
Recurring OOM every few hours (root cause discovered 2026-06-24)	GH Actions runners (ionos-localhost, ionos-localhost-2) each spawn python3 jobs using ~2GB RSS. With monitoring stack at ~5-6GB, 2 concurrent runner jobs → global OOM	Disable both runners permanently — all agent work runs on bms-4 (32GB)

CRITICAL: GH Actions runners must not run on vps-i1

vps-i1 has 8GB RAM. The monitoring stack uses ~5-6GB at steady state. Any GH Actions runner job that spawns a Claude Code Python process (~2GB) will push the server into global OOM.

Fix (one-time, already applied 2026-06-24):

systemctl stop 'actions.runner.radieu-p24-infra.ionos-localhost.service'
systemctl stop 'actions.runner.radieu-p24-infra.ionos-localhost-2.service'
systemctl disable 'actions.runner.radieu-p24-infra.ionos-localhost.service'
systemctl disable 'actions.runner.radieu-p24-infra.ionos-localhost-2.service'

If runners somehow get re-enabled: repeat the above. All agent work belongs on bms-4.

How to confirm

From local workstation:

Test-NetConnection -ComputerName 217.154.82.162 -Port 22 -InformationLevel Quiet

From IONOS console (VNC):

docker ps --filter "status=restarting"
cat /proc/loadavg

Step-by-step fix

1. Restart from IONOS console

Log into IONOS panel → select vps-i1 → Restart (hard reset if soft fails).

2. Start a fast-poller immediately (before the server boots)

Run from local workstation — this catches SSH the moment it opens:

$VPS = "root@217.154.82.162"
$KEY = "C:\Users\konar\.ssh\id_ed25519"
$SSH = @("-i", $KEY, "-o", "StrictHostKeyChecking=no", "-o", "ConnectTimeout=4", "-o", "BatchMode=yes")
 
for ($i = 1; $i -le 600; $i++) {
    if ((ssh @SSH $VPS "echo up" 2>$null) -eq "up") {
        Write-Host "SSH UP at attempt $i — stopping prometheus"
        ssh @SSH $VPS "docker stop -t 3 monitoring-prometheus-1 2>&1; echo DONE; uptime"
        break
    }
    Start-Sleep -Seconds 1
}

Prometheus WAL replay is the most common OOM crasher (needs 2g during startup). Stopping it immediately gives the rest of the system time to settle.

3. Deploy corrected docker-compose.yml via SCP

scp -i C:\Users\konar\.ssh\id_ed25519 `
    C:\code_2026\p24-infra\monitoring\docker-compose.yml `
    root@217.154.82.162:/opt/p24-infra/monitoring/docker-compose.yml

4. Restart containers in groups (prevents simultaneous-startup OOM)

cd /opt/p24-infra/monitoring
 
# Group 1: Core infra
docker compose up -d caddy prometheus alertmanager loki pushgateway
sleep 20
 
# Group 2: Monitoring dashboards
docker compose up -d grafana renderer thanos-sidecar thanos-query thanos-compactor uptime-kuma
sleep 20
 
# Group 3: All exporters
docker compose up -d queue-exporter cost-exporter backup-exporter pg-stats-exporter \
  vercel-exporter blackbox-exporter promtail waha-exporter n8n-bms4-exporter \
  n8n-cloud-exporter nexcon-exporter credential-exporter mezmo-exporter mezmo-agent
sleep 15
 
# Group 4: Auxiliary services
docker compose up -d gotenberg pdf-service p24-infra-mcp audit-engine

5. Verify stability

# Load should drop below 2.0 within 2-3 minutes
uptime
 
# No containers restarting
docker ps --filter "status=restarting"
 
# Memory headroom
free -h

Memory limits — DO NOT reduce

The monitoring stack has deliberately set mem_limits. Do not lower these:

Container	Limit	Why
prometheus	2g	WAL replay at startup spikes to ~1.8GB — anything lower causes OOM crash loop
thanos-compactor	1g	Block compaction operations
grafana	512m	Plugin loading
gotenberg	512m	Chromium headless initialization
loki	512m	Log ingestion buffer

Total configured limits (~9.25GB) exceed the 8GB RAM but this is intentional — containers rarely hit their full limits simultaneously. Do not reduce prometheus below 2g.

Prevention

• All containers in monitoring/docker-compose.yml must have mem_limit set
• Check docker compose config | grep mem_limit before any deploy to verify
• After any docker-compose.yml change, test by running: docker compose config --quiet
• The .env file must exist at /opt/p24-infra/monitoring/.env — CI syncs it from SOPS via secrets-sync.yml

Escalation

If server does not come back after 2 console restarts:

1. Check IONOS console for kernel panic messages in VNC console
2. SSH into server as ubuntu (backup user) to check: journalctl -xe --no-pager | tail -50
3. Consider full OS reinstall (Ubuntu 24.04 LTS preferred) using Ansible playbook at ansible/playbooks/provision-new-vps.yml
4. Prometheus TSDB data is in Docker volume monitoring_prometheus_data — backup before reinstall: docker run --rm -v monitoring_prometheus_data:/data -v /tmp:/backup alpine tar czf /backup/prometheus-data.tar.gz /data

---

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="vps-i1",
    result="success",  # "success" | "failed" | "skipped"
    detail="vps-i1 crash loop recovery — OOM fix applied, containers restarted",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'vps-i1', 'success', 'vps-i1 crash loop recovery — OOM fix applied, containers restarted', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''