p24-infra Docs

vps-i1-oom-outage

3 min read

Playbook: vps-i1 OOM Outage

What triggers this

A process running under claude-runner (UID 1000) on vps-i1 consumes all available RAM, causing the OOM killer to fire, which destabilizes Docker networking (veth storm in kernel log). Most commonly triggered by a Claude Code session invoked via SSH from n8n (BrandPilot AI workflow) that spawns a large Python process for brand content generation.

How to confirm

# Kernel OOM log
ssh root@217.154.82.162 "dmesg | grep -E 'oom|killed' | tail -20"
 
# Docker networking storm (many veth interfaces)
ssh root@217.154.82.162 "ip link | grep -c veth"
 
# Containers down
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose ps --format 'table {{.Name}}\t{{.Status}}'"

Step-by-step fix

  1. SSH in (usually still possible after OOM clears, Docker storm may prevent it briefly — wait 60s):

    ssh root@217.154.82.162

  2. Stop runners to prevent new Claude sessions spawning while recovering:

    systemctl stop actions.runner.radieu-p24-infra.ionos-localhost-1
systemctl stop actions.runner.radieu-p24-infra.ionos-localhost-2

  3. Bring monitoring stack back up:

    cd /opt/p24-infra/monitoring && docker compose up -d
docker compose ps

  4. Verify all containers healthy (caddy, grafana, prometheus, alertmanager, exporters).

  5. Start runners again:

    systemctl start actions.runner.radieu-p24-infra.ionos-localhost-1
systemctl start actions.runner.radieu-p24-infra.ionos-localhost-2

5b. Verify the sops-credential-liveness cron survived (#5066) — see docs/playbooks/vps-i1-outage.md step 7 / docs/playbooks/sops-credential-liveness.md §Escalation for the check command and fix. An OOM-triggered reboot is exactly the kind of event that has silently dropped this cron entry before without affecting other crons.

  1. Check memory headroom: 
    free -h
ps aux --sort=-%mem | head -10

Escalation path

If SSH is completely unreachable:

  • IONOS control panel → VNC console → log in as root
  • If the host is frozen: IONOS control panel → Reboot
  • After reboot: repeat step 3 (monitoring stack) and step 5 (runners)

Prevention

  • Runners are capped at 2GB (MemoryMax=2G in systemd unit) — if a job exceeds this, the cgroup kills the job, not the host
  • BrandPilot AI workflow now runs on bms-4 (32GB RAM) instead of vps-i1, eliminating the main trigger
  • Claude default model on vps-i1 should be Sonnet (not Opus) — Opus sessions use 2–3× more memory. Check: su -s /bin/bash claude-runner -c 'cat ~/.claude.json | grep model'

Key files

  • Runner systemd unit: /etc/systemd/system/actions.runner.radieu-p24-infra.ionos-localhost-2.service
  • Monitoring compose: /opt/p24-infra/monitoring/docker-compose.yml

Incident log

  • 2026-06-22 ~21:10 UTC: claude-runner python3 PID 501374, 5.9GB RSS, OOM killed. Trigger: BrandPilot brand wizard SSHing into vps-i1 and running Claude (Opus) for ~30s of parallel brand generation. Recovery: 15 min, all containers restored. Mitigations applied: 2GB runner cap, moved BrandPilot AI to bms-4.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="vps-i1",
    result="success",  # "success" | "failed" | "skipped"
    detail="vps-i1 OOM outage resolved — memory limits adjusted, monitoring stack restarted",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'vps-i1', 'success', 'vps-i1 OOM outage resolved — memory limits adjusted, monitoring stack restarted', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks