Playbook: OOM Auto-Remediation

What triggers this

The Linux OOM killer fires when available memory drops to zero. Kernel log: oom-kill: (decision), Out of memory: Killed process (kill).

Alert: OOMKillDetected in Alertmanager (rule in monitoring/prometheus/rules/infrastructure.yml). Pre-warning: LowMemoryAbsolute fires when free memory < 500 MB — act before OOM hits.

Common causes:

Server	Common trigger
vps-i1 (8 GB)	Prometheus WAL replay, GH Actions runner Python (2 GB each)
bms-3 (32 GB)	MongoDB primary under heavy query load
bms-4 (32 GB)	n8n workers spawning Claude Code sessions
bms-1/2 (32 GB)	Pinbox24 memory leak, MongoDB sync

How to confirm

# 1. Check Prometheus alert (replace SERVER_IP)
curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool | grep OOMKill
 
# 2. Read kernel OOM log on the affected server
ssh root@SERVER_IP 'journalctl -k --since "1 hour ago" | grep -i "out of memory\|oom_kill\|killed process"'
 
# 3. Check current memory pressure
ssh root@SERVER_IP 'free -h && cat /proc/meminfo | grep -E "MemAvail|MemFree|Cached"'
 
# 4. Check which process was killed (last OOM event)
ssh root@SERVER_IP 'dmesg | grep -A5 "Out of memory:"  | tail -20'

Step-by-step fix

Step 1 — Identify the killed process

ssh root@SERVER_IP 'journalctl -k | grep "Killed process" | tail -5'
# Example output: "Killed process 12345 (prometheus) total-vm:4194304kB..."

Note the process name. This is the victim (may not be the cause — OOM kills the biggest process first).

Step 2 — Find the memory hog (actual cause)

# Current top memory consumers
ssh root@SERVER_IP 'ps aux --sort=-%mem | head -20'
 
# For Docker hosts: which containers are using most RAM
ssh root@SERVER_IP 'docker stats --no-stream --format "table {{.Name}}\t{{.MemUsage}}\t{{.MemPerc}}"'
 
# Historical: what was running before the OOM (last 2h journal)
ssh root@SERVER_IP 'journalctl --since "2 hours ago" | grep -E "memory|OOM|kill" | tail -50'

Step 3 — Temporary relief (do this first, investigate after)

# If a known safe container is bloated, restart it (containers auto-restart via Docker)
ssh root@SERVER_IP 'docker restart CONTAINER_NAME'
 
# If a GH Actions runner process is the cause on vps-i1:
ssh root@SERVER_IP 'pkill -f "Runner.Listener" && systemctl restart actions.runner.* 2>/dev/null || true'
 
# Drop OS caches (safe, Linux recovers them on demand)
ssh root@SERVER_IP 'sync && echo 3 > /proc/sys/vm/drop_caches'
 
# Verify memory freed
ssh root@SERVER_IP 'free -h'

Step 4 — Investigate root cause from Mezmo logs

Query in Mezmo: host:SERVER_NAME level:error for the 30 min before the OOM timestamp. Look for: memory allocation errors, container startup storms, Python subprocess spawning.

# Pull Mezmo logs via CLI (replace DATE with ISO timestamp)
sops exec-env secrets/monitoring.env.sops \
  'python scripts/mezmo-manage.py export --from DATE --to DATE'

Step 5 — Permanent fix (choose based on root cause)

Root cause	Fix
Container has no `mem_limit`	Add `mem_limit: XYZm` to docker-compose service
Prometheus WAL replay	Reduce `--storage.tsdb.retention.time` or add `mem_limit: 2g`
n8n worker spawning too many Claude sessions	Reduce `N8N_CONCURRENCY_PRODUCTION` or add worker mem_limit
GH Actions runner Python	Add swap (`fallocate -l 2G /swapfile && mkswap /swapfile && swapon /swapfile`)
MongoDB query spike on bms-3	Check slow query log; add index or rate-limit the query
Pinbox24 leak on bms-1	Restart the leaking container; escalate to Pinbox24 dev team

After fix, deploy:

ssh root@SERVER_IP 'cd /opt/p24-infra/STACK_DIR && docker compose up -d'

Step 6 — Verify recovery

# Memory looks healthy
ssh root@SERVER_IP 'free -h'
 
# No OOM in last 10 min
ssh root@SERVER_IP 'journalctl -k --since "10 min ago" | grep -i "out of memory" | wc -l'
# Must return 0
 
# Prometheus alert cleared (wait ~5 min for scrape)
curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool | grep OOMKill
# Must return nothing

Escalation path

Iteration 1 (immediate): Apply temporary relief (Step 3). Document what was killed. Iteration 2 (within 1 hour): Identify root cause and apply permanent fix (Step 5). Iteration 3 (if OOM recurs after fix): Create a GitHub issue with label human-action, bug:

gh issue create --repo radieu/p24-infra \
  --title "[OOM] Recurring OOM on SERVER_NAME — 3 remediation iterations failed" \
  --label "bug,human-action" \
  --milestone "Triage" \
  --body "## OOM escalation\n\nServer: SERVER_NAME\nFirst event: TIMESTAMP\nKilled process: PROCESS_NAME\nFix attempts: describe what was tried\n\n## Required action\nHuman investigation needed — automated remediation exhausted."

Send Discord alert:

curl -s -X POST "$P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL" \
  -H "Content-Type: application/json" \
  -d '{"embeds":[{"title":"OOM ESCALATION — SERVER_NAME","color":15158332,"description":"3 remediation attempts failed. Human intervention required. See GH issue."}]}'

Post-reboot checklist (after OOM + hard reboot)

A hard reboot can leave containers down even with restart: unless-stopped — that policy only re-runs containers that were running when the daemon stopped. Containers that had been cleanly stopped (Exited (0)), or stacks not wired to a boot unit, do not come back. After any hard reboot, verify every stack is up. (Ref: #1313.)

# 1. All expected containers running on vps-i1?
ssh root@217.154.82.162 'docker ps --format "table {{.Names}}\t{{.Status}}"'
# Expect: monitoring-* stack, node_exporter, traccar, traccar-db all "Up"

If a stack is down, bring it up — both vps-i1 stacks are now wired to boot units:

Stack	Boot unit	Manual recovery
monitoring (`/opt/p24-infra/monitoring`)	`monitoring-stack.service`	`cd /opt/p24-infra/monitoring && docker compose up -d`
node_exporter	part of monitoring stack (`monitoring/docker-compose.yml`, `node-exporter` service)	`cd /opt/p24-infra/monitoring && docker compose up -d node-exporter`
traccar (`/root/traccar`)	`traccar.service`	`cd /root/traccar && docker compose up -d`

# 2. Confirm the boot units are enabled (survive future reboots)
ssh root@217.154.82.162 'systemctl is-enabled monitoring-stack traccar'
# Both must return "enabled"

Installing / repairing the traccar boot unit (one-time, source: services/traccar/traccar.service):

scp services/traccar/traccar.service root@217.154.82.162:/etc/systemd/system/traccar.service
ssh root@217.154.82.162 'systemctl daemon-reload && systemctl enable --now traccar'
ssh root@217.154.82.162 'systemctl status traccar --no-pager'

Test (simulate reboot without rebooting):

# Restarting the docker daemon re-applies restart policies + re-triggers oneshot boot units
ssh root@217.154.82.162 'systemctl restart docker && sleep 10 && docker ps --format "{{.Names}}: {{.Status}}"'
# node_exporter + traccar + traccar-db must all be "Up"

Note — node_exporter on vps-i1: previously ran as a standalone docker run container (exited 143 after the 2026-06-24 hard reset and did not return). It is now defined in monitoring/docker-compose.yml (node-exporter service, network_mode: host, container_name: node_exporter) so the monitoring stack owns its lifecycle. If a stray standalone node_exporter container or an node_exporter.service systemd unit is also present, remove/disable it first to avoid a port-9100 conflict: docker rm -f node_exporter; systemctl disable --now node_exporter 2>/dev/null || true then docker compose up -d node-exporter.

Prevention

Set mem_limit on every Docker container — see R7 in docs/standards/server-monitoring-standard.md
Add swap on vps-i1 (8 GB only) — 2 GB swap buys time for alert-to-response
Monitor LowMemoryAbsolute alert — fires at 500 MB free, ~2 min before OOM
Keep Mezmo enabled on all servers (R1) — needed for post-OOM forensics
Wire every stack to a boot unit — restart: unless-stopped is not enough after a hard reboot; ensure each compose stack has an enabled systemd unit (monitoring-stack.service, traccar.service) so it auto-starts on boot

Playbook reference: docs/standards/server-monitoring-standard.md §R4, R6

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="oom-target",
    result="success",  # "success" | "failed" | "skipped"
    detail="OOM auto-remediation executed — offending process killed, service restarted",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'oom-target', 'success', 'OOM auto-remediation executed — offending process killed, service restarted', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

oom-auto-remediation