Playbook: vps-i1 Completely Unreachable

Trigger

vps-i1 (217.154.82.162, IONOS) stops responding to SSH, ICMP, and HTTPS from both local workstation and external servers (bms-4).

Symptoms: Prometheus fires EndpointDown for grafana, prometheus, alertmanager, traccar — all 4 endpoints simultaneously. GitHub Actions runners on vps-i1 fail with “workflow file issue” (runner unreachable).

How to Confirm

# From local workstation
Test-NetConnection -ComputerName 217.154.82.162 -Port 22 -InformationLevel Detailed
# expect: TcpTestSucceeded=False, PingSucceeded=False
 
# From bms-4 (external verification — rules out local network issue)
ssh root@54.36.123.110 "ping -c 3 217.154.82.162"
# expect: 100% packet loss
 
# Web services
curl --max-time 10 https://grafana.vps-i1.infra.zintegrowana.online
# expect: timeout

Step-by-Step Fix

Log in to IONOS Control Panel — https://my.ionos.com → VPS → vps-i1
Check power state:
- If Stopped or Crashed → click Restart
- If Running but unreachable → click Reset (hard reboot)
Open KVM Console in IONOS panel if restart doesn’t help — check for:
- Kernel panic (filesystem full, OOM killer, bad update)
- Boot failure (grub error, broken initrd)
- Network service not starting

After SSH is restored — run the full recovery in one command:

ssh root@217.154.82.162 "
  echo '=== Docker stack ===' && \
  cd /opt/p24-infra/monitoring && \
  docker compose up -d && \
  sleep 10 && \
  docker compose ps && \
  echo '=== Local endpoint checks ===' && \
  curl -s -o /dev/null -w 'grafana: %{http_code}\n' http://localhost:3000/api/health && \
  curl -s -o /dev/null -w 'prometheus: %{http_code}\n' http://localhost:9090/-/healthy && \
  curl -s -o /dev/null -w 'alertmanager: %{http_code}\n' http://localhost:9093/-/healthy && \
  curl -s -o /dev/null -w 'traccar: %{http_code}\n' http://localhost:8082/api/server && \
  echo '=== Memory ===' && \
  free -h && \
  echo '=== Disk ===' && \
  df -h / /var
"

Check for OOM/kernel issues (if restart was required):

ssh root@217.154.82.162 "dmesg | grep -E 'oom|killed|panic' | tail -20"

Verify all 4 endpoints externally (expected: all 200):

foreach ($ep in @(
  "https://grafana.vps-i1.infra.zintegrowana.online/api/health",
  "https://prometheus.vps-i1.infra.zintegrowana.online/-/healthy",
  "https://alertmanager.vps-i1.infra.zintegrowana.online/-/healthy",
  "https://traccar.vps-i1.infra.zintegrowana.online/api/server"
)) {
  $code = (Invoke-WebRequest -Uri $ep -UseBasicParsing -TimeoutSec 10 -ErrorAction SilentlyContinue).StatusCode
  Write-Host "$ep : $code"
}

Verify the sops-credential-liveness cron survived the reset (#5066) — a reboot/reset can silently drop this ansible-managed crontab entry (and the out-of-band claude-runner age key it depends on) without dropping the rest of the crontab, and nothing re-applies the vps-i1 playbook automatically after a recovery:
```
ssh root@217.154.82.162 "crontab -l | grep -A1 sops-credential-liveness; test -f /home/claude-runner/.age/p24-infra-keys.txt && echo AGE_KEY_OK || echo AGE_KEY_MISSING"
```
If either is missing, see docs/playbooks/sops-credential-liveness.md §Escalation (SopsCredentialLivenessStopped firing after a vps-i1 reboot/OOM-recovery incident row) — re-run ansible-playbook ansible/playbooks/vps-i1.yml --tags sops-credential-liveness or reinstate the cron entry manually, then re-run the script once to repopulate the metric immediately rather than waiting for the next scheduled run.
Close GitHub issues once all 4 return 200:
- Close all open EndpointDown issues (e.g. #1126, #1127, #1128, #1129) with a note that they were resolved by VPS restart + docker compose up -d.

Escalation Path

If restart doesn’t help: open IONOS support ticket for hardware/hypervisor issue
If disk is full: df -h after recovery — clean /var/log, Docker image cache (docker system prune -f)
If OOM: check dmesg | grep -i oom — may need to reduce container memory limits; see vps-i1-oom-outage.md
If containers won’t start after docker compose up -d: check docker compose logs --tail=100 <service>

Prevention

Container memory limits (added in #1155) — every service in monitoring/docker-compose.yml now has a mem_limit (prometheus 2g, thanos-compactor 1g, sidecar/query/grafana/renderer/loki/gotenberg 512m, alertmanager/pdf-service/p24-infra-mcp/audit-engine/uptime-kuma 256m, all exporters + caddy 128m). A runaway container is now cgroup-killed instead of taking the 8 GB host down. Root cause of the 2026-06-23 OOM crash loop: the stack had no per-container limits, so Prometheus TSDB compaction (sustained ~500 MB/s reads + CPU) plus concurrent Node CI jobs exhausted RAM → OOM kill loop. Prometheus also caps compaction I/O with --storage.tsdb.max-block-duration=2h (already set).

Optional host buffer — 2 GB swap on vps-i1 (low priority; manual root action, not in the repo):

fallocate -l 2G /swapfile && chmod 600 /swapfile && mkswap /swapfile && swapon /swapfile
echo '/swapfile none swap sw 0 0' >> /etc/fstab

After changing any limit: cd /opt/p24-infra/monitoring && docker compose up -d and confirm no container enters a restart/OOM loop (docker stats, Grafana container-memory dashboard).
Monitor disk usage with Prometheus alert HighDiskUsage (threshold 85%)
p24-status.py runs every 15 min on vps-i1 — if the script itself is down, that’s the first signal
P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL must be set in secrets/monitoring.env.sops (was empty — fixed in PR #1066)
External uptime monitoring — IMPLEMENTED (2026-08-03, #5199) via scripts/check-vps-i1-uptime.sh, run by cron every 5 min on bms-4 (checks grafana.vps-i1.infra.zintegrowana.online/api/health → Discord alert + GH issue on the clean→fail edge, recovery notification on fail→clean). This closes the catch-22 described above: the check runs from a host that survives a vps-i1 outage. Originally this Prevention bullet suggested UptimeRobot (third-party SaaS); we chose a self-hosted check on bms-4 instead because (1) it needs no new external account or credential, (2) it reuses the Discord+GH-issue alerting pattern already proven by scripts/check-p24-auth-worker.sh, and (3) bms-4 (OVH) is a genuinely independent vantage point from vps-i1 (IONOS) — a different physical provider, not just a different VM on the same one.
GH Actions health-check workflow (health-check.yml) fails when vps-i1 runners are offline — this also serves as an indirect signal

Incident Log

Date	Issues	Duration	Root cause	Fix
2026-06-22 ~21:00 UTC	#1065	~15 min	OOM: claude-runner python3 5.9 GB RAM	VPS restart + 2 GB runner cap; BrandPilot moved to bms-4
2026-06-23 ~10:00 UTC	#1126–#1129	TBD	VPS fully offline (cause TBD after restart)	Pending human action: IONOS control panel restart
2026-06-23 (OOM crash loop)	#1155	—	No per-container memory limits → Prometheus compaction + Node CI jobs exhausted 8 GB RAM → OOM kill loop	Added `mem_limit` to all monitoring containers; see Prevention
2026-08-03	#5199	~unknown (host down at session start, resolved via IONOS Control Panel Reset)	Host/hypervisor-level unreachable — KVM console before reset showed rapid Docker veth churn consistent with a container crash-loop spiking CPU (same pattern as #1155), but dmesg ring buffer was lost on reboot so the specific culprit container couldn’t be confirmed	IONOS Control Panel Reset; full recovery verified (docker compose up -d, all containers healthy, all endpoints 200)

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="radieu",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="vps-i1",
    result="success",  # "success" | "failed" | "skipped"
    detail="vps-i1 outage response — IONOS control panel restart performed, stack restored",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('radieu', 'restart', 'vps-i1', 'success', 'vps-i1 outage response — IONOS control panel restart performed, stack restored', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

vps-i1-outage