p24-infra Docs

vps-h1-cpu-throttle

3 min read

Playbook: vps-h1 CPU Throttle Risk

Trigger: Alert VpsH1ThrottleRisk — CPU >80% sustained 45+ min
Host: root@72.60.32.61 (Hostinger vps-h1, Ubuntu 24.04)
Time budget: ~135 min before Hostinger throttle activates (throttle = after 180 min sustained high CPU)
Throttle reset: manual, once per week via Hostinger panel — avoid at all cost

Step 1 — Measure current state

ssh root@72.60.32.61 "top -bn2 -d2 | grep '%Cpu' | tail -1"
  • idle >80% → false alarm, close issue
  • idle <20% → proceed to Step 2

Step 2 — Identify culprit

Run all three:

# A) Which container?
ssh root@72.60.32.61 "docker stats --no-stream --format 'table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}'"
 
# B) Which system process?
ssh root@72.60.32.61 "ps aux --sort=-%cpu | head -12"
 
# C) Log file sizes?
ssh root@72.60.32.61 "find /var/lib/docker/containers -name '*-json.log' -exec du -sh {} \; | sort -rh | head -10"

Step 3 — React (autonomous, no approval needed)

Log file too large (>20MB in non-waha container)

# Identify container name
ssh root@72.60.32.61 "docker inspect --format='{{.Name}}' <container-id>"
 
# Truncate log
ssh root@72.60.32.61 "truncate -s 0 /var/lib/docker/containers/<id>/<id>-json.log"
 
# Restart that container (NOT waha)
ssh root@72.60.32.61 "cd /root && docker compose restart <service-name>"

Stuck process (not dockerd)

ssh root@72.60.32.61 "kill <pid>"
# Verify after 30s: ps aux | grep <pid>

cAdvisor or promtail CPU >5%

ssh root@72.60.32.61 "cd /root && docker compose restart cadvisor"
# or
ssh root@72.60.32.61 "cd /root && docker compose restart promtail"

Step 4 — Escalate to human (create GitHub issue)

Create a human-action issue when:

ConditionAction
dockerd uptime >30 days AND CPU >80%Request systemctl restart docker approval
CPU >80% with no identifiable causeRequest manual investigation
Alert VpsH1ThrottleImminent fires (2h)Immediate docker restart, notify via Discord
gh issue create --repo radieu/p24-infra \
  --title "🔴 [vps-h1] Docker restart needed — CPU throttle imminent" \
  --label "human-action,bug" \
  --body "## Situation
vps-h1 CPU sustained >80% for 2h. Hostinger throttle will activate within ~20 min.
 
## Required action
\`\`\`
ssh root@72.60.32.61 systemctl restart docker
\`\`\`
30 second WAHA downtime. WhatsApp session persists (volume).
 
## Diagnosis
$(docker stats --no-stream)
$(ps aux --sort=-%cpu | head -8)"

Step 5 — Verify resolution

ssh root@72.60.32.61 "top -bn2 -d2 | grep '%Cpu' | tail -1"
  • idle >80% → resolved, close issue with comment
  • idle <20% → escalate immediately

Known causes (historical)

DateCauseFix
2026-06-22cAdvisor 63MB logs + dockerd 87d uptime + healthcheck exec every 30struncate logs + disable healthcheck + restart dockerd

Prevention (already applied)

  • cAdvisor housekeeping_interval=60s (was 10s)
  • cAdvisor healthcheck disabled (Prometheus monitors :8080 directly)
  • Docker log rotation: max-size=10m, max-file=3 on all containers
  • Monthly dockerd restart — first Saturday of month at 21:30 UTC
    • Cron: 30 21 1-7 * 6 /root/docker-restart-monthly.sh
    • Script: /root/docker-restart-monthly.sh (repo: hostinger/scripts/docker-restart-monthly.sh)
    • WAHA downtime: ~30s, session preserved in volume
    • Discord notification on start + completion/failure

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="vps-h1",
    result="success",  # "success" | "failed" | "skipped"
    detail="vps-h1 CPU throttle remediated — resource limits adjusted",
    env="vps-h1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'vps-h1', 'success', 'vps-h1 CPU throttle remediated — resource limits adjusted', 'vps-h1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks