p24-infra Docs

resource-incident-monitoring

4 min read

Playbook — Mezmo-correlated resource incident analysis + Claude auto-triage

Issue: #1191 · Prerequisites: #1467 · Status: code merged, activation gated on #1467

A fast-path incident-response loop for resource pressure (CPU > 80% or RAM > 85% for 2 min). It correlates the metric spike with Mezmo logs (the 7-day contextual layer) for the affected server, records an incident, opens a GitHub issue, and dispatches a Claude agent to investigate live and react within documented playbook bounds.

This does not replace the existing HighCPU / HighMemory warning-digest path, nor alertmanager-escalation.py. It is an additive, faster, log-aware lane.

Data flow

Prometheus rule (ResourceIncidentCPU / ResourceIncidentRAM, category=resource-incident)
  → Alertmanager route `category=resource-incident` (group_wait 0s, first route — no email)
  → n8n webhook  https://n8n.bms-4.infra.zintegrowana.online/webhook/resource-incident
       1. Map instance IP → server label + Mezmo host
       2. Mezmo /v2/export  query=host:<label>  window=startsAt-5m → now
       3. Parse apps / error counts / sample ERROR lines → Markdown report
       4. Upsert dev_r_incidents (source=prometheus-resource-incident)
       5. Create GitHub issue (label: resource-incident)
       6. Dispatch resource-incident-triage.yml + Discord embed
  → GH Actions `resource-incident-triage.yml` (runs-on [self-hosted, bms4])
       Claude SSHes to the server (if reachable), correlates, takes playbook-bounded
       action, comments on the issue, closes or escalates (human-action).

Components (all in this repo)

ComponentFile
Prometheus rulesmonitoring/prometheus/rules/infrastructure.yml → group resource_incidents
Alertmanager route + receivermonitoring/alertmanager/alertmanager.yml.tpl → route/receiver resource-incident
n8n workflowinfra-src/n8n-workflows/resource-incident-analysis.json
GH Actions triage.github/workflows/resource-incident-triage.yml
Manual correlation CLIscripts/mezmo-manage.py query --server <label> --from <ts> --to <ts>
Incident storeSupabase dev_r_incidents (already exists, RLS enabled)

Trigger

A ResourceIncidentCPU or ResourceIncidentRAM alert fires (CPU > 80% / RAM > 85% for 2m), producing a new GitHub issue labelled resource-incident and a dev_r_incidents row.

Confirmation

# 1. Alert is firing in Prometheus
ssh root@217.154.82.162 "curl -s localhost:9090/api/v1/alerts | jq '.data.alerts[] | select(.labels.category==\"resource-incident\")'"
 
# 2. Alertmanager routed it (not to email)
ssh root@217.154.82.162 "curl -s localhost:9093/api/v2/alerts | jq '.[] | select(.labels.category==\"resource-incident\")'"
 
# 3. n8n received the webhook — check the resource-incident-analysis execution log in n8n UI
#    (https://n8n.bms-4.infra.zintegrowana.online → Executions)
 
# 4. Incident row recorded
#    SELECT * FROM dev_r_incidents WHERE source='prometheus-resource-incident' ORDER BY created_at DESC LIMIT 5;
 
# 5. Manual Mezmo correlation for the same window (epoch ms or ISO-8601 timestamps):
sops exec-env secrets/monitoring.env.sops \
  'python scripts/mezmo-manage.py query --server vps-i1 --from 2026-06-26T00:00:00Z --to 2026-06-26T00:10:00Z'

Runbook (what the Claude triage agent does — and what a human does manually)

  1. Read the Mezmo correlation report on the issue (top apps by log/error volume).
  2. If the server is SSH-reachable (bms-4 itself, vps-i1 via ssh root@217.154.82.162): docker stats --no-stream, ps aux --sort=-%mem | head, free -m, df -h.
  3. Correlate the spike with the log evidence; identify the offender.
  4. React only within an existing playbook (e.g. oom-auto-remediation.md, vps-h1-cpu-throttle.md): truncate an oversized container log, restart the confirmed non-critical offender, etc.
  5. Comment on the issue with cause / evidence / action / state; close if resolved.

Escalation

Add the human-action label and do not act when:

  • The cause is outside any documented playbook.
  • The offender is production-critical: WAHA (vps-h1), MongoDB (bms-2/bms-3), Pinbox24 (bms-1/bms-3).
  • The server is unreachable by SSH from bms-4 (vps-h1, bms-2, bms-3) and the metric is still critical — Mezmo-only investigation, then escalate.

Prevention

  • Right-size container memory limits (see docs/playbooks/oom-auto-remediation.md).
  • Keep Mezmo ingestion healthy on every server (#1467) so correlation has data; a server with no logs degrades to SSH-only investigation.
  • Tune the 2-minute for: window up if synthetic/transient spikes cause noise.

Activation (after prerequisites #1467 are green)

Editing live alerting is production-touching. Validate before deploying.

# Validate rules and the rendered Alertmanager config BEFORE deploy:
promtool check rules monitoring/prometheus/rules/infrastructure.yml
# render the template with envsubst (or the deploy script), then:
amtool check-config /path/to/rendered/alertmanager.yml
 
# Deploy on vps-i1:
ssh root@217.154.82.162 "cd /opt/p24-infra && git pull && cd monitoring && \
  docker compose up -d alertmanager && curl -X POST localhost:9090/-/reload"
  1. Import resource-incident-analysis.json into n8n (bms-4) and activate it; confirm the p24-infra-mezmo credential is attached to the Mezmo query node and that SUPABASE_SERVICE_KEY, P24_INFRA_GH_TOKEN, DISCORD_WEBHOOK_URL are set in the n8n env.
  2. Confirm the resource-incident label exists (gh label list).
  3. Reload Prometheus + restart Alertmanager (commands above).
  4. Synthetic test: stress-ng --cpu 0 --timeout 180s on a non-critical host → confirm the issue is created and the triage workflow runs.

Compliance

The Claude auto-triage step is an AI-driven system. It is registered in dev_r_ai_systems and docs/eu-ai-act-compliance.md (limited-risk: assists infra ops, no employment/scoring decisions; bounded to documented playbook actions with human-action escalation).

Graph View

Backlinks