healthcheck-design

P24-Infra Health Check — Design & Operations

Status: Design (proposed) Scope: All p24-infra elements — servers, containers, SaaS, CI/CD, data integrity Related: docs/nightly-checks-triage.md, docs/alert-response-runbook.md

---

1. Principles

1. Check frequency matches failure impact. A WAHA session drop at 02:00 costs more than a doc compliance gap — check it every 2 hours, not daily.
2. One label, one queue. All machine-generated issues get p24-infra-nc-alert. The hourly triage is the single consumer; no parallel alert paths that create duplicate noise.
3. Three-tier response. Auto-fix (safe, reversible, 0-human) → AI-Dev worker (code/config change, no data risk) → Human escalation (everything else). The hourly triage classifies and routes.
4. Transition-only notifications. Discord and Telegram fire on state change (OK→FAIL, FAIL→OK), not on every run. Repeat-fire is suppressed by checking if a matching open issue already exists.
5. AI-Dev workers are opt-in per issue type. Only the explicitly whitelisted issue patterns spawn an AI-Dev worker. Unknown patterns always escalate to human.

---

2. Elements Inventory & Check Method

2.1 Servers

Element	How	Tool	Checks
vps-i1 SSH reachability	ssh -o BatchMode=yes -o ConnectTimeout=15	nightly-infra-check	exits 0 = up
vps-h1 SSH reachability	same	nightly-infra-check	—
bms-2 SSH reachability	same	nightly-infra-check	—
bms-3 SSH reachability	same	nightly-infra-check	—
bms-4 SSH reachability	same	nightly-infra-check	—
vps-i1 CPU / RAM / network	node_exporter → Prometheus → Alertmanager	continuous	HighCPU alert at 85%
vps-h1 CPU / RAM / network	node_exporter → Prometheus → Alertmanager	continuous	same
All servers disk	df -h over SSH	nightly-infra-check	≥85% → P2; ≥95% → P1
OOM kill	grep -i oom /var/log/syslog over SSH	nightly-infra-check	any match → P1
bms-1 disk	df -h over SSH	nightly-infra-check	≥85% → P2; ≥95% → P1

2.2 Docker Containers

Applies to: vps-i1 (monitoring stack), vps-h1 (WAHA + Traefik), bms-3 (staging + traccar), bms-4 (n8n + Traefik).

Check	Method	Tool	Threshold
Container state	docker ps --format '{{json .}}' over SSH	nightly-infra-check	state ∉ {running} → P2
Restart count spike	compare RestartCount to previous	nightly-infra-check	+3 in 24h → P2
CPU / memory per container	cAdvisor → Prometheus	continuous	alert rules in monitoring/alerts/

Expected running containers per host:

vps-i1: caddy, prometheus, thanos-sidecar, thanos-query, grafana, grafana-image-renderer, alertmanager, loki, promtail, blackbox-exporter, uptime-kuma, queue-exporter, cost-exporter, pg-stats-exporter, backup-exporter, gotenberg, pdf-service, traccar, traccar-db, node_exporter, cadvisor

vps-h1: traefik, waha, node_exporter, cadvisor, promtail

bms-3: nginx-proxy, Pinbox24 staging containers, traccar, mongodb

bms-4: traefik, n8n (main + 3 workers), redis, node_exporter, cadvisor

2.3 Critical Business Services

These require application-level checks beyond Docker state.

Service	Check	Endpoint / Method	Passes when	Frequency
Traccar (GPS)	HTTP GET	traccar.vps-i1…	HTTP 2xx/3xx	every 2h
WAHA (WhatsApp)	HTTP GET server status	/api/server/status + X-Api-Key	HTTP 200	every 2h
WAHA session	HTTP GET session	/api/sessions/default → parse .status	"status":"WORKING"	every 2h
n8n (bms-4)	HTTP GET health	/healthz	HTTP 200	every 2h
n8n execution errors	REST API	/api/v1/executions?status=error&limit=25 → filter last 24h	0 errors	daily 00:30
Grafana	HTTP GET health	/api/health	HTTP 200	every 2h
Infisical CE	DECOMMISSIONED 2026-06-21 — no health check needed	N/A	N/A

WAHA note: /api/server/status returns 200 even when the session is FAILED (proven 2026-05-23 blackout). Must check both the server AND the session object.

2.4 Monitoring Stack

The monitoring stack is both a subject of health checks and the platform that performs some of them. This creates a partial self-monitoring loop.

Element	Check	Method	Passes when
Prometheus scrape targets	GET /api/v1/targets	nightly-infra-check	all health=up
Prometheus firing alerts	GET /api/v1/alerts	nightly-infra-check	0 state=firing
Alertmanager liveness	HTTP	Prometheus blackbox	HTTP 200
Grafana liveness	HTTP /api/health	health-check.yml	HTTP 200
Thanos query	PromQL thanos_query_store_apis_dns_failures_total	Prometheus alert	0
Backup exporter (Wasabi freshness)	PromQL backup_age_seconds	nightly-infra-check	all < 90000s (25h)
Log shipping (promtail → Loki)	Prometheus metric promtail_sent_bytes_total	Prometheus alert	increasing

2.5 Database & Storage

Element	Check	Method	Passes when
MongoDB rs0 state	mongosh --eval 'JSON.stringify(rs.status())' via SSH to bms-2	nightly-infra-check	exactly 1 PRIMARY; all members ∈ {PRIMARY, SECONDARY, ARBITER}
MongoDB replication lag	rs.status().members[].optimeDate diff	nightly-infra-check	lag < 30s
Supabase REST	GET /rest/v1/agent_sessions?select=count&limit=0	health-check.yml	HTTP 200
Wasabi backup freshness	backup_age_seconds metric (backup-exporter)	nightly-infra-check	< 25h per backup slot
Supabase audit action failures	GET /rest/v1/actions?last_run_status=eq.failed&…	nightly-infra-check	0 rows

2.6 CI/CD — GitHub Actions Runners

Runner	Check	Method	Passes when
ionos (et-oper)	systemctl is-active actions.runner.radieu-et-operational-platform.ionos via SSH	health-check.yml	output = active
kdp-ionos	systemctl is-active actions.runner.radieu-amazon-kdp-tango.kdp-ionos-runner via SSH	health-check.yml	output = active
hstgr	pgrep -c -f Runner.Listener via SSH	health-check.yml	count ≥ 1
p24-infra-nightly cron	check last run timestamp in /var/log/p24-infra-nightly.log	nightly-infra-check	ran within last 26h

2.7 Compliance & Governance

Check	Method	Tool	Threshold
Docs compliance gaps	Supabase dev_r_services?compliance_workbook=eq.no&status=eq.active	nightly-infra-check	0 rows = clean; any rows → P3 issue per gap
Credential rotation overdue	credential-exporter → Prometheus → Alertmanager	continuous	auto_rotate=true → auto-trigger credential-rotation.yml; auto_rotate=false → human
EU AI Act registry gaps	Supabase dev_r_ai_systems rows without last_reviewed within 30d	nightly-infra-check	0 rows
Audit engine action failures	Supabase audit.actions?last_run_status=eq.failed	nightly-infra-check	0 rows

---

3. Check Frequency Matrix

Every 15s  ──► Prometheus scrape (node_exporter, cAdvisor, custom exporters)
               → fires Alertmanager rules → email P1 / daily digest P2

Every 2h   ──► health-check.yml (GH Actions, ionos runner)
               Traccar · Grafana · n8n · WAHA (server + session) · Supabase · 3 runners
               → Discord embed on state change · GH issue on failure

Every 1h   ──► hourly-devops-triage (bms-4 Linux cron)
               Reads p24-infra-nc-alert issues → auto-fix OR ai-dev-spawn OR human escalation
               → Telegram on human-required (quiet 22:00–07:00 Warsaw)

Daily 00:30 ─► nightly-infra-check (GH Actions, ionos runner)
               SSH deep checks: Docker state · MongoDB rs0 · disk · Prometheus targets
               · Prometheus alerts · n8n execution errors · Wasabi backup freshness
               · docs compliance · audit action failures
               → creates p24-infra-nc-alert issues for each problem found

Daily 23:00 ─► p24-infra-nightly (vps-i1 cron, claude-runner)
               /process-issues on dev branch — triages + implements Design-ready issues

Weekly Mon  ─► credential-rotation.yml
               Checks overdue credentials via credential-exporter metrics
               → auto-rotates auto_rotate=true credentials
               → human-action issue + Telegram for auto_rotate=false

---

4. Issue Reporting

Every machine-generated problem becomes a GitHub issue in radieu/p24-infra.

4.1 Severity levels

Level	Criteria	Response SLA
P1-Critical	Service completely down for business operations (WAHA session FAILED, MongoDB no PRIMARY, bms-1 disk ≥95%, OOM kill, Supabase unreachable)	1 hour
P2-Warning	Degraded / at-risk but not fully down (disk 85–95%, container restarting, n8n execution errors, backup stale, scrape target down)	8 hours
P3-Info	Non-urgent gap (docs compliance missing, P3 audit action failure, AI Act registry stale)	72 hours

4.2 Issue format

Title:  [nightly-check] {Component}: {short description}
Labels: bug | human-action | patch   +   p24-infra-nc-alert
Milestone: Triage
Body:
  ## Problem detected by nightly infra check

  **Component:** {component}
  **Severity:** P1-Critical | P2-Warning | P3-Info
  **Detected at:** {UTC timestamp}

  ### What was found
  {description}

  ### Raw output

{relevant snippet — no secret values}

### Suggested action
{what the agent recommends}

4.3 Notification routing per severity

Severity	Discord	Telegram	GH Issue label
P1-Critical	Immediate red embed	Immediate (if 07:00–22:00 Warsaw)	p24-infra-nc-alert + human-action
P2-Warning	Next hourly triage summary if unresolved	Only if unresolved after 4h	p24-infra-nc-alert + bug
P3-Info	None	None	p24-infra-nc-alert + patch
Clean run	None	None	Comment on [nightly-health-status] issue

4.4 Deduplication rule

Before creating a new issue the agent searches for an open issue with the same [component] title fragment and label p24-infra-nc-alert. If one exists, it adds a comment instead of creating a duplicate. Dedup key: (alertname, server-label, component) — not fingerprint-based (avoid the 2026-06-08 duplicate storm).

---

5. Reaction Model — Three Tiers

The hourly triage agent is the single routing brain. It reads all open p24-infra-nc-alert issues and decides the tier.

p24-infra-nc-alert issue
         │
         ▼
  ┌─ Tier 1: Auto-fix ──────────────────────────────────────────┐
  │  Safe, reversible, known exact action                        │
  │  Examples: docker restart (container not Dead), Prometheus   │
  │  config reload, label patch for doc gap                      │
  │  Action: SSH → fix → verify → comment "Auto-fixed" + move   │
  │  issue to In Progress                                        │
  └──────────────────────────────────────────────────────────────┘
         │ not auto-fixable
         ▼
  ┌─ Tier 2: AI-Dev worker ─────────────────────────────────────┐
  │  Requires code/config change, no data-loss risk, no prod    │
  │  auth change, no secret handling                             │
  │  Examples: missing workbook doc, n8n workflow config error,  │
  │  failed audit action (workbook spec update), Prometheus     │
  │  rule misconfiguration, broken GH Actions workflow          │
  │  Action: label ai-dev-queued → SSH to AI-Dev node →         │
  │  spawn worker-issue.md → worker opens PR → issue In Progress│
  └──────────────────────────────────────────────────────────────┘
         │ not ai-dev-solvable
         ▼
  ┌─ Tier 3: Human escalation ──────────────────────────────────┐
  │  Always: MongoDB, data loss risk, credential rotation,       │
  │  production gateway restart, disk ≥95%, OOM, bms-1 disk,   │
  │  P1-Critical, unknown pattern                                │
  │  Action: label human-action → Telegram → Discord            │
  └──────────────────────────────────────────────────────────────┘

5.1 Auto-fix patterns (Tier 1)

Issue title pattern	Action	Safety check
Container '…' is in state: Exited|Restarting	docker restart {container} on the labeled server; re-check state after 10s	Only if not Dead; not on bms-1 (production Pinbox24)
Prometheus.*config reload	curl -X POST http://localhost:9090/-/reload	Check HTTP 200 response
Docs compliance	Label patch, comment with link to documentation standard	No SSH needed
rotation.*reminder (auto_rotate=true)	Trigger credential-rotation.yml via gh workflow run	Only for flagged auto_rotate=true credentials

5.2 AI-Dev worker patterns (Tier 2)

Issue title pattern	Spawned on	Worker task
Docs compliance.*missing workbook	AI-Dev-IO1 (vps-i1)	Create docs/{service}-operations.md, update dev_r_services
Audit:.*failed	AI-Dev-IO1	Investigate audit action, update workbook spec
GH Actions.*failed	AI-Dev-IO1	Read workflow, diagnose, fix, PR
Prometheus.*target.*DOWN (not infrastructure)	AI-Dev-IO1	Check probe config, fix endpoint path, PR
n8n.*execution error (known workflow)	AI-Dev-BMS4-1 (bms-4)	Read n8n execution log, fix workflow node, activate

How the worker is spawned:

# On the appropriate AI-Dev node (via SSH from the triage agent)
ssh claude-runner@217.154.82.162 \
  "cd /home/claude-runner/p24-infra && \
   claude --dangerously-skip-permissions -p '/worker-issue #{issue_number}'" &

Safety gates — AI-Dev workers NEVER auto-trigger when:

• Issue contains: MongoDB, P1-Critical, disk 9x%, secret rotation, production restart, rs0, quorum, data loss, OOM
• More than 2 AI-Dev workers already running (check agent_sessions table)
• Issue is labeled human-action (already escalated)

Worker SLA: 4 hours. If the spawned worker doesn’t close the issue or open a PR within 4 hours, the next triage run escalates to Tier 3 and sends Telegram.

5.3 Human escalation patterns (Tier 3 — always)

• MongoDB rs0 state change (any member)
• P1-Critical severity in body
• Disk ≥ 90% on any server
• OOM kill detected
• WAHA session FAILED (after auto-restart attempt fails)
• bms-1 (Pinbox24 production) any issue
• Secret / credential rotation (auto_rotate=false)
• Production service requiring downtime
• Unknown issue pattern (no match in Tier 1 or Tier 2)

---

6. Current Implementation vs. This Design

What’s already implemented

Component	Status
health-check.yml — HTTP + systemctl every 2h	✅ Live
nightly-infra-check.yml — Claude agent 00:30 UTC	✅ Live (GH Actions)
hourly-devops-triage — bms-4 Linux cron	✅ Live
p24-infra-nightly — 23:00 UTC process-issues	✅ Live
Prometheus + Alertmanager continuous alerts	✅ Live
Discord on health-check failure	✅ Live
Telegram on human-required triage	✅ Live
Dedup by (alertname, instance, credential)	✅ Fixed 2026-06-08

What’s missing / gaps

Gap	Priority	Status	Note
Tier 2: AI-Dev worker spawning in hourly triage	P2	✅ Done	Added $AI_DEV_PATTERNS, Invoke-AIDevWorker, SLA watchdog to hourly-devops-triage.md
Worker SLA watchdog (4h timeout → Telegram)	P2	✅ Done	Phase 1 of triage now checks ai-dev-queued issues older than 4h
bms-1 disk/Docker/OOM check	P1	✅ Done	Added bms-1 (root@94.23.26.113) to $SERVERS in nightly-infra-check.md; disk at 75%
P1-Critical bypasses Telegram quiet hours	P1	✅ Done	Send-TelegramAlert now accepts $severity; P1 skips the quiet-hours guard
health-check.yml IONOS runner blind spot	Design risk	✅ Done	Split into health-http (hstgr runner) + health-runners (ionos), each self-reporting; no paid runners
WAHA auto-restart attempt before escalation	P2	Open	docker compose up -d waha if container state ≠ running; only escalate if still down
Transition-only Discord (fires on every failure, not only on state change)	P3	Open	Spec 06 improvement doc
bms-4 runner check in health-check.yml	P2	Open	No check on AI-Dev-BMS4-1 runner yet
Cert expiry check	P3	Open	docs/improvements/12-cert-expiry-alerts.md
Nightly check: p24-infra-nightly last-run timestamp	P3	Open	Add to nightly check Phase list

---

7. Complete Check Ownership Table

Element	Checked by	How	Frequency	Creates issue?
vps-i1 SSH	nightly-infra-check	ssh BatchMode	00:30 UTC	yes if fails
vps-h1 SSH	nightly-infra-check	ssh BatchMode	00:30 UTC	yes if fails
bms-2 SSH	nightly-infra-check	ssh BatchMode	00:30 UTC	yes if fails
bms-3 SSH	nightly-infra-check	ssh BatchMode	00:30 UTC	yes if fails
bms-4 SSH	nightly-infra-check	ssh BatchMode	00:30 UTC	yes if fails
vps-i1 disk	nightly-infra-check	df -h via SSH	00:30 UTC	≥85% → P2, ≥95% → P1
vps-h1 disk	nightly-infra-check	df -h via SSH	00:30 UTC	same
bms-2 disk	nightly-infra-check	df -h via SSH	00:30 UTC	same
bms-3 disk	nightly-infra-check	df -h via SSH	00:30 UTC	same
bms-4 disk	nightly-infra-check	df -h via SSH	00:30 UTC	same
bms-1 disk	nightly-infra-check	df -h via SSH	00:30 UTC	75% — monitored
OOM kills	nightly-infra-check	grep syslog via SSH	00:30 UTC	P1
vps-i1 CPU/RAM	Prometheus + Alertmanager	node_exporter	15s	Alertmanager email
vps-h1 CPU/RAM	Prometheus + Alertmanager	node_exporter	15s	Alertmanager email
Docker containers (vps-i1)	nightly-infra-check	docker ps JSON	00:30 UTC	state anomaly → P2
Docker containers (vps-h1)	nightly-infra-check	docker ps JSON	00:30 UTC	same
Docker containers (bms-3)	nightly-infra-check	docker ps JSON	00:30 UTC	same
Docker containers (bms-4)	nightly-infra-check	docker ps JSON	00:30 UTC	same
Traccar HTTP	health-check.yml	curl	every 2h	GH issue + Discord
WAHA server status	health-check.yml	curl /api/server/status	every 2h	GH issue + Discord
WAHA session WORKING	health-check.yml	curl /api/sessions/default	every 2h	GH issue + Discord
Grafana /api/health	health-check.yml	curl	every 2h	GH issue + Discord
n8n /healthz (bms-4)	health-check.yml	curl	every 2h	GH issue + Discord
Supabase REST	health-check.yml	curl	every 2h	GH issue + Discord
GH runner ionos-et	health-check.yml	systemctl via SSH	every 2h	GH issue + Discord
GH runner kdp-ionos	health-check.yml	systemctl via SSH	every 2h	GH issue + Discord
GH runner hstgr	health-check.yml	pgrep via SSH	every 2h	GH issue + Discord
MongoDB rs0 state	nightly-infra-check	mongosh rs.status()	00:30 UTC	any bad state → P1
MongoDB replication lag	nightly-infra-check	optimeDate diff	00:30 UTC	>30s → P2
Prometheus targets	nightly-infra-check	/api/v1/targets	00:30 UTC	any down → P2
Prometheus firing alerts	nightly-infra-check	/api/v1/alerts	00:30 UTC	per alert severity
n8n execution errors	nightly-infra-check	n8n REST API	00:30 UTC	any errors → P2
Wasabi backup freshness	nightly-infra-check	backup_age_seconds metric	00:30 UTC	>25h → P2
Infisical CE health	REMOVED — Infisical CE decommissioned 2026-06-21	N/A	N/A	N/A
Docs compliance	nightly-infra-check	Supabase dev_r_services	00:30 UTC	any gap → P3
Audit action failures	nightly-infra-check	Supabase audit.actions	00:30 UTC	any failed → P2
Credential rotation	credential-rotation.yml	credential-exporter	weekly Mon	auto or human-action
EU AI Act registry	nightly-infra-check	Supabase dev_r_ai_systems	00:30 UTC	stale review → P3
p24-infra-nightly last run	missing — add to nightly	tail /var/log/p24-infra-nightly.log	00:30 UTC	if >26h old → P2

---

8. Blind Spots & Risk Mitigations

Risk	Impact	Mitigation
IONOS runner down → health-check.yml blind	P1: WAHA/Traccar check fails silently	Move HTTP-only steps to GitHub-hosted runner; keep SSH steps on ionos. Tracked in spec 06.
nightly-infra-check itself fails (Claude crash)	P2: No issues created, false OK	The GH Actions failure step creates a meta-issue + Discord.
health-check.yml creates duplicate issues on every 2h failure	Noise	Add transition guard: only create new issue if no open server-down issue exists (already partially implemented; verify dedup covers all steps).
bms-1 not in nightly check targets	P1 missed: disk at 75% and rising	Fixed: added bms-1 to $SERVERS array in nightly-infra-check.md.
AI-Dev worker hangs or produces broken PR	In Progress issue stuck	SLA watchdog (4h) added to hourly triage: if ai-dev-queued issue has no PR linked after 4h, escalate to Tier 3.
Telegram quiet hours (22:00–07:00 Warsaw) suppress P1	Overnight P1 goes unnoticed until morning	Quiet hours exempt P1-Critical — send regardless of hour. Only P2/P3 respect quiet hours.

---

9. Reference: Label & Milestone Semantics

Label	Meaning in health-check context
p24-infra-nc-alert	Created by an automated check — processed by hourly triage
human-action	Triage classified as Tier 3 — Telegram sent, waiting for human
ai-dev-queued	Triage classified as Tier 2 — AI-Dev worker spawned
bug	P1 or P2 issue (fixable defect)
patch	P3 issue (docs, compliance)
server-down	Created by health-check.yml on HTTP/systemctl failure
infra-check-fail	Added to existing server-down issue on repeat failure
resolved	Added when auto-closed (service recovered)

Milestone	Pipeline stage
Triage	Freshly created — not yet assessed
In Progress	Being worked (auto-fix or AI-Dev worker active)
Review	PR open, awaiting review/merge

---

10. Related Documents

• nightly-checks-triage.md — agent skill specs + SSH access table
• alert-response-runbook.md — per-alert human reaction matrix
• 06-consolidate-health-checks.md — spec to move HTTP probes off ionos runner
• password-rotation-procedures.md — credential rotation procedures
• elements.md — full element registry (source of truth: dev_r_services)