Health Check Observability

How often each check runs, how long it takes, what resources it uses, what is exposed in Prometheus/Grafana, and what happens automatically on failure.

---

1. Check Duration & Resource Budget

health-check.yml — every 2 hours (GH Actions)

Runs as two parallel jobs on separate runners.

Job	Runner	Steps	Typical duration	GH Actions minutes
health-http	hstgr (self-hosted, vps-h1)	5 curl checks + backstop + report	~1–2 min	free
health-runners	ionos (self-hosted, vps-i1)	3 SSH checks + report	~1 min	free

Total per run: ~2 min elapsed, 0 billed minutes (both jobs on free self-hosted runners). Per day: 12 runs × 0 billed minutes = free.

Resource cost per step:

• curl HTTP probe: negligible — single HTTPS request, <1 s
• WAHA: two requests (server status + session) — <2 s
• SSH systemctl is-active: <3 s per runner (SSH handshake + command)
• GH issue create/update: single API call — <1 s

nightly-infra-check.yml — 00:30 UTC daily (GH Actions, ionos runner)

Single job, Claude agent, ~25–45 min depending on server response times.

Phase	What	Typical time
Import Infisical secrets	REMOVED (Infisical CE decommissioned 2026-06-21)	N/A
Phase 1 — Docker (5 servers)	5× SSH + docker ps	~20–40 s
Phase 2 — MongoDB rs0	1× SSH + mongosh rs.status()	~10–15 s
Phase 3 — Disk (6 servers)	6× SSH + df -h	~20–40 s
Phase 4 — Prometheus targets	1× HTTPS REST	~3 s
Phase 5 — Prometheus alerts	1× HTTPS REST	~2 s
Phase 6 — n8n executions	1× HTTPS REST	~3 s
Phase 7 — Wasabi backup freshness	1× PromQL query	~3 s
Phase 8 — Infisical CE	REMOVED (decommissioned 2026-06-21)	N/A
Phase 9 — Supabase compliance	1× REST query	~3 s
Phase 10 — Audit action failures	1× REST query	~3 s
Phase 11 — Report + GH issue(s)	gh CLI calls	~5–30 s per issue
Claude agent overhead	LLM inference per phase	~15–30 s
Total		~3–8 min clean; ~10–20 min with issues

Resource cost: Uses the ionos runner (free). The Claude agent runs as the IONOS runner process — no separate VPS cost. SSH connections are stateless and closed immediately after each command.

hourly-devops-triage — every hour (bms-4 Linux cron)

Claude agent, typically 2–5 min.

Activity	Time
Fetch triage issues (gh CLI)	~3 s
SLA watchdog (gh CLI per stale issue)	~2 s each
Per issue: classify + act	~5–15 s (Tier 1 SSH fix: +10 s)
Tier 2 SSH spawn	~5 s (fire-and-forget)
Telegram API call	~1 s
Total (0 issues)	~10 s
Total (5 issues, mixed)	~1–2 min

Resource cost: Runs on bms-4 as claude-runner. Uses Claude Max OAuth — no API credit cost.

---

2. Prometheus & Grafana Exposure

What is currently scraped by Prometheus

Prometheus on vps-i1 scrapes the following targets relevant to health checks:

Target	Job	Port	Metrics
vps-i1 host	node_exporter	:9100	CPU, RAM, disk, network, load
vps-h1 host	node_exporter	:9100	same
vps-i1 containers	cadvisor	:8080	Per-container CPU/RAM/restart count
vps-h1 containers	cadvisor	:8080	same
Supabase queue depths	queue_exporter	:9200	p24_queue_depth{queue}
Supabase slow queries	pg_stats_exporter	:9201	p24_slow_query_count
Wasabi backup freshness	backup_exporter	:9220	backup_age_seconds{backup_name}
Vercel/Supabase costs	cost_exporter	:9210	p24_cost_usd{service}
Credential rotation age	credential_exporter	:9230	p24_credential_days_overdue{credential}
HTTP probes	blackbox_exporter	:9115	probe_success, probe_duration_seconds

What health-check results are NOT yet in Prometheus

The health-check.yml and nightly-infra-check results are GitHub issue-native only — they are not pushed to Prometheus. This is a gap.

Check	In Prometheus?	Gap
WAHA session status	Partial — blackbox probes HTTP, not session state	Session WORKING/FAILED not exposed
Grafana HTTP	Yes — blackbox probe	✅
n8n HTTP	Yes — blackbox probe	✅
Traccar HTTP	Yes — blackbox probe	✅
MongoDB rs0 PRIMARY count	No	Not scraped — nightly check only
GH runner status	No	SSH check only, not a metric
Disk usage (bms-1, bms-2, bms-3)	No	node_exporter only on vps-i1 and vps-h1
n8n execution error count	No	nightly check only
Nightly check pass/fail	No	GH issue only
Hourly triage: issues processed	No	stdout only

Grafana dashboards

Dashboard	What it shows	Datasource
p24-infra Health	Custom exporters: backup age, cost, queue depth, credential rotation	Thanos Query (Prometheus)
Node Exporter Full	Host CPU/RAM/disk/network for vps-i1 and vps-h1	Thanos Query
cAdvisor	Per-container resource usage	Thanos Query
Blackbox	HTTP probe success/latency per endpoint	Thanos Query

Currently no Grafana dashboard shows: MongoDB rs0 state, GH runner status, nightly check history, hourly triage activity.

---

3. What Happens on Failure — Automated Reaction Matrix

health-check.yml failure (every 2h)

health-http or health-runners job FAILS
         │
         ▼
health-report job (always runs)
         │
         ├── Discord embed: "❌ service1 service2 failed" (every run while failing)
         │
         └── GitHub issue:
               • First failure → create issue labeled server-down,triage,infra-check-fail
               • Repeat failures → comment "Still failing: …" on existing issue
               • Recovery → close issue, add label resolved

The server-down issue has label p24-infra-nc-alert — picked up by hourly triage within 1 hour.

Triage reaction:

• HTTP failures (Traccar, Grafana, n8n, WAHA) → match HUMAN_REQUIRED_PATTERNS or escalate as unknown → Tier 3: Telegram to human
• WAHA session FAILED specifically → HUMAN_REQUIRED_PATTERNS (“WAHA”) → Tier 3 immediately
• Runner down → Tier 3 immediately (production restart pattern)

nightly-infra-check.yml failure (00:30 UTC)

Nightly agent runs all 11 phases
         │
         ├── Each problem found → create GH issue labeled p24-infra-nc-alert
         │     P1-Critical → also: label human-action added by triage within 1h
         │     P2-Warning  → triage within 1h: Tier 1, 2, or 3
         │     P3-Info     → labeled patch, queued for AI-Dev or human
         │
         └── Agent itself crashes → Discord embed (red) + meta GH issue

Hourly triage reaction per tier

Tier	Trigger	Automated action	Human notified?
Tier 1 — Auto-fix	Docker container Exited/Restarting	SSH docker restart, verify, move issue to In Progress	No
Tier 1 — Auto-fix	Prometheus config reload needed	curl -X POST /-/reload, verify HTTP 200	No
Tier 1 — Auto-fix	Docs compliance gap	Label patch, add comment	No (Tier 2 may spawn)
Tier 1 — Auto-fix	Credential rotation (auto_rotate=true)	Trigger credential-rotation.yml	No
Tier 2 — AI-Dev	Docs compliance missing workbook	SSH spawn worker-issue on AI-Dev-IO1	No (SLA watchdog at 4h)
Tier 2 — AI-Dev	Audit action failed	SSH spawn worker on AI-Dev-IO1	No
Tier 2 — AI-Dev	GH Actions workflow failed	SSH spawn worker on AI-Dev-IO1	No
Tier 2 — AI-Dev	Prometheus target DOWN	SSH spawn worker on AI-Dev-IO1	No
Tier 2 SLA watchdog	ai-dev-queued > 4h, no PR	Relabel human-action	Yes — Telegram P2
Tier 3 — Human	MongoDB, OOM, disk ≥90%, P1	Label human-action	Yes — Telegram
Tier 3 — Human	Credential rotation (auto_rotate=false)	Label human-action	Yes — Telegram P2
Tier 3 — Human	Production service restart	Label human-action	Yes — Telegram
Tier 3 — Human	Unknown pattern	Label human-action	Yes — Telegram P3

Prometheus Alertmanager reaction (continuous)

Alertmanager fires independent of the above. Overlap is intentional — belt-and-braces.

Alert	Routing	Action
EndpointDown (any HTTP probe)	P1 → immediate email	Also triggers alertmanager-escalation.yml → GH issue [infra-alert] → hourly triage
HighCPU	P2 → daily digest email	GH issue
CredentialRotationCritical	P1 → immediate email	GH issue per credential
CredentialRotationOverdue	P2 → daily digest	GH issue
PromtailNotSendingLogs	P2 → daily digest	GH issue
BackupExporterDown	Suppressed (known flap)	Only fires after 1h

---

4. Full Failure → Resolution Flow

Problem occurs (e.g. WAHA session drops at 03:00 UTC)
         │
         ├── [Continuous] Prometheus blackbox probe fails → Alertmanager fires
         │   → email to radieu@gmail.com (P1 immediate)
         │   → alertmanager-escalation.yml → [infra-alert] GH issue
         │
         ├── [Every 2h] health-check.yml detects WAHA FAIL
         │   → Discord embed ❌
         │   → Creates/updates server-down GH issue with label p24-infra-nc-alert
         │
         ├── [Next hour] hourly-devops-triage picks up p24-infra-nc-alert issue
         │   → WAHA matches HUMAN_REQUIRED_PATTERNS
         │   → Labels human-action
         │   → Sends Telegram (P1 bypasses quiet hours → fires at 03:xx Warsaw)
         │
         └── Human resolves → WAHA session recovers
             → next health-check.yml run: all green → closes server-down issue
             → Discord: "All services healthy again"

---

5. Observability Gaps — Recommended Next Steps

Gap	Effort	Value	Issue
Push nightly check pass/fail as Prometheus gauge	Low	Grafana history of check health	New
Expose WAHA session status as Prometheus metric (custom exporter)	Medium	Real-time session state in Grafana	New
Add node_exporter to bms-2, bms-3, bms-4	Medium	Disk/CPU alerts for all servers	New
Grafana dashboard: nightly check timeline + triage activity	Medium	Single-pane health view	New
Transition-only Discord for health-check (not every-run)	Low	Reduce noise	spec 06
bms-4 runner health check in health-check.yml	Low	Detect AI-Dev runner down	New

---

6. Related Documents

• healthcheck-design.md — check frequency, element inventory, tier definitions
• nightly-checks-triage.md — agent skill specs
• alert-response-runbook.md — per-alert human reaction matrix
• monitoring-stack-operations.md — Prometheus/Grafana/Alertmanager ops