p24-infra Docs

et-op-app-health-monitoring

5 min read

et-operational-platform — App-Layer Runtime Health Monitoring

Status: p24-infra side prepared (gated); blocked on production deploy of the app-side /api/health route Issue: radieu/p24-infra#4013 · reconciliation radieu/p24-infra#6342 Cross-repo dependency: radieu/et-operational-platform#1207 (route implemented) → radieu/et-operational-platform#1688 (promote to production main)

Investigation 2026-08-16 (#6342) — route is live on staging, NOT production. The route file src/pages/api/health.ts exists on the staging and rc branches but is absent from main. The production Vercel project et-operational-platform tracks main, so https://et-operational-platform.vercel.app/api/health returns 404, while the staging project et-operational-platform-7ktl (tracks staging) returns the healthy contract at https://et-operational-platform-7ktl.vercel.app/api/health. #1207 was closed as shipped but never promoted past rc. Do NOT repoint the probe at the -7ktl staging URL to make it go green — that would monitor staging, not production, defeating the purpose. The gated blackbox_app_health target already points at the production URL and stays gated until #1688 promotes the route to main. Prod-vs-staging Vercel mapping: infrastructure-overview.md §3.

1. Problem

et-operational-platform (Vercel, Next.js) monitoring covered only two signals:

  • Blackbox http_2xx on https://et-operational-platform.vercel.app/ every 30 s → EndpointDown / EndpointSlow (monitoring/prometheus/prometheus.yml, rules/synthetic.yml).
  • vercel-exporter (job vercel_deployments) → deployment state + VercelProductionDeploymentFailed (rules/infrastructure.yml).

Neither detects an application-layer / runtime failure: the app process is up and the root URL returns 200, but a critical dependency (Supabase/DB) is unreachable so key API calls fail. That gap was invisible to Prometheus/Alertmanager — runtime errors were only visible ad-hoc in the Vercel UI.

2. Design decision — why the /api/health endpoint (not a vercel-exporter extension)

Two complementary directions were considered for #4013:

DirectionCoverageCostVerdict
A. /api/health dependency endpoint + blackbox probeDirectly measures the missing signal — dependency (Supabase/DB) reachability from inside the running appLow: reuses existing blackbox-exporter + Alertmanager + EndpointDown/EndpointSlow. No new container/exporter. One app route + one gated scrape job.Chosen (MVP)
B. Extend vercel-exporter with runtime error-rate metricsWeaker/indirect — genuine runtime-error data needs Vercel Log Drains or the Observability query API (plan-gated, higher effort). Deployment-state errors are already covered by vercel_deployment_state.Higher: new API integration, uncertain data availability on current plan, still doesn’t measure dependency health.Deferred

Rationale: Direction A closes exactly the described gap (app up but dependency down) at the lowest incremental infra cost, and the alerting comes for free from the existing generic blackbox rules. Direction B’s genuinely-new value (runtime error-rate) depends on a Vercel API surface that is plan-gated and uncertain, and would not by itself detect an unreachable Supabase. If per-route runtime error-rate is later needed, it can be added on top of A without rework.

3. Endpoint contract (implemented app-side — see et-operational-platform#1207)

GET /api/health, unauthenticated, side-effect free, fast (polled every 30 s).

Healthy — every critical dependency reachable:

HTTP 200
{ "status": "ok", "checks": { "supabase": "ok" }, "ts": "<ISO8601>" }

Degraded — one or more critical dependencies unreachable:

HTTP 503
{ "status": "degraded", "checks": { "supabase": "error" }, "ts": "<ISO8601>" }

Never emit secret values (connection strings, keys) in the body or logs — dependency names + ok/error only.

4. p24-infra wiring (this repo)

4.1 Blackbox module — http_health_json (live)

Added to monitoring/prometheus/blackbox.yml. probe_success=1 only when the target returns HTTP 200 and the body matches "status":"ok":

http_health_json:
  prober: http
  http:
    valid_status_codes: []            # 2xx only — a 503 "degraded" fails the probe
    fail_if_body_not_matches_regexp:
      - '"status"\s*:\s*"ok"'

A module definition is inert until a scrape target references it, so shipping it now is safe.

4.2 Scrape job — blackbox_app_health (gated / commented)

Added, commented out, to monitoring/prometheus/prometheus.yml:

# - job_name: blackbox_app_health
#   params: { module: [http_health_json] }
#   static_configs:
#     - targets: [ https://et-operational-platform.vercel.app/api/health ]
#   relabel_configs: [ ... standard blackbox relabel ... ]

It stays commented until the app route ships. The previous /api/health target was removed because the route 404’d and fired a permanent false EndpointDown — do not re-enable prematurely.

4.3 Alerting — no new rule needed

Once enabled, the /api/health target is covered by the existing rules in rules/synthetic.yml:

  • EndpointDown (probe_success == 0, for: 2m, critical) — fires when the app reports a degraded dependency (503 or non-ok body) or is unreachable. This is the app-layer health alert.
  • EndpointSlow (probe_duration_seconds > 2, for: 5m, warning) — dependency-check latency.

Adding a bespoke probe_success-based rule would double-fire with EndpointDown, so it is deliberately omitted.

5. Activation checklist (after et-operational-platform#1688 deploys the route to production main)

  1. Verify the route is on main, not just staging/rc (#6342). #1207 shipped the route to staging/rc only; production tracks main. Confirm src/pages/api/health.ts is present on the et-op main branch before enabling — otherwise the probe fires a permanent false EndpointDown.
  2. Confirm the production URL GET https://et-operational-platform.vercel.app/api/health returns the contract in §3 (a 200 on the -7ktl staging URL is not sufficient — that is a different Vercel project; see the investigation note at the top of this doc).
  3. Uncomment the blackbox_app_health job in monitoring/prometheus/prometheus.yml.
  4. Validate: promtool check config prometheus/prometheus.yml (see CLAUDE.md).
  5. Deploy config (deploy-monitoring-config.yml) and confirm the target is UP in Prometheus.
  6. Compliance: register the probe in Supabase dev_r_services (compliance_workbook='yes') and add a Grafana panel for probe_success{instance=".../api/health"} + probe_duration_seconds.

6. Compliance

  • No new deployed p24-infra service/container is added by this change (module + gated job only), so dev_r_services / Grafana updates are deferred to activation (step 5 above).
  • No secrets touched. VERCEL_TOKEN and other credentials are unaffected; the probe uses no auth.

Graph View