Plan #4018 — Stack Connectivity Health dashboard (6 stacks)

Status: Design (Code-change-design ready for /review-plan) Owner-issue: #4018 Related closed sub-tasks: #4012 (W3 monitoring parity), #4013 (et-oper /api/health) Branch: plan/4018-stack-connectivity-health

1. Goal (one sentence)

Ship a single Grafana “Stack Connectivity Health” dashboard proving that each of the 6 production stacks (W3, W4, et-oper, et-lager, n8n-bms4, n8n-cloud) is actually connected to every one of its datastore dependencies using that stack’s own application credentials — not admin, not proxy HTTP-200, not “the container is running” — so that a stale/rotated/dropped credential is visible in Prometheus within one scrape cycle.

2. Audit baseline (what “connectivity” means today)

Confirmed 2026-07-12 by reading monitoring/exporters/, monitoring/prometheus/prometheus.yml, monitoring/prometheus/rules/, and per-stack docker-compose files. Only two exporters do a real credentialed connect to an app-owned datastore:

Existing coverage	Where it runs	Credential used	Datastore
`mailgun-pipeline-exporter` (#3688)	bms-1	`MAILGUN_MONGODB_URL` (w4_app-scoped, `pinbox24-backends.env.sops`)	w4_db Mongo rs0
`pg-stats-exporter` (#38)	vps-i1	Supabase `infra-ops` role (monitoring, not the et-oper project)	Supabase Postgres

Everything else is one of:

Admin/prometheus-user probe — mongodb-exporter-bms2/-bms3 (bms-4), p24-status rs.status() — connects but not on app creds, so a bad app credential is invisible.
Passive log signal — mezmo-exporter derives pinbox24_response_time_p99_ms, pinbox24_mongo_timeouts_5m_total etc. from shipped logs (both v42-prod and v32-prod after #4012). Tells us the app is unhappy, does not confirm the connection itself.
HTTP/TLS/TCP uptime — blackbox, blackbox_bms1 (#3739), vercel-exporter (#4013 deployment-level). Proves the box responds, does not touch the datastore.
API-poll — n8n-cloud-exporter, n8n-bms4-exporter. Managed / API surface only.

Per-stack gap matrix (this is what the dashboard has to close):

Stack	Auth-to-own-Mongo	Auth-to-own-Redis	Auth-to-own-Postgres	Auth-to-own-RabbitMQ	Runtime-health signal
W4 (v42-prod, bms-1)	partial — only mailgun path (`mailgun-pipeline-exporter`)	none	n/a	none	mezmo (`pinbox24_*` v42)
W3 (v32-prod, bms-1/bms-3)	none — only blackbox `/api/i18n/langs` 200 as a proxy	none (post-#3712 local `redis-v32`)	n/a	n/a	mezmo (`pinbox24_*` v32, after #4012)
et-oper (Vercel)	none — no probe of `et_oper→w4_db` (`PINBOX24_MONGODB_URI`)	n/a	none — no probe of Supabase et-oper project on the app’s own key	n/a	`/api/health` (after #4013) + `vercel_deployments`
et-lager (Vercel)	none — Atlas cluster never touched	n/a	n/a	n/a	HTTP 200 sign-in + Vercel deploy state only
n8n bms-4	n/a	broker Redis: `redis_bms4` job (Prometheus scrape of `redis_exporter` — admin, not n8n creds)	none — `n8n-postgres` container never probed on n8n’s own creds	n/a	`/healthz` + execution metrics
n8n cloud	n/a	n/a	n/a	n/a	uptime + `n8n_cloud_scrape_ok` (managed — ceiling)

3. Architecture — why this cannot be one exporter on vps-i1

Three hard constraints force a 3-zone split. Each is verified in-repo:

SOPS credentials live where the stack runs. secrets/pinbox24-w{3,4}.env.sops are distributed by .github/workflows/secrets-sync.yml (sync-bms-1 job) to /opt/p24-infra/bms-1/pinbox24-w{3,4}.env. secrets/n8n-bms4.env.sops lands on bms-4 as /opt/p24-infra/bms-4/.env. vps-i1 never receives them and must never receive them (widening the blast radius of the vps-i1 age recipient would let a compromised monitoring host read production DB credentials — a security regression this plan explicitly refuses).
IP allowlists. OVH DBaaS Redis (#3812 history) and Atlas allowlists whitelist bms-1 / bms-4 / the Vercel edge, not vps-i1 (217.154.82.162). A vps-i1 probe would be rejected at the DBaaS ACL before any credential check runs, silently producing up=0 for the wrong reason.
Docker-network reachability. redis-v32, redis-v42, and wkhtml-v42-prod are on prod-v-4-net / test-net on bms-1 only (see infra-src/pinbox24/w{3,4}/docker-compose.yml). vps-i1 has no route. This is the same reason mailgun-pipeline-exporter (#3688) and blackbox-exporter-bms1 (#3739) already run on bms-1 — the pattern is established.

Resulting split:

Zone A — bms-1 / bms-4 hosts (self-hosted stacks)

Scope: W3, W4, n8n-bms4 datastore connectivity.

Deliverable: one new exporter, stack-connectivity-exporter (working name), deployed twice — once on bms-1 (probes W3, W4), once on bms-4 (probes n8n Postgres, n8n Redis broker). Each instance reads the <stack>.env file already on its host, extracts the app-owned connection string per datastore, opens a bounded-timeout connection using the safe extraction pattern from mailgun-pipeline-exporter/app.py, runs the cheapest read that a real client would do (ping for Mongo/Redis, SELECT 1 for Postgres, basic.qos for AMQP), and exports:

stack_datastore_up{stack="w4", datastore="mongo_w4_db",   cred_source="pinbox24-w4.env"}       0|1
stack_datastore_latency_seconds{stack, datastore, cred_source}                                gauge
stack_connectivity_exporter_check_errors_total{stack, datastore, reason="timeout|auth|dns|refused|other"} counter
stack_connectivity_exporter_last_check_timestamp_seconds{stack, datastore}                    gauge

Every gauge series is initialised at import time so a hard exporter crash cannot masquerade as “stack still up” (the Pinbox24NoLogs lesson in rules/pinbox24.yml). Every external call uses bounded timeouts (mirror MONGO_TIMEOUT_MS=10_000 in mailgun-pipeline-exporter/app.py:102); “never a hung connection” is a hard rule.

Zone B — Vercel-hosted stacks (no host we can put a container on)

Scope: et-oper, et-lager.

Deliverable: an /api/connectivity route in each app (extending et-oper’s already-shipped /api/health from #4013). The route:

Runs on the Vercel edge, sees runtime env vars from the SOPS-synced Vercel project settings.
For each dependency, does a real credentialed operation from within the request handler (Supabase: select 1 limit 1 via the app’s own service key; Mongo: db.command({ ping: 1 }) with the app’s PINBOX24_MONGODB_URI / Atlas cluster URI).
Returns strict JSON — { "supabase": {"up": true, "latency_ms": 42, "cred_source": "SUPABASE_SERVICE_ROLE_KEY"}, "mongo_w4_db": {...}, ... } — no wrapping, no HTTP 200 on partial failure.
Blackbox http_2xx on this route stays as HTTP-uptime, but the same target also gets an http_2xx probe with an added JSON-content match, plus a new vercel-connectivity-exporter (vps-i1) that fetches this JSON every 60 s and republishes it as stack_datastore_up{...} with the same label schema as Zone A. Prometheus never has to parse JSON in a relabel rule (that was rejected once already — brittle, unreadable).

Why the extra exporter and not just blackbox regex. blackbox regex matching turns any malformed response into “the whole check failed” and gives us no latency-per-dependency. A tiny polling exporter (~120 LoC pattern of queue-exporter) gives per-dependency up + latency_ms and can distinguish “the /api/connectivity route itself is 500” (HTTP layer failure) from “the route says supabase is down” (application-layer failure) — these are different alert paths.

Zone C — n8n cloud (managed, no changes)

The existing n8n-cloud-exporter API-poll (n8n_cloud_scrape_ok) is the ceiling. Managed cloud does not expose datastore internals; noted as “managed — API-poll only” on the dashboard, no new code required.

4. Presentation layer

One new Grafana dashboard: stack-connectivity-health.json, a 6-row × N-column matrix of stack_datastore_up{...} cell colouring, with:

Header row: stack name, colour-coded overall status (min of that stack’s cells).
Cell tooltip: cred_source label + last-check timestamp + latency + last error reason.
One time-series panel per stack showing 24 h avg_over_time(stack_datastore_up[5m]) — cliff edges (rotation-day drops) are the primary human read.
Panel for stack-connectivity-exporter self-health (up{job=~"stack_connectivity_.*"} + stack_connectivity_exporter_check_errors_total), so a dead exporter shows up as its own cell, never as green cells.
p24-status.py (scripts/) grows a stack_connectivity section that reads the same metric via the Prometheus HTTP API and prints a one-line-per-stack summary for the status page.

Alert rules (new file monitoring/prometheus/rules/stack-connectivity.yml):

StackDatastoreDown — stack_datastore_up == 0 for 3m → P2 Discord.
StackDatastoreLatencyHigh — stack_datastore_latency_seconds > 2 for 10m → info-only.
StackConnectivityExporterDown — up{job=~"stack_connectivity_.*"} == 0 for 5m → P1 (silent monitoring is worse than no monitoring — this was #2492).
Every rule validated with promtool check rules before commit (CLAUDE.md §Monitoring stack).

5. Security decision to record in the plan

Decision A — reuse app credentials from SOPS on bms-1/bms-4.

Pro: no new credential to rotate; the exporter proves the same credential the app uses; reuse follows the exact pattern from mailgun-pipeline-exporter/app.py (already in prod).
Con: broadens the set of processes reading the app SOPS file on that host by one container.

Decision B — mint a new read-only monitoring role per datastore.

Pro: least privilege — the exporter cannot write.
Con: doubles the credential inventory; a rotation drift between “app cred” and “monitor cred” hides the exact failure class we are trying to detect (the app cred rotated, the monitor cred did not, so the dashboard says green while prod is broken — #3688 in a slightly different suit).

Recommendation (to be ratified in /review-plan): Decision A for Mongo/Redis/Postgres credentials on bms-1/bms-4 (matches shipped pattern, no drift risk); Decision B only for Supabase (Vercel path already uses the anon key at edge — a scoped monitoring role there is cheap and does not create a rotation-drift risk because the Vercel /api/connectivity route reads its runtime env at request time, not at container start). Never any secret value in any exporter log, PR body, GitHub comment, or Discord notification — only the key name and a cred_source label pointing at the SOPS file (never at the value). This is enforced by the Credential Safe-Handling Rules block in agent-prompts/worker-issue.md and by CLAUDE.md §Secrets.

6. Decomposition into implementation issues

Each row becomes one PR / issue after /review-plan merges:

#	Deliverable	Zone	Repo	Weight	Depends on
A-1	`stack-connectivity-exporter` code + Dockerfile + tests	A	`radieu/p24-infra`	light	—
A-2	Deploy A-1 to bms-1 (docker-compose + prometheus.yml target + ufw allowlist for vps-i1)	A	`radieu/p24-infra`	infra-task	A-1
A-3	Deploy A-1 to bms-4 (docker-compose + prometheus.yml target + ufw allowlist for vps-i1)	A	`radieu/p24-infra`	infra-task	A-1
B-1	Extend et-oper `/api/health` → `/api/connectivity` (per-dep JSON, no HTTP 200 on partial fail)	B	`radieu/et-operational-platform`	light	—
B-2	Add et-lager `/api/connectivity` (Atlas ping on app key)	B	`radieu/et-lager`	light	—
B-3	`vercel-connectivity-exporter` on vps-i1 that fetches B-1/B-2 JSON and republishes as `stack_datastore_up{...}`	B	`radieu/p24-infra`	light	B-1, B-2
D-1	Grafana dashboard `stack-connectivity-health.json`	dashboard	`radieu/p24-infra`	light	A-2, A-3, B-3
D-2	Alert rules `stack-connectivity.yml` + promtool validation	dashboard	`radieu/p24-infra`	light	D-1
D-3	`scripts/p24-status.py` — add `stack_connectivity` section	dashboard	`radieu/p24-infra`	light	D-1

Not new issues (already done, referenced for completeness): #4012 (mezmo v32-prod parity) and #4013 (et-oper /api/health) — both closed; B-1 extends #4013’s shipped route rather than adding a second endpoint.

7. Files-to-change signature (per Step 2a scope-targeted reads)

A-1 will touch:

monitoring/exporters/stack-connectivity-exporter/{app.py,Dockerfile,requirements.txt,requirements-dev.txt,tests/}
docs/stack-connectivity-exporter-operations.md
Supabase dev_r_services row (via MCP or REST fallback per CLAUDE.md compliance)

A-2 / A-3 will touch:

bms-1/docker-compose.yml (service block, mirroring mailgun-pipeline-exporter)
bms-4/docker-compose.yml (service block)
bms-1/.env.example + bms-4/.env.example (documented key list, no values)
monitoring/prometheus/prometheus.yml (new stack_connectivity_bms1 + stack_connectivity_bms4 jobs)
Playbook docs/playbooks/add-new-monitoring-exporter.md — cross-link only (already covers the pattern)
Ufw allowlist entries → executed via infra-task worker (SSH), never inline in a dev-issue PR

B-1 (in radieu/et-operational-platform):

src/pages/api/connectivity.ts (or extends /api/health)
Existing service files only — no new business logic beyond dep-ping

B-2 (in radieu/et-lager):

src/pages/api/connectivity.ts + Atlas client re-use (import from existing service; do NOT create a new Mongo client per request — one long-lived cached client, bounded timeouts)

B-3 will touch:

monitoring/exporters/vercel-connectivity-exporter/{app.py,Dockerfile,requirements.txt,tests/}
monitoring/docker-compose.yml (add service block on vps-i1)
monitoring/prometheus/prometheus.yml (stack_connectivity_vercel job)
docs/vercel-connectivity-exporter-operations.md
Supabase dev_r_services row

D-1/D-2/D-3:

monitoring/grafana/provisioning/dashboards/stack-connectivity-health.json
monitoring/prometheus/rules/stack-connectivity.yml
scripts/p24-status.py (extend, do not rewrite)

8. Regression risks (must appear in each downstream PR)

A rotation-drift silent-green. If A-1 caches a connection object at process start, a rotated app cred will still show up=1 from the cached socket for up to POLL_INTERVAL_S. Mitigation: every check opens a fresh short-lived client (Mongo serverSelectionTimeoutMS=10s, no persistent pool), same as mailgun-pipeline-exporter/app.py:102. Unit test: mock a ServerSelectionTimeoutError on the second call, assert up flips to 0 within one cycle.
DBaaS ACL rejection masquerading as auth failure. A dropped IP allowlist entry looks like “connection refused” to the exporter but is a network-layer denial, not an auth denial. The reason= label on stack_connectivity_exporter_check_errors_total must distinguish refused/timeout/auth/dns — otherwise the runbook can’t tell “the allowlist expired” from “the credential expired”. Unit tests cover each error class.
Vercel edge cold start latency. First request to /api/connectivity on a cold Vercel function can take >5 s and time out the exporter, false-firing StackConnectivityExporterDown. Mitigation: POLL_INTERVAL_S=60, http timeout=15, and a for: 5m on the alert.
A dashboard cell going missing. If the exporter stops producing a series, Grafana panel shows “no data” instead of red. Rules on absent() per-stack cell need to exist, otherwise a dead exporter looks like a silent success.
Widening SOPS reader set on bms-1. The new exporter container reads pinbox24-w4.env / pinbox24-w3.env (Decision A). This is architecturally identical to mailgun-pipeline-exporter, but sys-security should sign off in /review-plan. Document in the PR body under Security review required.

9. Manual test checklist (post-deploy)

Zero-state: with the exporter running but the target datastore intentionally paused (docker stop redis-v32 briefly), stack_datastore_up{stack="w3",datastore="redis"} = 0 within POLL_INTERVAL_S.
Rotation drill: rotate V32_REDIS_PASSWORD via the standard playbook. Dashboard cell turns red within POLL_INTERVAL_S + for: window; goes green again after secrets-sync.yml completes. Cred-drift class is caught.
promtool check rules monitoring/prometheus/rules/stack-connectivity.yml clean.
promtool check config monitoring/prometheus/prometheus.yml clean.
Exporter unit tests pass (python -m pytest monitoring/exporters/stack-connectivity-exporter/tests/).
Grafana dashboard renders on all 6 stacks with no “no data” panels once A-2/A-3/B-3 are deployed.
No secret value in any commit, PR body, Discord embed, or agent-session log — only key names and cred_source labels.
dev_r_services in Supabase has a row for stack-connectivity-exporter and vercel-connectivity-exporter (compliance).

10. Dependencies

Existing mailgun-pipeline-exporter (#3688) — pattern template.
Existing secrets-sync.yml distribution — no new secrets, but SOPS-file allowlist per host must not change.
promtool in monitoring stack — already in Dockerfile.
No new external service; no new SaaS.

11. Out of scope

Rewriting mongodb-exporter-bms2/bms3 — those are admin-cred exporters for rs0 replica-set metrics, a different purpose.
Adding datastore-server exporters (e.g. redis_exporter per new container). Only connectivity from the app’s own credential is in scope for this dashboard; datastore-side metrics are the respective db-performance / containers dashboards’ job.
Rewriting n8n-cloud-exporter — managed ceiling accepted.

12. Iteration loop

Per issue-body instructions and CLAUDE.md §Creating GitHub Issues:

This plan is posted as the ## Code-change-design comment on #4018 by the same worker run.
Run /review-plan 4018 — Haiku review pass, at most 3 cycles.
On approval, the individual implementation issues in §6 are filed by a follow-up run and dispatched by weight.
Merge to main; no dev branch (p24-infra ships from main directly, CLAUDE.md §Branching).

Written by dev-coder session plan/4018-stack-connectivity-health on vps-h1, 2026-07-12.

p24-infra Docs

Explorer

plan-4018-stack-connectivity-health