plan-4132-w3-w4-monitoring-strategy

Plan #4132 — W3/W4 backend + et-op/et-lager monitoring strategy

Status: Design (Code-change-design ready for /review-plan) Owner-issue: #4132 Branch: plan/4132-w3-w4-monitoring-plan Closest parent plan: #4018 — Stack Connectivity Health dashboard (READ FIRST — this plan builds on it, does not duplicate it) Motivating incident: #4078 / PR #4086 — 2026-07-13 W3 RESO import outage (silent post-secrets-sync Mongo-auth desync)

Plan only — do NOT implement. Deliverable is this document + the ## Code-change-design comment on #4132. Each row in §7 becomes its own implementation issue after /review-plan.

---

1. Goal (one sentence)

Define what signals the W3 (V32_*) and W4 (V42_*) Pinbox24 backends — and their two Vercel consumers, et-operational-platform and et-lager — need in order to catch the failure classes that current monitoring is blind to (the #4078 auth-desync, a stalled import path, a silent crash-loop, an unreachable S3 write-path), and how each signal is produced, where it runs, how often, and where its alert routes — expressed as a signal-inventory table, a gap analysis against today’s stack, and a P0-first rollout.

2. Relationship to #4018 — the boundary (read this before adding anything)

#4018 and #4132 are adjacent, not overlapping. Keeping the boundary explicit is the single most important decision in this plan, because both touch the same six stacks.

Dimension	Owner	Mechanism
”Is each stack authenticated to its own datastore on its own app credential, visible in Prometheus within one scrape?” (Mongo/Redis/Postgres ping, latency, up)	#4018	stack-connectivity-exporter (bms-1/bms-4, Zone A) + /api/connectivity route + vercel-connectivity-exporter (Zone B). Metric stack_datastore_up{stack,datastore,cred_source}
#4078-class desync caught at the moment of deploy (verify the just-synced credential authenticates before the sync job reports success)	#4132 (§6a)	Post-deploy verification gate inside secrets-sync.yml per-stack jobs
RESO import E2E path health (parse → Mongo write → S3 store actually completing, not just each dependency being reachable)	#4132 (§6b)	RESO import synthetic monitor (register-record freshness SLA)
S3 / Wasabi write-path (a real put/get round-trip on the app bucket cred, not just TCP/TLS reachability)	#4132 (§5, signal S3-1)	Extend the connectivity exporter with an S3 write-path check, or a dedicated probe
Container crash-loop / restart-loop per W3/W4 container (v32-prod-reso, v32-prod-socket, v42 family)	#4132 (§5, signal C-1)	cadvisor restart counters + a new alert rule; complements existing pm2-exporter heap
WebSocket / socket gateway health (v32-prod-socket, socket.w3.pinbox24.com)	#4132 (§5, signal H-3)	Blackbox WS-aware probe
Consumer → producer dependency correlation (which W3/W4 API each of et-op / et-lager needs, how a backend outage surfaces to the user, how to root-cause an app error spike back to W3/W4)	#4132 (§4B, §6c)	Dependency map + user-flow synthetic checks + a correlation recording rule
The 6-stack Grafana connectivity matrix + stack-connectivity.yml alert rules	#4018	stack-connectivity-health.json

Rule of thumb: if the question is “can this stack reach its datastore right now with its own key” → it belongs to #4018. If the question is “did the business workflow actually complete end-to-end” or “was a bad credential shipped by the sync itself” → it belongs to #4132.

Where #4132 needs a per-datastore up signal (e.g. to correlate), it consumes #4018’s stack_datastore_up series rather than minting its own. #4132 therefore soft-depends on #4018 Zone A shipping, but its two P0 deliverables (§6a post-sync gate, §6b import synthetic) stand alone and can ship first.

3. Current coverage baseline (confirmed 2026-07-14, this repo)

Sources: monitoring/docker-compose.yml, monitoring/prometheus/prometheus.yml, monitoring/prometheus/blackbox.yml, monitoring/prometheus/rules/*.yml, monitoring/alertmanager/alertmanager.yml.tpl, infra-src/pinbox24/w{3,4}/docker-compose.yml, .github/workflows/secrets-sync.yml, docs/playbooks/v32-prod-mongoose-authfail-storm.md.

W3/W4 topology (both stacks run on bms-1 94.23.26.113; MongoDB rs0 on bms-2 PRIMARY / bms-3 SECONDARY, arbiter bms-4):

• W3 (v32-prod image): v32-prod (api.w3.pinbox24.com), v32-prod-reso (w3.reso-integration-addrecords.pinbox24.com — the RESO Excel import service), v32-prod-socket (socket.w3.pinbox24.com), s3-v32-prod (Wasabi proxy), local redis-v32, three cron-v32-*. Datastores: w3_db on rs0 (app user w3_app), local redis-v32, Wasabi buckets s3-v32-prod-*. Secrets: secrets/pinbox24-w3.env.sops.
• W4 (v42-prod image): v42-prod (api.w4.pinbox24.com), v42-notify-prod, s3-v42-prod / s3-v2-v42-prod, mailgun-v42-prod, redis-v42, wkhtml-v42-prod, pdf-gen-v42-prod, git-deploy-v42-prod. Datastores: w4_db on rs0 (app user w4_app), local redis-v42, Wasabi buckets s3-v42-prod-*, RabbitMQ on bms-4 (54.36.123.110:5672). Secrets: secrets/pinbox24-w4.env.sops + secrets/pinbox24-backends.env.sops.

What exists today (by signal class):

Class	Covered by	Verdict
HTTP uptime	blackbox http_2xx on api.w3/api.w4 /api/i18n/langs, et-lager /auth/sign-in; blackbox_ssl on 9 W3/W4 vhosts	✅ endpoint responds; ❌ says nothing about datastore or workflow
Container resource	cadvisor (bms-1), pm2-exporter heap (#4014, v32-prod/v42-prod), blackbox_bms1 TCP probe (wkhtml-v42-prod:80, #3739)	✅ liveness/heap; ❌ no crash-loop alert per RESO/socket container
App-layer log signal	mezmo-exporter → pinbox24_response_time_p99_ms, pinbox24_mongo_timeouts_5m_total for v42 and v32 (#4012)	✅ app unhappy; ❌ reactive (symptom, not cause)
Mailgun→Mongo pipeline	mailgun-pipeline-exporter (bms-1, real w4_app Mongo connect, #3688)	✅ the one true app-cred connect today (W4 mail path only)
MongoDB rs0	mongodb-exporter (admin/prometheus user, bms-2/bms-3)	✅ replset health; ❌ admin cred — a bad w3_app/w4_app password is invisible (this is exactly #4078)
Per-stack app-cred auth	**planned in 4018 (stack_datastore_up)	🔜 closes the auth-visibility gap generically
S3 write-path	backup-exporter / cost-exporter touch Wasabi on infra creds	❌ no probe on the app bucket cred, no write round-trip
RESO import E2E	—	❌ nothing — the #4078 blind spot
Post-sync desync guard	secrets-sync.yml per-stack health checks exist but do not verify app-cred Mongo auth after write	⚠️ partial — see §6a
Consumer correlation	et-op /api/health blackbox scaffold (#4013, gated); et-lager HTTP 200 only	❌ no W3/W4→consumer dependency signal

Alert routing today: Alertmanager → (mostly) n8n alertmanager-to-incidents-v3 → meta-dispatcher /alert-ingest → dev_r_alert_events staging → batch_stage_alerts() pg_cron → one GH issue per (alertname, service) batch (#3919); self-monitoring/security/hardware take the direct-email-only bypass. Script/cron errors use the P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL Discord embed + GH issue standard. All new #4132 alerts route through the existing batched path — no new webhook, no new receiver.

4. Signal inventory (the WHAT × HOW)

⛳ = new in #4132 · 🔁 = delivered by #4018 (referenced, not re-built) · ✅ = already exists. “Where” = the host/service that produces the signal. Alert route column names the alert-rule group.

4A. W3/W4 backend signals

#	Signal	How (mechanism)	Where it runs	Interval	Alert route (rule group)	Owner
C-1 ⛳	Container restart-loop (v32-prod-reso, v32-prod-socket, s3-v32-prod, v42 family)	container_restart_count delta from cadvisor (already scraped) + new rule	prometheus (vps-i1) reads cadvisor bms-1	60s	new pinbox24-containers.yml → batched GH	infra-ops
C-2 ✅	PM2 heap / process alive	pm2-exporter pm2_heap_usage_ratio (#4014)	pm2-exporter (bms-1)	30s	pinbox24.yml (Pinbox24HeapUsageHigh)	infra-ops
M-1 🔁	w3_app/w4_app Mongo auth on the app credential	stack_datastore_up{stack,datastore="mongo_w{3,4}_db"}	stack-connectivity-exporter (bms-1)	30–60s	stack-connectivity.yml (#4018)	infra-ops
M-2 ⛳	Mongo auth verified at deploy time (catch #4078 before symptoms)	post-write mongosh ping on the just-synced app cred inside the sync job	GH Actions (secrets-sync.yml)	on every W3/W4 sync	job fails loud → Discord + GH issue	CI
R-1 🔁	redis-v32/redis-v42 auth on the app credential	stack_datastore_up{datastore="redis"}	stack-connectivity-exporter (bms-1)	30–60s	stack-connectivity.yml (#4018)	infra-ops
S3-1 ⛳	Wasabi write-path on the app bucket cred (put small object → get → delete)	extend stack-connectivity-exporter with an S3 round-trip check → stack_s3_writepath_up{stack,bucket}	stack-connectivity-exporter (bms-1)	300s	new rows in stack-connectivity.yml	infra-ops
H-1 ✅	W3/W4 HTTP API uptime + latency	blackbox http_2xx on /api/i18n/langs	blackbox (vps-i1)	30s	synthetic.yml (EndpointDown/Slow)	infra-ops
H-2 ✅	W3/W4 vhost TLS cert expiry (9 domains)	blackbox_ssl ssl_cert	blackbox (vps-i1)	300s	synthetic.yml (CertExpiring*)	infra-ops
H-3 ⛳	v32-prod-socket WebSocket handshake health	blackbox probe with WS upgrade (or TCP-connect fallback) on socket.w3.pinbox24.com	blackbox (vps-i1)	60s	synthetic.yml	infra-ops
Q-1 ✅	W4 RabbitMQ / Redis Bull broker reachable	blackbox_tcp 5672; redis_bms4 job	blackbox/redis-exporter	30s	synthetic.yml / queues.yml	infra-ops
Q-2 ⚠️	W4 Direct-API-Ingestion overlap (duplicate flood)	Redis singleton-lock (#4008) — verify impl landed; add w4ingest_lock_contention_total	n8n (bms-4)	per run	n8n.yml	infra-ops
I-1 ⛳	RESO import E2E — register records with status “import udany” arriving within SLA	import synthetic monitor (register-record freshness query)	new reso-import-exporter on bms-1 (Mongo app cred)	300s	new reso-import.yml (RESOImportStalled)	infra-ops
A-1 ✅	Mongoose disconnect / auth-fail storm (reactive)	mezmo-exporter pinbox24_mongo_timeouts_5m_total, MongooseDisconnect	mezmo-exporter (vps-i1)	60s	pinbox24.yml	infra-ops

4B. Consumer signals (et-operational-platform, et-lager — both Vercel/Next.js)

Dependency facts (from secrets/et-operational-platform.env.sops key map + et-lager route analysis): et-op talks to w4_db Mongo directly via PINBOX24_MONGODB_URI and consumes GPS / document-ingest / inspection webhooks; it has no documented direct W3 API call (W3 status reaches it only through shared w4_db data). et-lager is a thin proxy — nearly every route forwards the user JWT to https://api.w4.pinbox24.com/api/*, so a W4 outage surfaces directly as et-lager request failures; it also holds its own MongoDB Atlas cluster.

#	Signal	How	Where it runs	Interval	Alert route	Owner
E-1 🔁	et-op → w4_db Mongo auth on PINBOX24_MONGODB_URI	/api/connectivity + vercel-connectivity-exporter (#4018 B-1/B-3)	Vercel edge + vps-i1	60s	stack-connectivity.yml (#4018)	app + infra
E-2 🔁	et-lager → Atlas ping on app key	/api/connectivity (#4018 B-2)	Vercel edge + vps-i1	60s	stack-connectivity.yml (#4018)	app + infra
E-3 ✅	et-op / et-lager frontend uptime	blackbox http_2xx (et-op /, et-lager /auth/sign-in)	blackbox (vps-i1)	30s	synthetic.yml	infra-ops
E-4 ⛳	User-flow synthetic on the critical W4-dependent path (et-lager protocol/invoice lookup that proxies to api.w4)	synthetic check hitting a read-only W4-backed route (unauth-safe or scoped test account)	blackbox (or a lightweight synthetic exporter) vps-i1	300s	new consumer-synthetic.yml	infra-ops
E-5 ⛳	Correlation — et-op/et-lager error-spike ↔ W4 backend down	Prometheus recording rule joining consumer HTTP-5xx rate with stack_datastore_up{stack="w4"} / pinbox24_* v42 signals	prometheus (vps-i1)	60s	consumer-synthetic.yml (info-only, aids triage)	infra-ops
E-6 ⛳ (dep)	et-op /api/health app-layer probe activation	un-gate blackbox_app_health job (#4013) once cross-repo et-operational-platform#1207 ships	prometheus (vps-i1)	30s	synthetic.yml	app + infra

5. Gap analysis vs current stack

Ranked by the failure class each closes, worst-first:

1. [P0] #4078 auth-desync is invisible at deploy time. rs0 is watched on the admin cred, so a stale w3_app/w4_app password shipped by secrets-sync.yml produces a Mongoose auth storm that only mezmo-exporter sees — after the import already failed. Closed by M-2 (§6a post-sync gate, catches it in the sync job) + M-1 (#4018 continuous exporter, catches drift between deploys).
2. [P0] No proof the RESO import path completes. Reachability of Mongo + S3 individually ≠ the parse→write→store workflow succeeding. Closed by I-1 (§6b synthetic import monitor).
3. [P1] No S3 write-path on the app bucket cred. A rotated/dropped V32_s3Bucket_* / S3_ACCESS_KEY_ID looks green (infra-cred exporters still work). Closed by S3-1.
4. [P1] No crash-loop alert for RESO/socket/s3 containers. cadvisor scrapes restart counts but no rule fires. Closed by C-1.
5. [P1] Consumer blast-radius invisible. A W4 outage hits et-lager users immediately but there is no signal that ties the user-facing failure to the W4 root cause. Closed by E-4/E-5 and, once shipped, E-1/E-2 (#4018).
6. [P2] Socket gateway only TCP-probed indirectly. Closed by H-3.
7. [P2] W4 ingestion overlap — mitigation (#4008 Redis lock) filed but implementation must be verified; add contention metric Q-2.

6. The two P0 deliverables in depth

6a. secrets-sync post-deploy verification gate (signal M-2) — catch #4078 at the source

Problem it closes: secrets-sync.yml decrypts pinbox24-w3.env.sops, writes backend-environment.env on bms-1, recreates containers, and runs health checks — but those checks (container running, PM2 online, endpoint 200) **passed during 4078 because the auth storm was a retry loop, not an immediate hard crash. The sync reported success while the import path was broken.

Design: add a per-stack credential-auth assertion to the W3 and W4 sync jobs, executed on bms-1 immediately after the env write and container recreate, before the job is allowed to report success:

• Extract the just-deployed app connection string with the safe pattern (grep one key into a shell var, never echo — CLAUDE.md §Secrets), then run inside the running container: docker exec v32-prod mongosh "$APP_URI" --quiet --eval 'db.runCommand({ping:1})' expecting { ok: 1 }. Repeat for W4 (v42-prod), and for the S3 bucket cred (a bounded head-bucket/put).
• On failure: fail the job loud (non-zero exit) → the standard P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL Discord embed + a GH issue tagged bug (Error Notification Standard). The stale credential is never left silently deployed. Roll-forward guidance links docs/playbooks/v32-prod-mongoose-authfail-storm.md.
• No secret value in any log — assertion prints only ok/FAIL and the key name.

Why this complements #4018 rather than duplicating it: #4018’s exporter catches drift on a 30–60s poll between deploys; the gate catches a bad deploy at the deploy, turning a silent success into a failed sync run the operator sees immediately. Both are wanted; neither replaces the other.

6b. RESO import E2E synthetic monitor (signal I-1) — generalize the incident-verify check

Problem it closes: the #4078 blind spot — no continuous proof that “email/feed → parse → Mongo w3_db write → Wasabi store” completes. The existing recovery playbook docs/playbooks/v32-prod-mongoose-authfail-storm.md proves it once, by hand after an incident; this generalizes that manual check into a continuous monitor.

Design: a small reso-import-exporter (≈120 LoC, mirrors mailgun-pipeline-exporter) on bms-1 (must run there — rs0 app cred + Docker-network reachability, same constraints as #4018 Zone A), reading secrets/pinbox24-w3.env’s w3_app connection string:

• Every POLL_INTERVAL_S (300s default), query the RESO register collection for the newest record whose import-status field equals the success sentinel "import udany" (string given by the issue body), and export:

  reso_import_last_success_timestamp_seconds        gauge   # newest "import udany" record's createdAt
  reso_import_records_last_window_total{status}      gauge   # counts by status in a rolling window
  reso_import_exporter_check_errors_total{reason}    counter # auth|timeout|dns|refused|other
  reso_import_exporter_last_check_timestamp_seconds  gauge
  

• Alert RESOImportStalled: time() - reso_import_last_success_timestamp_seconds > SLA (SLA TBD with the business — start at e.g. 24h, tune to the real import cadence) → P2 batched GH issue. Also RESOImportExporterDown (up==0 for 5m) → P1, because silent monitoring is worse than none (the #2492 / #4018 §4 lesson).
• Bounded timeouts, fresh short-lived Mongo client per check (no cached pool → a rotated cred flips the metric within one cycle), all gauges initialised at import so an exporter crash cannot read as “import healthy”.

Blocking dependency — D-0 (see §8): the exact collection name and status field name for the RESO register are not hard-coded anywhere in the repo. The issue’s referenced discovery step (docs/playbooks/reso-import-post-incident-verify.md) does not exist; the closest real playbook is v32-prod-mongoose-authfail-storm.md, which names w3_db and status "import udany" but not the confirmed collection/field. This must be resolved (short read of v32-prod-reso source, or a one-time mongosh inspection by an infra-task/human with bms-1 access) before I-1’s query is final.

6c. Consumer correlation (signals E-4/E-5)

et-lager is a JWT proxy over api.w4.pinbox24.com, so W4 health is et-lager health for most routes. Deliver: (1) a synthetic read against one W4-backed et-lager route on a scoped test account (gated on the test-account work 2833), and (2) a Prometheus recording rule that surfaces “consumer 5xx up while stack_datastore_up{stack="w4"}==0 or pinbox24_* v42 degraded” as a single triage hint, so an et-lager error spike points straight at the W4 root cause instead of a cold investigation.

7. Prioritized rollout → decomposition into implementation issues

Each row is one issue/PR filed after /review-plan. P0 first, and both P0 items are self-contained (no dependency on #4018 shipping).

#	Deliverable	Signal	Priority	Repo	Weight	Depends on
R0-1	Post-sync credential-auth gate in secrets-sync.yml W3+W4 jobs (Mongo ping + S3 head on app cred, fail loud)	M-2, S3-1(partial)	P0	p24-infra	light	—
R0-2	reso-import-exporter code + Dockerfile + tests (query configurable pending D-0)	I-1	P0	p24-infra	light	D-0
R0-3	Deploy R0-2 to bms-1 (docker-compose + prometheus job + ufw allowlist) + reso-import.yml rules	I-1	P0	p24-infra	infra-task	R0-2
R1-1	pinbox24-containers.yml crash-loop alert rules (cadvisor restart deltas, RESO/socket/s3 containers)	C-1	P1	p24-infra	light	—
R1-2	S3 write-path check in stack-connectivity-exporter → stack_s3_writepath_up + rules	S3-1	P1	p24-infra	light	#4018 A-1
R1-3	Consumer correlation recording rule + consumer-synthetic.yml	E-5	P1	p24-infra	light	—
R2-1	v32-prod-socket WS-aware blackbox probe	H-3	P2	p24-infra	light	—
R2-2	et-lager W4-backed user-flow synthetic (scoped test account)	E-4	P2	p24-infra	light	#2833 test acct
R2-3	Verify #4008 Redis singleton-lock impl + add w4ingest_lock_contention_total	Q-2	P2	p24-infra	light	#4008 impl
—	(reference only) Per-stack app-cred connectivity, et-op/et-lager /api/connectivity, dashboard	M-1,R-1,E-1,E-2	—	#4018	—	#4018
—	(reference only) Un-gate blackbox_app_health	E-6	—	p24-infra	trivial	et-op#1207

Compliance for each new exporter/probe (CLAUDE.md §Compliance): add a dev_r_services row (compliance_workbook='yes') + an ops doc in docs/ (docs/reso-import-exporter-operations.md, etc.). If any signal is judged AI-powered it also updates dev_r_ai_systems — none here are.

8. Open dependencies

• D-0 (blocks R0-2/R0-3): confirm the RESO register collection name and import-status field name in w3_db. Not in-repo; the issue’s cited docs/playbooks/reso-import-post-incident-verify.md does not exist — the real reference is docs/playbooks/v32-prod-mongoose-authfail-storm.md, which gives w3_db + status "import udany" but not the confirmed collection/field. Resolve via a short v32-prod-reso source read or a one-time mongosh inspection (infra-task/human, bms-1). Until then R0-2 ships with the collection/field as env-configurable placeholders.
• Soft dep on #4018 Zone A for M-1/R-1/S3-1 metric reuse — the P0 items do not need it.
• #4013 / et-op#1207 for E-6 (et-op /api/health un-gate) — cross-repo, already scaffolded.
• Test accounts #2831 (W4) / #2833 (et-lager), blocked on SOPS rebuild #2835, for E-4.
• Documentation follow-up: create the missing docs/playbooks/reso-import-post-incident-verify.md (or fix the incident issue’s reference) as part of R0-2 so the naming is consistent.

9. Files-to-change signature (planning artifact; real edits happen in the §7 issues)

This plan itself only adds docs/plans/plan-4132-w3-w4-monitoring-strategy.md. The downstream issues will touch, per row:

• R0-1: .github/workflows/secrets-sync.yml (W3 + W4 job post-deploy step only)
• R0-2/R0-3: monitoring/exporters/reso-import-exporter/{app.py,Dockerfile,requirements*.txt,tests/}, bms-1/docker-compose.yml, bms-1/.env.example, monitoring/prometheus/prometheus.yml, monitoring/prometheus/rules/reso-import.yml, docs/reso-import-exporter-operations.md
• R1-1: monitoring/prometheus/rules/pinbox24-containers.yml
• R1-2: monitoring/exporters/stack-connectivity-exporter/app.py (extend), stack-connectivity.yml
• R1-3/R2-1/R2-2/R2-3: monitoring/prometheus/rules/consumer-synthetic.yml, monitoring/prometheus/blackbox.yml + prometheus.yml, n8n workflow export

10. Regression risks (carry into each downstream PR)

• Post-sync gate false-fails a good deploy (transient rs0 election, PM2 not yet warm) → bounded retries (e.g. 3× over 30s) before the gate declares failure; the gate must not make secrets-sync flappy. Distinguish auth failure (real, block) from timeout/refused (transient, retry).
• Import synthetic false-alarms on a legitimately quiet window (no new RESO records overnight) → SLA must match real import cadence; consider a business-hours-aware threshold. Start permissive, tighten with data.
• Silent-green from a cached exporter connection → fresh short-lived client per check, gauges init at import, *ExporterDown alert on up==0 (the #4018 §8 rule, reused verbatim).
• S3 write-path leaves litter → the round-trip must delete its probe object; use a dedicated .monitoring/probe-<ts> key prefix and a lifecycle-safe cleanup.
• Widening the SOPS reader set on bms-1 by one more exporter (reso-import-exporter reads pinbox24-w3.env) — architecturally identical to mailgun-pipeline-exporter; sys-security signs off in /review-plan (same call as #4018 Decision A).
• Alert volume — all new rules route through the existing batch_stage_alerts() batching (#3919); no rule may POST GitHub/Discord directly.

11. Manual test checklist (for the downstream implementation PRs)

• Post-sync gate: intentionally deploy a known-bad app cred to a staging W3 (bms-3), confirm the sync job fails with a Discord embed + GH issue, and no green “sync succeeded”.
• Post-sync gate: good cred → job passes, gate prints ok only (no secret value in the log).
• Import synthetic: with a fresh “import udany” record present, reso_import_last_success_timestamp_seconds advances; artificially age it past SLA → RESOImportStalled fires and batches to one GH issue.
• Import synthetic: docker stop reso-import-exporter → RESOImportExporterDown fires within 5m.
• promtool check rules clean on every new rule file; promtool check config clean.
• Crash-loop rule: docker restart v32-prod-reso a few times → C-1 alert fires.
• No secret value in any commit, workflow log, PR body, Discord embed, or agent-session output — key names and cred_source labels only.
• dev_r_services rows exist for reso-import-exporter (+ any new probe); ops docs written.

12. Iteration loop

1. This plan is posted as the ## Code-change-design comment on #4132 (same worker run) and PR’d to main as docs/plans/plan-4132-w3-w4-monitoring-strategy.md.
2. /review-plan 4132 — Haiku review, ≤3 cycles.
3. On approval, the §7 rows (R0-1 first) are filed as implementation issues by a follow-up run and dispatched by weight; P0 items ship before P1/P2.
4. Merge to main — no dev branch (p24-infra ships from main, CLAUDE.md §Branching).

---

Written by dev worker session plan/4132-w3-w4-monitoring-plan on bms-4, 2026-07-14. Plan only — no implementation. Builds on and stays disjoint from #4018.