Incident Record: NC-Alert Dispatch Outage (2026-07-02)

Severity: P1 — Critical alert issues unprocessed, two EndpointDown alerts firing unacknowledged Status: Under investigation (as of 2026-07-02) Incident window: 2026-07-01 ~20:23Z through 2026-07-02 (ongoing at time of writing) Discovered by: Manual GitHub issue query during p24-infra session on 2026-07-02 Runbook: docs/playbooks/nc-alert-dispatch-outage.md

Summary

The nc-alert dispatch pipeline failed in three cascading layers on the night of 2026-07-01/02, leaving 7 alert issues unprocessed for 6+ hours. Two EndpointDown alerts (Pinbox24 API and OVH DBaaS TCP) fired continuously from ~01:31Z with no automated response. The failure was invisible to monitoring because nc-alert-instant-dispatch.yml treated CF Worker HTTP 500 responses as a non-fatal ::warning::, causing GH Actions runs to appear green while dispatch was completely broken. No Discord alerts or GH issue notifications were generated for the dispatch failure itself.

Timeline (UTC)

Time	Event
2026-07-01 ~20:23Z	`nc-alert-orchestrator` spawn failure on unknown worker processing issue #2492 (Pinbox24NoLogs). Exit code 255, no session created. Queue-dispatcher auto-rerouted to bms-4.
2026-07-01 ~20:23–20:30Z	bms-4 reroute also silently fails. Issue #2492 left in Triage, unprocessed. No Discord/GH alert fired.
2026-07-02 ~01:31Z	`p24-meta-dispatcher` CF Worker begins returning HTTP 500 on all `POST /queue-issue` calls. Root cause unknown at time of writing (CF Worker exception, likely Supabase auth or code-level error).
2026-07-02 01:31Z onward	`nc-alert-instant-dispatch.yml` fires multiple times (triggered by new alert labels) but each run exits 0 (`::warning::` does not set failure). Zero Discord alerts, zero GH issues created for the dispatch failure.
2026-07-02 (overnight)	vps-i1 heartbeat goes stale (issue #2517). Queue-dispatcher auto-cancels `nc-alert-batch` rows destined for vps-i1 — second layer of blocking stacks on top of the CF Worker 500.
2026-07-02 ~morning	Alert issues #2517–#2522 created by Alertmanager webhooks and labeled `p24-infra-nc-alert`. All 7 issues sit in Triage with no orchestrator activity.
2026-07-02 (this session)	Manual `gh issue list` query reveals the backlog. Incident discovered. Documentation begins.

Impact

Alert issues unprocessed (7 total)

Issue	Alert name	Status at discovery
#2492	Pinbox24NoLogs	Unprocessed since 2026-07-01 ~20:23Z (12+ h)
#2517	vps-i1 heartbeat stale	Unprocessed; also blocked dispatch as secondary cause
#2518	EndpointDown — Pinbox24 API	CRITICAL — firing since ~01:31Z, ~8+ h unacknowledged
#2519	EndpointDown — OVH DBaaS TCP	CRITICAL — firing since ~01:31Z, ~8+ h unacknowledged
#2520	(alert issue)	Unprocessed
#2521	(alert issue)	Unprocessed
#2522	(alert issue)	Unprocessed

Services affected

Pinbox24 API (#2518): EndpointDown alert firing; production availability unclear. Pinbox24 serves fleet-tracking customers — any actual downtime has direct customer impact.
OVH DBaaS TCP (#2519): EndpointDown alert firing; MongoDB connectivity from bms-4 may be impaired if the DBaaS TCP probe is a live dependency.
nc-alert dispatch pipeline: Fully blocked for ~8h. Any new Alertmanager firing during this window received no automated response.
ALL dispatch (dev-issue and nc-alert-batch): CF Worker POST /queue-issue returning 500 blocked every job type, not just nc-alert. Any issue queued to the worker queue during this window was not inserted.

Human notification gap

Zero Discord embeds sent for any of the 7 alert issues.
Zero GH bug issues created for the CF Worker failure itself.
GH Actions UI showed all NC-Alert Instant Dispatch runs as green (conclusion: success).
The outage was only discovered by an active human operator running a manual query.

Root Cause Chain

1. nc-alert-orchestrator spawn failure (vps-i1, exit=255)
       ↓  auto-reroute to bms-4 also failed (silent)
2. CF Worker p24-meta-dispatcher started returning HTTP 500
   on POST /queue-issue (all job types)
       ↓
3. nc-alert-instant-dispatch.yml uses ::warning:: (not exit 1)
   for non-201 HTTP responses → GH run shows green
       ↓
4. No Discord embed / no GH issue created for dispatch failure
       ↓
5. vps-i1 heartbeat stale → dispatcher cancels nc-alert-batch rows (second blocking layer)
       ↓
RESULT: 7 alert issues unprocessed for 6–12h; 2 critical EndpointDown alerts
        firing with no automated response; no human notification generated

Contributing factors

Silent masking in workflow (::warning:: vs exit 1): The primary reason the failure was not detected for 6+ hours. All other automation correctly observed the dispatch failure — only the notification step was missing.
nc-alert-batch hard-affinity to vps-i1: When vps-i1 goes stale, nc-alert-batch jobs cannot fall over to bms-4, adding a second failure surface.
CF Worker not monitored by Prometheus: p24-meta-dispatcher CF Worker exceptions are not captured by any Prometheus metric or Alertmanager rule. Agent reports are the only discovery mechanism for this failure mode.
Initial orchestrator spawn failure (#2492 at 20:23Z): The primary trigger that escalated into the CF Worker failure. The relationship between the spawn failure and the CF Worker 500 is not fully established — may be coincidental timing or may indicate a resource pressure event.

Fixes Applied

Immediate (this session)

Incident documentation: this record + docs/playbooks/nc-alert-dispatch-outage.md created so future on-call agents can diagnose and recover within minutes instead of re-deriving from scratch.
Manual triage of stalled issues: Issues #2492, #2517–#2522 identified. Critical issues #2518 and #2519 (EndpointDown) flagged for priority human review.

Required follow-up (open items)

PR / Issue	Description	Priority
#2518	EndpointDown — Pinbox24 API: verify current Pinbox24 API status; close or escalate	P0
#2519	EndpointDown — OVH DBaaS TCP: verify OVH DBaaS / MongoDB TCP connectivity; close or escalate	P0
CF Worker investigation	Root-cause the `p24-meta-dispatcher` HTTP 500; check Wrangler logs / CF dashboard; fix or rollback	P1
Workflow hardening PR	Fix `nc-alert-instant-dispatch.yml` to use `exit 1` + Discord embed on non-201 responses	P1
vps-i1 heartbeat fix	Restore `p24-heartbeat.timer` on vps-i1 (see #2517)	P1
nc-alert-batch failover	Allow bms-4 as fallback target when vps-i1 is absent	P2
CF Worker monitoring	Add Prometheus alert or synthetic probe on CF Worker health	P2
Re-process #2492, #2520–#2522	Once pipeline is restored, re-label to trigger orchestrator	P2

Lessons Learned

1. Warning-level logging is not a failure signal

::warning:: in GitHub Actions sets a yellow annotation but exits 0 — the run appears successful in the UI, in status checks, and in notification webhooks. Any infrastructure operation whose failure must be detected must use exit 1 or equivalent. This incident cost 6+ hours of undetected downtime because the GH Actions job looked healthy.

Action: Audit all GH Actions workflows in radieu/p24-infra for ::warning:: on critical operations (dispatch, deploy, sync). Replace with exit 1 + Discord alert where the failure would block an operational pipeline.

Prometheus scrapes VPS metrics but not Cloudflare Worker error rates. A CF Worker returning 500 on every request generates no alert — the failure exists only in CF’s internal logs and in the error responses received by callers.

Action: Add a synthetic health-check cron (e.g., from bms-4) that POST /queue-issue-probes the CF Worker every 5 minutes and pushes the HTTP status to Pushgateway. Alertmanager can then fire a DispatcherWorkerUnhealthy alert.

3. Single-node nc-alert affinity amplifies blast radius

nc-alert-batch jobs being hard-affined to vps-i1 means a vps-i1 issue (which is independently a separate failure class) immediately blocks all alert processing. Alert processing should be the most resilient pipeline — it should survive the outage of any single node.

Action: Implement bms-4 as a fallback for nc-alert-batch so jobs can be processed even when vps-i1 is absent.

4. Cascade detection requires correlation, not just individual monitors

Each individual failure (spawn failure, CF Worker 500, vps-i1 stale heartbeat, dispatch silently green) was individually observable in isolation, but no single alert correlated them into “the nc-alert pipeline is down.” The symptom that actually surfaced the problem was a human operator noticing old, uncommented alert issues.

Action: Add a dedicated NCAlertPipelineStalled Prometheus alert: if any p24-infra-nc-alert issue is older than 15 minutes without an orchestrator comment, fire a Discord critical alert. This transforms the detection signal from “human notices stale issues” to “automated alert within 15 minutes of first failure.”

References

Playbook: docs/playbooks/nc-alert-dispatch-outage.md
Related playbooks:
GitHub workflow: .github/workflows/nc-alert-instant-dispatch.yml
Affected issues: #2492, #2517, #2518, #2519, #2520, #2521, #2522

p24-infra Docs

Explorer

2026-07-02-nc-alert-dispatch-outage