p24-infra Docs

nc-alert-dispatch-outage

9 min read

Playbook: NC-Alert Dispatch Outage

Failure class: The nc-alert pipeline stops processing alert issues. Alert issues accumulate in the Triage milestone with the p24-infra-nc-alert label but receive no orchestrator comments and are never resolved.

First seen: 2026-07-01/02. See incident record: docs/incidents/2026-07-02-nc-alert-dispatch-outage.md

1. Recognizing This Failure Class

Primary symptoms

  • Prometheus / Alertmanager shows alert rules FIRING, but the corresponding GitHub issues never get a comment from an orchestrator agent (no <!-- nc-alert-orchestrator --> block).
  • gh issue list --repo radieu/p24-infra --label p24-infra-nc-alert --state open returns issues older than ~10 min that are still in the Triage milestone.
  • GH Actions shows NC-Alert Instant Dispatch runs that completed with exit 0 but whose logs contain ::warning::Dispatch returned HTTP 5xx — the run appeared green despite the CF Worker returning a server error.
  • No #infra-errors Discord embed was sent for the failed dispatch (silent failure).

Secondary symptoms

  • vps-i1 heartbeat stale (vps-i1-heartbeat metric not updated for >5 min), causing the queue-dispatcher on bms-4 to auto-cancel any nc-alert-batch jobs destined for vps-i1.
  • dev_r_worker_queue rows for job_type = nc-alert-batch pile up with status = cancelled.
  • p24-meta-dispatcher CF Worker returns 500 on all POST /queue-issue calls.

2. Confirmation Commands

Run these from your Windows dev machine or from a connected terminal session.

2a — Check unprocessed alert issues

gh issue list --repo radieu/p24-infra --label p24-infra-nc-alert --state open --json number,title,createdAt,milestone | ConvertFrom-Json | Format-Table number,title,createdAt

Issues older than 10 minutes with no milestone progression signal a stalled pipeline.

2b — Check CF Worker health

# Load QUEUE_API_KEY silently
$env:QUEUE_API_KEY = (Get-Content "C:\code_2026\p24-infra\secrets\.env.local" -ErrorAction SilentlyContinue | Select-String "^QUEUE_API_KEY=").ToString().Split("=",2)[1].Trim()
# If not in .env.local, decrypt from SOPS:
# $env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# $env:QUEUE_API_KEY = (sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops | Select-String "^QUEUE_API_KEY=").ToString().Split("=",2)[1]
 
$response = Invoke-WebRequest -Uri "https://p24-meta-dispatcher.radieu.workers.dev/queue-issue" `
  -Method POST `
  -Headers @{ "Authorization" = "Bearer $env:QUEUE_API_KEY"; "Content-Type" = "application/json" } `
  -Body '{"issue_number":0,"repo":"radieu/p24-infra","job_type":"nc-alert-batch","weight":"light"}' `
  -UseBasicParsing -ErrorAction SilentlyContinue
Write-Host "HTTP: $($response.StatusCode)"
$env:QUEUE_API_KEY = ""

Expected: 201. If 500, the CF Worker itself is down — see §4.1.

2c — Check vps-i1 heartbeat (Supabase)

# On bms-4 or any server with psql access via socat tunnel
psql "host=localhost port=15432 user=postgres sslmode=disable" -c \
  "SELECT server_label, last_heartbeat_at, NOW() - last_heartbeat_at AS age, enabled
   FROM dev_r_server_capacity WHERE server_label IN ('vps-i1','bms-4');"

If vps-i1 age is > 5 min and nc-alert-batch jobs are hard-affined to vps-i1, the dispatcher will cancel them. See §4.3.

2d — Check queue state

-- On Supabase (via psql on bms-4 or Management API)
SELECT status, job_type, COUNT(*)
FROM dev_r_worker_queue
WHERE job_type IN ('nc-alert-batch','nc-alert-instant')
GROUP BY status, job_type
ORDER BY COUNT(*) DESC;
 
-- Cancelled jobs detail
SELECT id, status, error_message, created_at
FROM dev_r_worker_queue
WHERE job_type = 'nc-alert-batch' AND status = 'cancelled'
ORDER BY created_at DESC LIMIT 10;

2e — Check recent NC-Alert Instant Dispatch GHA runs

gh run list --repo radieu/p24-infra --workflow "NC-Alert Instant Dispatch" --limit 10

Look for runs that completed with success but dispatched to a 500 response (check the logs: gh run view <run-id> --log).

2f — Check nc-alert-orchestrator spawn on vps-i1

ssh root@217.154.82.162 'journalctl --since "2 hours ago" | grep -i "nc-alert\|spawn\|orchestrat" | tail -40'

3. Failure Modes and Fixes

3.1 CF Worker p24-meta-dispatcher returning 500

Cause: The Cloudflare Worker p24-meta-dispatcher threw an unhandled exception. Common causes:

  • Supabase SUPABASE_SERVICE_ROLE_KEY secret rotated in the Worker’s environment but not updated in the CF Worker secrets (Worker uses the old key → Supabase rejects with 401 → Worker propagates as 500).
  • Worker code deployed with a bug (check CF dashboard deploy history).
  • Supabase project unavailable (transient).

Diagnose:

# Check CF Worker error logs via Wrangler (from any machine with Wrangler CLI)
npx wrangler tail p24-meta-dispatcher --format pretty 2>&1 | head -50
 
# Or check Cloudflare dashboard:
# dash.cloudflare.com → Workers & Pages → p24-meta-dispatcher → Logs

Fix:

  1. If Supabase key mismatch: update SUPABASE_SERVICE_ROLE_KEY in the CF Worker secrets: 
    npx wrangler secret put SUPABASE_SERVICE_ROLE_KEY --name p24-meta-dispatcher
# (paste the value from secrets/monitoring.env.sops — do NOT display in chat)
  2. If bad deploy: roll back via CF dashboard (Workers & Pages → p24-meta-dispatcher → Deployments → Rollback).
  3. If Supabase transient: wait 5 minutes and retry.

Verify: repeat the probe in §2b and confirm 201.

3.2 vps-i1 heartbeat stale — dispatcher cancels nc-alert-batch jobs

Cause: vps-i1 p24-heartbeat cron stopped updating dev_r_server_capacity. The queue-dispatcher treats vps-i1 as absent and cancels any jobs assigned to it instead of holding them.

Diagnose:

ssh root@217.154.82.162 'systemctl status p24-heartbeat.timer --no-pager'
ssh root@217.154.82.162 'journalctl -u p24-heartbeat.service --since "30 min ago" --no-pager | tail -20'

Fix:

# Restart the heartbeat timer
ssh root@217.154.82.162 'systemctl restart p24-heartbeat.timer && systemctl status p24-heartbeat.timer --no-pager'
 
# Confirm heartbeat row is fresh (within 2 min)
# (check Supabase dev_r_server_capacity — see §2c)

If vps-i1 is genuinely down (OOM, kernel panic, etc.), see docs/playbooks/vps-i1-outage.md for the full recovery procedure. Until vps-i1 is back, nc-alert-batch jobs will continue to be cancelled — escalate if vps-i1 is unreachable.

Enable bms-4 as temporary nc-alert target (if vps-i1 will be down > 30 min):

-- Re-route nc-alert-batch to bms-4 temporarily
-- (This assumes nc-alert-batch is not hard-coded to vps-i1 in the dispatcher logic)
-- Check first:
SELECT server_label, nc_alert_enabled, enabled FROM dev_r_server_capacity;
-- If nc_alert_enabled column exists and vps-i1 is the only one set:
UPDATE dev_r_server_capacity SET nc_alert_enabled = true WHERE server_label = 'bms-4';

Note: If nc-alert-batch is hard-affined to vps-i1 via logic in queue-dispatcher-loop.py rather than via a DB column, a code change is needed — file an issue and use a manual workaround (§3.4) while that is resolved.

3.3 nc-alert-orchestrator spawn failure on vps-i1 (exit=255, no session)

Cause: spawn-worker.sh on vps-i1 failed to create a systemd-run cgroup session. Common causes: vps-i1 OOM at spawn time, systemd unit limit exhausted, or the orchestrator worker binary (claude) not found on PATH for claude-runner.

Diagnose:

ssh root@217.154.82.162 'journalctl -u "spawn-worker-*" --since "2 hours ago" --no-pager | tail -40'
ssh root@217.154.82.162 'systemctl --user -M claude-runner@ list-units --state=failed | head -20'
ssh root@217.154.82.162 'free -h && df -h / /var'

Fix:

# If OOM: check and restart any crashed services
ssh root@217.154.82.162 'systemctl restart p24-oom-killer-monitor.service 2>/dev/null; true'
 
# If systemd unit limit: reset failed units
ssh root@217.154.82.162 'systemctl reset-failed && echo "done"'
 
# Re-queue the stuck alert issues manually (see §3.4 manual dispatch)

If the spawn failure is systematic (not transient), check docs/playbooks/vps-i1-crash-loop-recovery.md and docs/playbooks/spawn-worker-sudo-scope-failure.md.

3.4 Manual re-dispatch of stalled alert issues

Use this when the automated pipeline is blocked and you need to process specific alert issues immediately.

Step 1 — Identify stalled issues:

gh issue list --repo radieu/p24-infra --label p24-infra-nc-alert --state open --json number,title,createdAt | ConvertFrom-Json | Sort-Object createdAt

Step 2 — Re-queue via CF Worker (once CF Worker is healthy again):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:QUEUE_API_KEY = (sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops | Select-String "^QUEUE_API_KEY=").ToString().Split("=",2)[1]
 
# Dispatch the nc-alert-batch orchestrator
$body = '{"issue_number":0,"repo":"radieu/p24-infra","job_type":"nc-alert-batch","weight":"light","role":"orchestrator"}'
$r = Invoke-WebRequest -Uri "https://p24-meta-dispatcher.radieu.workers.dev/queue-issue" `
  -Method POST `
  -Headers @{ "Authorization" = "Bearer $env:QUEUE_API_KEY"; "Content-Type" = "application/json" } `
  -Body $body -UseBasicParsing
Write-Host "Dispatched: HTTP $($r.StatusCode) — $($r.Content)"
$env:QUEUE_API_KEY = ""

Step 3 — If CF Worker is still down, spawn a background agent directly:

# Open a separate Claude Code session and run the nc-alert-orchestrator skill manually:
# /nc-alert-orchestrator

Or spawn via the Agent tool in the current session targeting the affected issues.

4. Prevention Notes

4.1 HTTP 5xx masking in nc-alert-instant-dispatch.yml

Current (broken) behavior: The workflow uses ::warning:: (which does not set exit 1) when the CF Worker returns a non-201 status. The GH Actions run appears green even when dispatch fails.

Required fix (PR pending):

# In .github/workflows/nc-alert-instant-dispatch.yml
# Replace the ::warning:: line with a proper error:
if [ "$HTTP_CODE" != "201" ] && [ "$HTTP_CODE" != "200" ]; then
  echo "::error::Dispatch returned HTTP $HTTP_CODE — creating alert issue"
  # Trigger Discord + GH issue creation for the dispatch failure
  curl -s -X POST "$DISCORD_WEBHOOK_URL" \
    -H "Content-Type: application/json" \
    -d "{\"embeds\":[{\"title\":\"🔴 NC-Alert Dispatch FAILED\",\"color\":15158332,\"description\":\"CF Worker returned HTTP $HTTP_CODE. Alert issues may be unprocessed.\"}]}"
  exit 1
fi

After this fix is merged, any CF Worker 500 will turn the GH Actions run red AND trigger a Discord alert — eliminating the 6-hour blind spot.

4.2 nc-alert-batch hard-affinity to vps-i1

If nc-alert-batch is hard-affined to vps-i1 in the dispatcher, a vps-i1 outage silently stalls all alert processing. Consider:

  • Adding bms-4 as a fallback target for nc-alert-batch when vps-i1 is absent.
  • OR: implementing a Prometheus alert on nc-alert-batch queue depth > 0 for > 10 minutes.

4.3 No Discord alert on silent dispatch failure

The standard error notification standard (Discord embed + GH issue) is not triggered when the GH workflow exits 0 despite a failed dispatch. Once the workflow hardening PR (§4.1) is merged, this is resolved.

5. Escalation Path

ConditionActionOwner
CF Worker 500 persists > 30 minCheck Cloudflare status page; file bug issue in p24-infra; use manual re-dispatch (§3.4)p24-infra Claude
vps-i1 unreachable > 30 minFollow docs/playbooks/vps-i1-outage.md; route nc-alert-batch to bms-4p24-infra Claude
Critical alerts (#EndpointDown, servicedown) unprocessed > 1hManually triage the alert issues; create human-action issue for owner reviewp24-infra Claude → Human
Spawn failure systematic (exit=255 on all attempts)Check docs/playbooks/spawn-worker-sudo-scope-failure.md; escalate to human-actionp24-infra Claude → Human
CF Worker key rotation neededUse /role-secret-manager to rotate SUPABASE_SERVICE_ROLE_KEY in CF WorkerSecret-manager session
  • docs/playbooks/worker-queue-operations.md — queue stuck jobs, OOM, failover
  • docs/playbooks/vps-i1-outage.md — full vps-i1 recovery procedure
  • docs/playbooks/vps-i1-crash-loop-recovery.md — OOM / crash loop on vps-i1
  • docs/playbooks/queue-dispatch-cross-project.md — CF Worker API reference
  • docs/playbooks/spawn-worker-sudo-scope-failure.md — spawn exit=255 diagnosis
  • docs/playbooks/hourly-triage-outage.md — nightly triage pipeline outage (separate but related)
  • docs/incidents/2026-07-02-nc-alert-dispatch-outage.md — incident record for this failure class

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="nc-alert-dispatch",
    result="success",  # "success" | "failed" | "skipped"
    detail="nc-alert dispatch outage resolved — service restarted and alert routing restored",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'nc-alert-dispatch', 'success', 'nc-alert dispatch outage resolved — service restarted and alert routing restored', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks