p24-infra Docs

hourly-triage-outage

8 min read

Playbook: Nightly Triage Outage

Migration note (2026-06-25): the DevOps triage moved hourly → nightly (now runs daily at 20:00 UTC via run-nightly-triage.sh, skill nightly-devops-triage). Log is /var/log/nightly-devops-triage.log; lock is /home/claude-runner/nightly-devops-triage.lock. Alerts are now NightlyTriage*. The pushed metric name p24_hourly_triage_last_run_timestamp is retained for continuity.

Trigger: NightlyTriageMissedRun fires (critical) — no triage run for >26h.

Quick confirmation

Do this first — it costs one query and rules out the most common false alarm. The alert reads dev_r_scheduled_runs; the pushgateway metric is written independently, so if it shows a recent success the triage ran and only its audit write failed → skip straight to Cause 7, and leave cron and the lock file alone.

# From vps-i1 — ground truth for "did the triage actually run?"
curl -s 'http://localhost:9090/api/v1/query?query=p24_hourly_triage_last_run_timestamp' \
  | jq -r '.data.result[] | "\(.metric.status) \(.value[1]|tonumber|todate)"'

Then, for a genuine outage (Causes 1–6):

# Is cron running?
ssh root@54.36.123.110 'systemctl status cron --no-pager | head -5'
 
# When did the triage last run?
ssh root@54.36.123.110 'tail -20 /var/log/nightly-devops-triage.log'
 
# Is there a stale lock?
ssh root@54.36.123.110 'ls -la /home/claude-runner/nightly-devops-triage.lock 2>/dev/null && echo EXISTS || echo "no lock"'
 
# Is a triage currently running?
ssh root@54.36.123.110 'pgrep -a -u claude-runner | grep triage'

Cause 1: Stale lock file

Symptom: Cron fires (appears in journalctl -u cron), but log has no new entries; cron session closes within 1 second.

Why: If the lock file is owned by root, exec 200>/home/claude-runner/nightly-devops-triage.lock fails → set -e kills the script before EXIT trap registers.

Fix:

ssh root@54.36.123.110 'rm -f /home/claude-runner/nightly-devops-triage.lock && echo "Cleared"'

Then trigger a manual run:

ssh root@54.36.123.110 'su -s /bin/bash claude-runner -c "bash /opt/p24-infra/scripts/run-nightly-triage.sh" >> /var/log/nightly-devops-triage.log 2>&1 &'

Prevention: Lock file path is /home/claude-runner/nightly-devops-triage.lock — owned by claude-runner. Only root-run triage scripts could recreate this. Avoid running the triage script as root manually.

Cause 2: Claude runner auth expired (401)

Symptom: Log shows Failed to authenticate. API Error: 401 Invalid authentication credentials immediately after Starting nightly-devops-triage.

Fix: Run the re-auth script from your local workstation:

python d:\tmp\reauth-bms4.py

See playbook: GitHub issue #981.

Cause 3: ANTHROPIC_API_KEY overrides OAuth

Symptom: Log shows Auth error: Credit balance is too low (or similar API billing error).

Why: ANTHROPIC_API_KEY loaded from bms-4/.env overrides claude-runner’s OAuth credentials. The API key is for n8n workflows, not for claude-runner sessions.

Fix (server patch, until PR #1012 merges):

# Verify the script has the unset line
ssh root@54.36.123.110 'grep "unset ANTHROPIC_API_KEY" /opt/p24-infra/scripts/run-nightly-triage.sh'
 
# If missing, add it (run-nightly-triage.sh line before export HOME=):
# Manual edit or re-deploy from PR #1012

Permanent fix: PR #1012 adds unset ANTHROPIC_API_KEY before the auth check.

Cause 4: Shell-special chars in .env values break source

Symptom: Log shows line N: SomeToken: command not found errors; triage exits before writing the “Starting” message.

Why: set -a; source bms-4/.env interprets &, |, ; etc. in password values as shell operators.

Fix (server patch):

# Deploy the safe-parser version of the script
scp scripts/run-nightly-triage.sh root@54.36.123.110:/opt/p24-infra/scripts/run-nightly-triage.sh

Permanent fix: PR #1012 replaces source with a data-safe key=value parser.

Cause 5: socat-supabase zombie (false positive cascade)

Symptom: HourlyTriageMissedRun fires alongside N8nHighFailureRatio, N8nWorkflowPersistentlyFailing (mezmo-alert-router, WAHA Monitor), and ReportNotGenerated — all at once. N8n itself is running but workflow failure rate is >30%.

Why: socat-supabase.service on bms-4 is in a crash loop (Address already in use on port 15432). A zombie socat process holds the TCP LISTEN socket, preventing new socat instances from starting. n8n workflows that connect to Supabase via direct PostgreSQL (port 15432) all fail, including the hourly triage trigger.

Fix: See playbook socat-supabase-zombie-crash-loop.md. TL;DR:

ZOMBIE_PID=$(ss -tlnp | grep 15432 | grep -oP 'pid=\K[0-9]+')
kill $ZOMBIE_PID && sleep 1 && systemctl restart socat-supabase

Triage is NOT broken — it will recover automatically once socat is fixed.

Cause 6: Log file owned by root (permissions denied)

Symptom: Cron fires as claude-runner, but the log file /var/log/nightly-devops-triage.log is owned by root (e.g. someone ran the script as root manually). The log contains only garbage (directory listing of /root/*) or is empty. No Starting nightly-devops-triage line appears.

Why: set -euo pipefail causes the script to abort on the first >> $LOG_FILE write that fails with permission denied. The EXIT trap is not yet registered at that point, so no cleanup or pushgateway push happens either. The garbage content in the log comes from cron’s stdout/stderr redirect when the >> shell redirect in the user crontab encounters the unwritable file — the shell produces an error message that includes the current working directory listing.

Fix:

ssh root@54.36.123.110 'chown claude-runner:claude-runner /var/log/nightly-devops-triage.log && chmod 664 /var/log/nightly-devops-triage.log'

Then trigger a manual run to verify:

ssh root@54.36.123.110 'su -s /bin/bash claude-runner -c "/opt/p24-infra/scripts/run-nightly-triage.sh" &'
# After ~30s check:
ssh root@54.36.123.110 'grep "Starting nightly" /var/log/nightly-devops-triage.log | tail -3'

Prevention (PR #1817): run-nightly-triage.sh now checks log file writability at the top (before set -euo pipefail can kill it) and attempts chown/fallback to /tmp. Running the triage script as root manually remains unsafe — always use su -s /bin/bash claude-runner -c ....

Cause 7: Triage ran fine — the audit-log write failed (false “missed run”)

Symptom: health-check.yml → agent-activity fails with ✗ Agent activity: last nightly triage NNh ago (> 26h) / reason=stale_run, or NightlyTriageMissedRun fires — but the log on bms-4 shows a normal, complete run.

This is the one cause where nothing is actually down. The detector reads a single source, dev_r_scheduled_runs.started_at. If the triage runs but cannot record the run, the check sees a stale row and blames a triage outage. Before #4611 both writes were curl -s … > /dev/null || true, so a 401 was indistinguishable from a 201 and the failure was completely silent.

Confirm in one command — is the triage actually alive?

The pushgateway push in _cleanup is independent of Supabase, so it is the ground truth:

# On vps-i1 (Prometheus scrapes bms-4's pushgateway — no SSH hop needed)
curl -s 'http://localhost:9090/api/v1/query?query=p24_hourly_triage_last_run_timestamp' \
  | jq -r '.data.result[] | "\(.metric.status) \(.value[1]|tonumber|todate)"'
  • Recent timestamp with status=success → the triage is healthy; only the write path is broken. Do not touch cron or the lock file. Go to the fix below.
  • Stale or missing timestamp → a genuine outage. Use Causes 1–6.

Cross-check the table itself:

KEY=$(grep -m1 '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/monitoring/.env | cut -d= -f2-)
curl -s 'https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/dev_r_scheduled_runs?job_name=eq.nightly-devops-triage&order=started_at.desc&limit=3&select=started_at,status' \
  -H "apikey: $KEY" -H "Authorization: Bearer $KEY" | jq -c '.[]'
unset KEY

A gap in this list on a night the pushgateway recorded success confirms the diagnosis.

Why it happens: the deployed SUPABASE_SERVICE_ROLE_KEY on bms-4 (/opt/p24-infra/bms-4/.env) is stale, revoked, or unset, so the PostgREST write 401s. Standing symptom: #1329. A rotation window is the classic trigger — #4611 was caused by the #4583 rotation deploying a bad key at 19:33Z, 27 min before the 20:00Z run, later rolled back by #4624.

Fix: re-deploy a current key, then confirm the next run records:

gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=bms-4
# Verify the key on bms-4 works WITHOUT printing it:
ssh root@54.36.123.110 'KEY=$(grep -m1 "^SUPABASE_SERVICE_ROLE_KEY=" /opt/p24-infra/bms-4/.env | cut -d= -f2-); \
  curl -s -o /dev/null -w "%{http_code}\n" "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/dev_r_scheduled_runs?limit=1" \
  -H "apikey: $KEY"; unset KEY'   # PLAYBOOK: hourly-triage-outage.md

200 = fixed. 401 = the key in SOPS is itself wrong → secret-manager must regenerate it.

Note the check self-heals as soon as one run records successfully — the age comparison is against the newest row, so the alert clears at the next 20:00Z run without a manual reset.

Prevention (#4611): run-nightly-triage.sh now captures %{http_code} on both dev_r_scheduled_runs writes and calls _audit_write_failed() on any non-2xx or missing key — logging the status, sending the standard Discord embed, and filing one deduplicated GH issue that names the pushgateway cross-check. The write stays non-fatal (the triage still runs), it is just no longer silent. Regression test: scripts/tests/test_triage_audit_write_failure.py.

Detector cross-check (#4639): health-check.yml now cross-checks the pushgateway before declaring an outage. A standalone triage-metric job on the [self-hosted, bms4] runner reads p24_hourly_triage_last_run_timestamp{status="success"} from bms-4’s local pushgateway (localhost:9091), and health-report re-labels a stale_run/no_run whose triage actually ran (fresh success sample within the 26h window) as reason=audit_write_broken. That branch:

  • files a 🟠 (lower-severity) [agents] issue instead of the 🔴 outage issue, and is dropped from the overall health-failed set, so it no longer raises the red server-down issue;
  • its issue body says the triage is alive and points here (Cause 7) plus the bms-4 SUPABASE_SERVICE_ROLE_KEY distribution (#1329) — not at cron/lock diagnosis.

A genuine outage (pushgateway also stale/missing, or the metric job unreachable) still fails agent-activity exactly as before → Causes 1–6.

Escalation

If none of the above apply: check journalctl -u cron --since "1 hour ago" on bms-4 and look for errors at the cron level. Verify /etc/cron.d/p24-infra-nightly-triage exists (entry: 0 20 * * * claude-runner /opt/p24-infra/scripts/run-nightly-triage.sh) and the PATH includes /usr/local/bin.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="hourly-triage",
    result="success",  # "success" | "failed" | "skipped"
    detail="Hourly triage job outage resolved — cron/service restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'hourly-triage', 'success', 'Hourly triage job outage resolved — cron/service restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks