hourly-triage-watchdog

Playbook: Nightly Triage Watchdog

Migration note (2026-06-25): the DevOps triage moved hourly → nightly — it now runs daily at 20:00 UTC via run-nightly-triage.sh (cron file /etc/cron.d/p24-infra-nightly-triage, skill nightly-devops-triage). Log: /var/log/nightly-devops-triage.log. Alerts renamed to NightlyTriage*. The pushed metric p24_hourly_triage_last_run_timestamp keeps its name for alert/dashboard continuity. This playbook (filename unchanged) is also referenced by the claude-runner OAuth token-refresh alerts.

What triggers this playbook

• Alert NightlyTriageMissedRun (Prometheus, severity: critical): no p24_hourly_triage_last_run_timestamp metric pushed in the last 26 hours (>2h past the daily 20:00 UTC cadence). Triage cron is not running.
• Alert NightlyTriageStuck (Prometheus, severity: warning): triage started (status=running heartbeat) but has not finished in >2 hours. Likely hung.
• GitHub issue 🔴 [agents] Triage nie uruchomił się... (health-check.yml, every 2h): dev_r_scheduled_runs shows no run in 2h OR In Progress issues not updated in >4h.

Read the Condition: line in the alert first (#4360). The GitHub issue fires on four distinct conditions and the body states which one tripped:

Condition:	Meaning	Where to go
no_run	No nightly-devops-triage row in dev_r_scheduled_runs at all	Step 1 below (SSH bms-4)
stale_run	Last run older than 26h	Step 1 below (SSH bms-4)
audit_write_broken	Triage is alive (pushgateway confirms a recent status="success") — only the dev_r_scheduled_runs write failed	hourly-triage-outage.md Cause 7 — do not SSH cron/lock; fix the bms-4 SUPABASE_SERVICE_ROLE_KEY (#1329)
stalled_issues	Triage is healthy — issues sit in In Progress >4h	§Stalled In Progress issues — do not SSH

Before the #4360 fix the body always claimed a triage outage and pointed at /var/log/hourly-devops-triage.log (a path that has not existed since the 2026-06-25 change), so stalled_issues firings were repeatedly misdiagnosed as a dead cron. Always confirm the cron’s health against dev_r_scheduled_runs before touching bms-4.

---

Stalled In Progress issues (Condition: stalled_issues)

The triage cron ran fine; the backlog is what is stuck. Nothing on bms-4 needs attention.

# Which issues are stalled (>4h without an update), split the way the watchdog splits them?
AGED=$(gh issue list --repo radieu/p24-infra --milestone "In Progress" \
  --json number,updatedAt,title,labels \
  --jq '[.[] | select(.updatedAt < (now - 14400 | todate))]')
 
# Agent-actionable — these are what the alert counts:
echo "$AGED" | jq -r '.[] | select(([.labels[].name] | index("human-action")) | not)
  | "\(.number)\t\(.updatedAt)\t\(.title)"'
 
# human-action — aged but NOT counted (see below):
echo "$AGED" | jq -r '.[] | select([.labels[].name] | index("human-action"))
  | "\(.number)\t\(.updatedAt)\t\(.title)"'

For each agent-actionable one: check whether a worker died mid-job (queue row running/claimed with no progress), whether the PR already merged and the issue just needs closing, or whether it is genuinely blocked on a human. Re-dispatch, close, or move it out of In Progress — the alert clears once none remain stalled.

human-action issues are excluded from the count (#4373)

Since #4373 the watchdog counts only issues an agent can advance. Issues labelled human-action are parked by definition — awaiting a credential rotation, a 2FA prompt, or a provider console — so counting them made this alert re-fire every 2 hours indefinitely. It was also self-referential: the watchdog files its own alert issue with the human-action label into In Progress, so after 4h the alert counted itself and kept its own condition true.

They are not hidden. Aged human-action issues still appear in the step log, in the blocked_count / blocked_list job outputs, and as a dedicated row plus a note in the alert body.

Convention — where a stuck issue belongs:

Situation	Milestone
Agent can still act (re-dispatch, close, finish the PR)	stays in In Progress
Confirmed human-gated (credential rotation, 2FA, provider console, waiting on a person)	move to Blocked + label human-action
Work is complete / PR merged	move to Main and close

Moving a confirmed human-gated issue to Blocked is the correct resolution — it keeps In Progress meaning “an agent is or should be working on this”, which is exactly what this watchdog measures. Do not lower an issue’s priority just to clear the alert; the milestone move is a routing change, not a de-prioritisation.

---

Step 1 — Diagnose

SSH to bms-4 and collect evidence:

ssh ubuntu@54.36.123.110
 
# Is the cron entry there?
sudo -u claude-runner crontab -l
 
# What does the log say?
tail -50 /var/log/nightly-devops-triage.log
 
# Token refresh log
tail -20 /var/log/claude-token-refresh.log
 
# Is a triage process running right now?
ps aux | grep -E 'claude|run-nightly'
 
# Is claude-runner auth valid?
sudo su -s /bin/bash claude-runner -c 'claude -p say-ok'

Check Supabase audit log:

SELECT run_id, started_at, ended_at, status, duration_s, exit_code
FROM dev_r_scheduled_runs
ORDER BY started_at DESC
LIMIT 10;

---

Step 2 — React by failure mode

Cron present but script never runs — missing executable bit (#1790, 2026-06-27)

Symptom: the cron entry exists, the log shows no Starting nightly-devops-triage line for days, dev_r_scheduled_runs has no recent nightly-devops-triage row, and the pushgateway metric is frozen at an old timestamp. The cron invokes the path directly (claude-runner /opt/.../run-nightly-triage.sh), so if the file is not executable cron silently fails to run it — no log, no audit row, no heartbeat.

# Confirm: is the wrapper executable?
stat -c '%A %U:%G' /opt/p24-infra/scripts/run-nightly-triage.sh
# BAD:  -rw-r--r-- root:root        (cron cannot exec it)
# GOOD: -rwxr-xr-x claude-runner:claude-runner
 
# Fix (needs root — the file/dir are root-owned, claude-runner has no sudo on bms-4):
chmod 755 /opt/p24-infra/scripts/run-nightly-triage.sh
chown claude-runner:claude-runner /opt/p24-infra/scripts/run-nightly-triage.sh
# If you only have docker-group access (no sudo), use a privileged container to chmod/chown:
docker run --rm -v /opt/p24-infra/scripts:/x redis:7-alpine \
  sh -c 'chmod 755 /x/run-nightly-triage.sh && chown 1001:1001 /x/run-nightly-triage.sh'

Root cause was a git-tracked mode of 100644 on scripts/run-nightly-triage.sh (the hourly→nightly migration committed it non-executable). The permanent fix is git update-index --chmod=+x so every checkout gets 100755 — landed via the #1790 PR. Also clear the stale frozen series if the metric name was reused: curl -X DELETE http://localhost:9091/metrics/job/hourly-devops-triage.

Cron fires but the run produces ZERO log output — killed in pre-flight (#3491, 2026-07-09)

Symptom: identical to the exec-bit case above (log frozen, no Starting line, pushgateway series stale/gone) but the wrapper IS executable and cron DID invoke it. The distinguishing check is that /var/log/syslog shows the CRON[...] CMD (...run-nightly-triage.sh) line at 20:00, yet the script wrote nothing — not even the Starting line it emits at the end of pre-flight (before the auth check). The process was killed within the first ~1–2 s, before pre-flight finished.

Do not be fooled by an auto-closed NightlyTriageMissedRun issue. The alert clears when the pushgateway series has no data, which also happens when pushgateway restarts (e.g. on a host reboot — it wipes in-memory metrics unless --persistence.file is set). A cleared alert therefore does not prove triage recovered. Always confirm against the log, not the alert state.

# Distinguish "cron never fired" from "cron fired but the run died". claude-runner cannot read
# /var/log/syslog directly and has no journalctl access; on bms-4 use a read-only container
# (docker group, no sudo) to read the host log:
docker run --rm -v /var/log:/hostlog:ro alpine:3 sh -c \
  '{ cat /hostlog/syslog /hostlog/syslog.1 2>/dev/null; zcat /hostlog/syslog.*.gz 2>/dev/null; } \
   | grep -iE "run-nightly-triage|Startup finished|oom-kill|Killed process" | tail -30'
# CRON CMD line present + no Starting line in the triage log ⇒ killed during pre-flight.
 
# Confirm the pushgateway series was wiped (explains a false alert-clear):
curl -s http://localhost:9091/metrics | grep nightly-devops-triage   # empty ⇒ series gone
uptime                                                               # recent boot ⇒ pushgateway restarted
 
# Reproduce pre-flight in a cron-minimal env to prove the script logic is/ isn't the fault
# (temp log + temp lock, stops before the heavy claude skill — no side effects):
awk '/Starting nightly-devops-triage/{print;print "exit 0";exit}{print}' \
  /opt/p24-infra/scripts/run-nightly-triage.sh > /tmp/nlt-pf.sh
sed -i 's#/var/log/nightly-devops-triage.log#/tmp/nlt.log#;s#/home/claude-runner/nightly-devops-triage.lock#/tmp/nlt.lock#' /tmp/nlt-pf.sh
env -i SHELL=/bin/bash PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
  HOME=/home/claude-runner LOGNAME=claude-runner USER=claude-runner bash /tmp/nlt-pf.sh; cat /tmp/nlt.log

Fix / next steps (needs root — an infra-task worker without sudo can only diagnose):

• If a single failure lines up with a reboot (Startup finished near 20:00) → that night was collateral from the reboot; the missed run is recovered by the next nightly run. Re-push nothing — let a real run repopulate the metric.
• If runs die on a stable host (no reboot) → run it interactively at/after 20:00 UTC and watch stderr, which cron discards: sudo -u claude-runner /opt/p24-infra/scripts/run-nightly-triage.sh. Also check journalctl -u cron -S '20:00', dmesg/cgroup-OOM, and any watchdog/pkill that targets claude-runner processes at that hour (bms-4 is a busy multi-agent host — suspect a memory/cgroup kill or a cleanup job racing the launch).
• Never manually fire a full recovery triage to “fix” the metric — it sends outward-facing Telegram/Discord/GitHub notifications and real escalations, and risks duplicating the 20:00 run.

Cron entry missing

# Add all cron entries (idempotent)
# File: /etc/cron.d/p24-claude-agents  (deployed by secrets-sync.yml)
# Or add manually to claude-runner crontab:
crontab -u claude-runner -l | { cat; \
  echo "0 20 * * * /opt/p24-infra/scripts/run-nightly-triage.sh"; \
  echo "30 0 * * * /opt/p24-infra/scripts/p24-infra-nightly.sh"; \
} | sort -u | crontab -u claude-runner -
 
# Token refresh cron (root) — versioned in scripts/cron.d/, deploy via ansible
# claude-runner role OR copy the file directly:
install -m 0644 /opt/p24-infra/scripts/cron.d/claude-token-refresh /etc/cron.d/
install -m 0644 /opt/p24-infra/scripts/cron.d/claude-auth-synthetic-check /etc/cron.d/

The claude-auth-synthetic-check cron (15-min cadence) does a real authenticated ping to api.anthropic.com and pushes p4_claude_auth_valid{machine="…"} to the Pushgateway, so a multi-host refresh_token race (#3803) surfaces within 30 min in Grafana/Alertmanager instead of “user notices the queue is stuck”.

claude-runner auth — confirm WHICH host before anything else (#4470)

ClaudeAuthSyntheticFailing names a host via the machine label on p4_claude_auth_valid. Before 2026-07-22 that label was not trustworthy: every host pushed into the one Pushgateway group {job="claude-auth-synthetic"} (grouping key was job only), so each 15-min push replaced its siblings’ sample. One series existed fleet-wide, wearing whichever machine pushed last — #4470 paged for machine="bms-4" when the failing probe was vps-i1’s. #4470 moved machine into the grouping key (.../job/claude-auth-synthetic/machine/<MACHINE>), so each host now has its own series and its own staleness marker.

Ground truth is always the host’s own files, never the label alone:

# On each claude-runner host — which host is actually 401ing?
tail -5 /var/log/claude-auth-synthetic.log      # "Auth INVALID on <host>" names the real machine
jq . /var/lib/p24/claude-auth-state.json        # consecutive_failures, last_error

A stale un-grouped series can no longer linger: the probe reaps the legacy group on every run. Note the probe covers account-1 only (/home/claude-runner/.claude/.credentials.json) — an ecotrans / claude-runner-2 failure shows up in ClaudeRunnerTokenExpired{account=…}, not here (#4429).

claude-runner auth — access token expired (headless refresh)

The refresh-claude-token.py script runs every 2h and refreshes the access token automatically. If triage failed with 401 and the refresh log shows an error, run manually:

# Manual refresh (runs as root, writes credentials.json as claude-runner)
python3 /opt/p24-infra/scripts/refresh-claude-token.py
 
# Verify
tail -5 /var/log/claude-token-refresh.log
su -s /bin/bash claude-runner -c 'claude -p say-ok'

claude-runner auth — refresh token expired (months, needs browser re-auth)

This happens when refresh-claude-token.py logs "refresh token expired" or exits 1 with HTTP 400/401. A Discord alert and GitHub issue with human-action label are created automatically.

Retired-host suppression (#5678). Both refresh-claude-token.py and claude-auth-synthetic-check.py gate GitHub-issue creation on the host’s dispatch state: gh_issue_unless_exists() first calls _host_dispatch_enabled(MACHINE), a best-effort lookup of dev_r_server_capacity.enabled. On a host that is enabled=false (vps-i1, vps-h1 — retired as Claude worker hosts) the auth condition is permanently non-actionable — nobody re-auths a retired host — so no issue is filed (the run still logs the reason and Discord/Telegram still fire as the visible backstop, mirroring #5592’s non-goals). This is what stops the duplicate-issue re-file loop: the pre-existing open-issue dedupe only suppresses while a duplicate is open, so each manual close used to spawn a fresh one on the next run. The lookup fails OPEN — missing Supabase creds, a network error, or an absent capacity row all treat the host as enabled, so an enabled host (bms-4) whose lookup momentarily fails never goes silent. Only a definitive enabled=false suppresses.

Token lifecycle:

• accessToken (sk-ant-oat01-...) — 8–12h, refreshed headlessly every 2h by refresh-claude-token.py
• refreshToken (sk-ant-ort01-...) — months, stored in credentials.json; only expires rarely

When refresh token is expired — full browser re-auth (minutes, no code needed):

# Start tmux session on the server
ssh root@54.36.123.110 "tmux kill-session -t claude_auth 2>/dev/null; \
  tmux new-session -d -s claude_auth -x 220 -y 50; \
  tmux send-keys -t claude_auth \"runuser -l claude-runner -c 'claude auth login'\" Enter; \
  sleep 12; tmux capture-pane -t claude_auth -p"

Open the URL shown in your regular Chrome (where you are logged into claude.ai). Click Authorize. After redirect to platform.claude.com/oauth/code/callback, copy the full displayed string — format is CODE#STATE (include the # and everything after it).

# Paste full CODE#STATE string (including the # separator and state part)
ssh root@54.36.123.110 "tmux send-keys -t claude_auth 'CODE#STATE_HERE' Enter; \
  sleep 10; tmux capture-pane -t claude_auth -p"
# Expected: "Login successful."
 
# Verify
ssh root@54.36.123.110 "su -s /bin/bash claude-runner -c \"claude -p 'say: ok' --print\""
 
# Cleanup
ssh root@54.36.123.110 "tmux kill-session -t claude_auth 2>/dev/null"

Notes:

• Playwright/MCP browser does NOT work — browser has no claude.ai session and Cloudflare blocks it
• The reauth-bms4.py script requires interactive stdin — use SSH+tmux directly
• Same procedure applies to vps-i1: replace 54.36.123.110 with 217.154.82.162

Triage hung (status=running >2h)

# Find and kill the hung process
ps aux | grep 'run-nightly-triage\|claude.*dangerously'
# kill <pid>
# The cleanup trap in run-nightly-triage.sh will automatically:
#   - UPDATE dev_r_scheduled_runs with status=failed
#   - Push status=failed to pushgateway

Script crashing (check log for errors)

# Check SOPS-deployed env is present
ls -la /opt/p24-infra/bms-4/.env
 
# Re-deploy secrets if missing
# (trigger secrets-sync.yml manually from GitHub Actions)
 
# Manual test run
sudo -u claude-runner /opt/p24-infra/scripts/run-nightly-triage.sh

---

Step 3 — Verify recovery

# After fix, confirm next run completes
tail -f /var/log/nightly-devops-triage.log
 
# Confirm Supabase shows a new row with status=success
# SELECT status, started_at, duration_s FROM dev_r_scheduled_runs
# WHERE job_name = 'nightly-devops-triage'
# ORDER BY started_at DESC LIMIT 3;
 
# Confirm Prometheus metric present in bms-4 pushgateway:
ssh root@54.36.123.110 \
  "curl -s http://localhost:9091/api/v1/metrics | grep p24_hourly_triage"
# Expected: p24_hourly_triage_last_run_timestamp with status=success

---

Audit trail

Every run writes to dev_r_scheduled_runs (Supabase):

• INSERT at start: status=running, started_at=NOW()
• UPDATE at end: status=success|failed, ended_at, duration_s, exit_code

Pushgateway metric: p24_hourly_triage_last_run_timestamp{machine="ns3101999",status="running|success|failed"}

• Pushed by scripts/run-nightly-triage.sh directly to localhost:9091 on bms-4
• Prometheus on vps-i1 scrapes bms-4 pushgateway at 54.36.123.110:9091 (external port, 0.0.0.0 binding)

Prometheus scrapes pushgateway every 60s → alerts fire after 5 min sustained condition.

---

Prevention

• refresh-claude-token.py runs every 2h (root cron) — access token always fresh before triage
• Cron entry documented in script header (line 3)
• Lock file (/home/claude-runner/nightly-devops-triage.lock) prevents overlap runs
• set -euo pipefail + trap EXIT ensures cleanup always runs even on failure
• git pull --ff-only at start keeps skill definition fresh (non-fatal if fails)
• scripts/run-nightly-triage.sh is git-tracked 100755 — keep the executable bit (cron invokes the path directly; a 100644 mode = silent non-run, the #1790 outage). Verify with git ls-files -s.

---

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="hourly-triage-watchdog",
    result="success",  # "success" | "failed" | "skipped"
    detail="Hourly triage watchdog triggered — stale job cleared and triage restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'hourly-triage-watchdog', 'success', 'Hourly triage watchdog triggered — stale job cleared and triage restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''