Playbook: Agent Activity Watchdog Alert Response

Sibling playbook: hourly-triage-watchdog.md covers the Prometheus/metric side (NightlyTriageMissedRun, NightlyTriageStuck) and the token-refresh mechanics. This playbook is the response runbook for the GitHub watchdog issue 🔴 [agents] Triage nie uruchomił się lub issues In Progress bez obsługi raised by health-check.yml every 2h. Start here when that issue fires; cross-reference the sibling for the metric/auth deep-dives.

What triggers this

health-check.yml opens/refreshes the watchdog issue when either condition holds:

No triage run recorded in dev_r_scheduled_runs within the expected window (triage is daily at 20:00 UTC since the 2026-06-25 hourly→nightly migration), OR
In Progress issues not updated in >4h — i.e. agent_tasks rows stuck in a non-terminal state (claimed/in_progress) with no recent heartbeat.

The alert is a symptom. The job of this playbook is to find the underlying cause, apply any autonomous fix, and escalate anything that needs a human — without closing the issue until a triage run actually succeeds.

Step 1 — Diagnose (decide which of the two conditions fired)

A worker running on bms-4 (e.g. as claude-runner-2) cannot sudo to claude-runner, so prefer the Supabase + log evidence below over crontab -l. All Supabase reads use the service-role REST API (SUPA_HB_URL / SUPA_HB_KEY injected by spawn-worker.sh) when the Supabase MCP is unavailable (SUPA_MCP_AVAILABLE=0).

1a — Did the last triage run, and did it succeed?

curl -s "${SUPA_HB_URL}/rest/v1/dev_r_scheduled_runs?select=run_id,job_name,started_at,ended_at,status,exit_code&order=started_at.desc&limit=8" \
  -H "apikey: ${SUPA_HB_KEY}" -H "Authorization: Bearer ${SUPA_HB_KEY}" | python3 -m json.tool

No recent row at all → cron not firing (see hourly-triage-watchdog.md §“Cron entry missing” and §“missing executable bit” — the #1790 silent-non-run mode).
A recent row with status=failed, exit_code=1 → the cron fired but the run died. Read the log to find out why (1b). This is the most common real cause.

1b — Why did the run fail? (read the triage log)

# World-readable; filter out any credential-shaped lines before printing
grep -v -iE 'token|secret|sk-ant|password|apikey' /var/log/nightly-devops-triage.log | tail -40

Match the failure line to a cause:

Log line	Cause	Fix path
`CRITICAL: claude-runner auth check failed` + `You've hit your monthly spend limit`	Monthly spend / usage limit on the `claude-runner` Claude Max subscription	§2a — human action
`Claude auth check: OK` then `You've hit your monthly spend limit` mid-run	Same spend limit, hit partway through a run	§2a — human action
`401` / `Invalid bearer token` / refresh log shows error	Access token expired & headless refresh failed	sibling playbook §“access token expired”
`refresh token expired` / HTTP 400 on refresh	Refresh token expired (rare, months)	sibling playbook §“refresh token expired — browser re-auth”
Python traceback / missing `.env`	Script crash / missing secrets	sibling playbook §“Script crashing”
No `done` line, process still alive >2h	Hung run	sibling playbook §“Triage hung”

1c — Are there stuck In Progress tasks?

curl -s "${SUPA_HB_URL}/rest/v1/agent_tasks?select=id,github_issue_id,status,claimed_by,claimed_at&status=in.(claimed,in_progress,running)&order=claimed_at.asc" \
  -H "apikey: ${SUPA_HB_KEY}" -H "Authorization: Bearer ${SUPA_HB_KEY}" | python3 -m json.tool

Anything claimed_at older than ~10 min with a dead session is a release candidate (Step 2c).

Step 2 — React by failure mode

2a — Monthly spend / usage limit reached (the 2026-06-27 outage)

Symptom (confirmed cause of issue #1537): the nightly triage failed because the claude-runner Claude Max subscription hit its monthly spend limit. The 20:00 UTC run aborted in ~4s at the auth check:

[..T20:00:01Z] Starting nightly-devops-triage (run_id=nightly-triage-...)
[..T20:00:02Z] CRITICAL: claude-runner auth check failed — triage aborted
[..T20:00:02Z] Auth error: You've hit your monthly spend limit · raise it at claude.ai/settings/usage
[..T20:00:05Z] nightly-devops-triage done: status=failed duration=4s exit=1

This is NOT autonomously fixable — a billing/usage cap can only be lifted by the account owner. Distinguish it clearly from an expired token: the refresh token is healthy and the cron is firing; the request is rejected by the API, not unauthenticated.

Confirm it is the spend limit, not a token problem:

# Refresh token health (months-long; should NOT be the cause)
grep -v -iE 'token|secret|sk-ant' /var/log/claude-token-refresh.log | tail -5
# Per-subscription usage snapshot (written by scripts/check-sub-usage.py)
python3 -m json.tool /home/claude-runner/.claude/sub-status.json 2>/dev/null   # claude-runner (may be perm-restricted)
python3 -m json.tool /home/claude-runner-2/.claude/sub-status.json 2>/dev/null # claude-runner-2 fallback sub

Action — escalate, do not retry:

Check for an existing escalation issue before opening a new one — the spend-limit human-action is already tracked by #1873 (CRITICAL: claude-runner monthly spend limit reached on bms-4, labels bug + human-action). If it is open, reference it rather than duplicating.
If no such issue exists, create one with label human-action: 🔴 CRITICAL: claude-runner monthly spend limit reached on <host> — raise limit, body pointing the owner to claude.ai/settings/usage, and send a Discord alert via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (red).
The limit clears when the owner raises it or at the monthly billing reset — triage recovers on the next 20:00 UTC window automatically once capacity returns.

Resilience follow-up (#1792 — subscription balancing): bms-4 runs a second subscription, claude-runner-2, which is independently healthy when claude-runner is capped (verified during this outage: claude-runner-2 was at ~1% 7d utilization while claude-runner was rejected). The durable fix is to route triage/worker jobs to claude-runner-2 when sub-1 is at its limit — tracked in #1792. Until that lands, triage is single-sub and a sub-1 cap stalls it.

2b — Cron not firing / token expired / hung / crashing

These modes are fully documented in the sibling playbook — do not duplicate the steps here:

Cron entry missing → hourly-triage-watchdog.md §“Cron entry missing”
Cron present, script never runs (missing exec bit, #1790) → §“missing executable bit”
Access token expired (headless refresh) → §“access token expired”
Refresh token expired (browser re-auth) → §“refresh token expired”
Triage hung (>2h) → §“Triage hung”
Script crashing → §“Script crashing”

2c — Stuck In Progress tasks (the >4h condition)

Run the canonical release RPC — it releases tasks claimed >10 min by agents with no heartbeat and returns the count released:

curl -s -X POST "${SUPA_HB_URL}/rest/v1/rpc/release_stalled_tasks" \
  -H "apikey: ${SUPA_HB_KEY}" -H "Authorization: Bearer ${SUPA_HB_KEY}" \
  -H "Content-Type: application/json" -d '{}'
# Returns an integer: the number of tasks released.

Known limitation (observed 2026-06-28): very old claimed rows (days old) whose claimed_by session no longer exists in agent_sessions are not released by release_stalled_tasks() (the function matches live-but-stale sessions, so orphaned rows whose session row is gone fall through). They are inert — they do not hold the per-issue mutex against new work — but they will keep satisfying the watchdog’s “>4h In Progress” heuristic until purged. If they are the only reason the watchdog fired (triage itself is succeeding), treat as benign noise; a targeted cleanup belongs in a separate maintenance task, not this incident.

Step 3 — Verify recovery

# A new successful run appears after the next 20:00 UTC window
curl -s "${SUPA_HB_URL}/rest/v1/dev_r_scheduled_runs?select=status,started_at,exit_code&job_name=eq.nightly-devops-triage&order=started_at.desc&limit=3" \
  -H "apikey: ${SUPA_HB_KEY}" -H "Authorization: Bearer ${SUPA_HB_KEY}" | python3 -m json.tool
# Confirm 0 releasable stalled tasks remain
curl -s -X POST "${SUPA_HB_URL}/rest/v1/rpc/release_stalled_tasks" \
  -H "apikey: ${SUPA_HB_KEY}" -H "Authorization: Bearer ${SUPA_HB_KEY}" -H "Content-Type: application/json" -d '{}'

Do not close the watchdog issue until a triage run shows status=success. If the cause was a spend limit (§2a), the issue stays open and human-action-labelled until the owner raises the limit (or the monthly reset clears it) and the next window succeeds — it is gated on #1873.

Escalation path

Cause	Owner	Autonomous?	Tracking
Monthly spend / usage limit	Account owner (raise at `claude.ai/settings/usage`)	❌	#1873 (human-action), #1792 (durable fix)
Refresh token expired	Owner (browser re-auth, minutes)	❌	sibling playbook §“refresh token”
Access token expired	`refresh-claude-token.py` (2h cron)	✅	auto
Cron missing / non-executable	Worker (re-add / chmod)	✅	#1790
Stuck In Progress tasks	`release_stalled_tasks()`	✅	`cleanup-stalled-tasks` skill

Prevention

scripts/check-sub-usage.py polls each subscription’s unified rate-limit headers and writes ~/.claude/sub-status.json (usage_pct, reset_at) — the early-warning signal for an approaching spend cap, before triage hard-fails.
The triage wrapper runs an auth pre-check that fails fast (exit 1, ~4s) on a rejected subscription instead of burning a long run — keep that pre-check; it makes the log line in §2a the authoritative diagnosis.
Land #1792 (route to claude-runner-2 when sub-1 is capped) to remove the single-subscription dependency that turns one capped account into a full triage outage.

p24-infra Docs

Explorer

agent-watchdog-alert-response