Server Health Registry + Priority-aware Dispatch

Issue: TBD (set after issue creation) Status: Plan — awaiting review before implementation Author: orchestrating session, 2026-06-30 Labels: feat, plans-review, worker-queue

Planning document. Extends dev_r_server_capacity with heartbeat columns and updates the dispatcher to shed low-priority jobs when capacity drops. No code ships until plan review passes.

1. Problem statement

The worker queue currently runs on bms-4 only as the active dispatch node. If bms-4 becomes unavailable (OS crash, OOM, Docker failure, network partition) all autonomous work stops immediately with no automatic recovery path.

We are adding vps-h1 as a second dispatch node and lap1 (Acer laptop) as a local third node. With multiple nodes the dispatcher must:

Know which nodes are alive before routing a job.
Shed lower-priority work when fewer nodes are available, so remaining capacity is reserved for the most critical jobs.
Never silently queue a job on a dead node (current behaviour).

What already exists (reuse, don’t rebuild)

Asset	Relevant to this design
`public.dev_r_server_capacity` (Supabase)	Per-server config: `server_label`, `max_workers_prime/night`, `max_weight_prime/night`, `enabled`, `worker_ram_gb`. Dispatcher reads this every cycle.
`scripts/queue-dispatcher-loop.py`	Selects enabled servers, fills free worker slots. `weight_filter()` already maps `max_weight` → job type gates. `_auto_heal_weight()` already un-degrades weight floors.
Weight enum	`light` < `heavy` < `playwright` — already in dispatcher + DB.
`monitoring/prometheus/rules/ai-workers.yml`	Alerting rules fire on worker slot exhaustion, stale dispatch.
`dev_r_worker_queue` statuses	`queued/claimed/running/done/failed/oom_killed/cancelled` — re-dispatch + OOM handling exist.

2. Design

2.1 Schema change — add heartbeat columns to `dev_r_server_capacity`

-- migration: 049_server_heartbeat.sql
ALTER TABLE public.dev_r_server_capacity
  ADD COLUMN IF NOT EXISTS last_heartbeat  TIMESTAMPTZ,
  ADD COLUMN IF NOT EXISTS server_status   TEXT NOT NULL DEFAULT 'unknown'
    CHECK (server_status IN ('healthy', 'degraded', 'down', 'unknown'));
 
COMMENT ON COLUMN public.dev_r_server_capacity.last_heartbeat IS
  'Timestamp of most recent heartbeat write from the server itself. NULL = never seen.';
COMMENT ON COLUMN public.dev_r_server_capacity.server_status IS
  'Computed health: healthy (hb < 5 min), degraded (hb 5–15 min), down (hb > 15 min or NULL).';

A Supabase function refreshes server_status from last_heartbeat on every upsert:

CREATE OR REPLACE FUNCTION public.dev_r_server_capacity_refresh_status()
RETURNS TRIGGER LANGUAGE plpgsql AS $$
BEGIN
  NEW.server_status := CASE
    WHEN NEW.last_heartbeat IS NULL                             THEN 'unknown'
    WHEN NEW.last_heartbeat > NOW() - INTERVAL '5 minutes'     THEN 'healthy'
    WHEN NEW.last_heartbeat > NOW() - INTERVAL '15 minutes'    THEN 'degraded'
    ELSE                                                             'down'
  END;
  RETURN NEW;
END;
$$;

A scheduled Supabase pg_cron job re-evaluates status every minute even when no heartbeat arrives (so a crashed server transitions healthy → degraded → down without needing a write from the dead box):

SELECT cron.schedule('server-status-refresh', '* * * * *',
  $$UPDATE public.dev_r_server_capacity
    SET server_status = CASE
      WHEN last_heartbeat IS NULL                          THEN 'unknown'
      WHEN last_heartbeat > NOW() - INTERVAL '5 minutes'  THEN 'healthy'
      WHEN last_heartbeat > NOW() - INTERVAL '15 minutes' THEN 'degraded'
      ELSE                                                      'down'
    END
    WHERE enabled = TRUE$$);

2.2 Heartbeat script — runs on each server

File: scripts/server-heartbeat.sh (deployed to /opt/p24-infra/scripts/ on each node)

#!/usr/bin/env bash
# Writes heartbeat + busy slot count to dev_r_server_capacity.
# Runs every 2 minutes via systemd timer or cron.
# PLAYBOOK: server-heartbeat.md
set -euo pipefail
 
SERVER_LABEL="${SERVER_LABEL:?SERVER_LABEL not set}"   # e.g. 'bms-4'
SUPABASE_URL="${SUPABASE_URL:?}"
SUPABASE_SERVICE_KEY="${SUPABASE_SERVICE_KEY:?}"
 
BUSY=$(find /tmp/p24-worker-*.pid -maxdepth 0 2>/dev/null | wc -l || echo 0)
 
curl -fsS -X PATCH \
  "${SUPABASE_URL}/rest/v1/dev_r_server_capacity?server_label=eq.${SERVER_LABEL}" \
  -H "apikey: ${SUPABASE_SERVICE_KEY}" \
  -H "Authorization: Bearer ${SUPABASE_SERVICE_KEY}" \
  -H "Content-Type: application/json" \
  -d "{\"last_heartbeat\": \"$(date -u +%Y-%m-%dT%H:%M:%SZ)\", \"current_workers\": ${BUSY}}"

Cron entry (every 2 min):

*/2 * * * * /opt/p24-infra/scripts/server-heartbeat.sh >> /var/log/p24-heartbeat.log 2>&1

lap1 (Windows): PowerShell equivalent using Invoke-RestMethod; pid count via Get-Process -Name claude | Measure-Object.

2.3 Dispatcher change — health gate before routing

scripts/queue-dispatcher-loop.py — two targeted changes:

Change A: filter dead servers before selecting target

Current code selects enabled = TRUE. Add:

# Only dispatch to servers with recent heartbeat.
# server_status is refreshed by pg_cron every minute.
HEALTHY_STATUSES = {"healthy", "degraded"}   # 'degraded' = hb 5-15 min, still try
 
servers = supabase.table("dev_r_server_capacity") \
    .select("*") \
    .eq("enabled", True) \
    .in_("server_status", list(HEALTHY_STATUSES)) \
    .execute().data

Change B: capacity ratio → weight floor

After fetching healthy servers, compute fleet capacity ratio and tighten the weight floor:

total_slots  = sum(s["max_workers_prime"] for s in all_enabled_servers)
healthy_slots = sum(s["max_workers_prime"] for s in servers)  # healthy only
capacity_ratio = healthy_slots / total_slots if total_slots else 0
 
# Weight floor: shed low-priority jobs when capacity is reduced
if capacity_ratio >= 0.75:
    effective_weight_floor = "light"      # normal — all jobs accepted
elif capacity_ratio >= 0.5:
    effective_weight_floor = "heavy"      # shed playwright
elif capacity_ratio >= 0.25:
    effective_weight_floor = "heavy"      # same — shed playwright (no "ultra-heavy" tier yet)
else:
    effective_weight_floor = "light"      # <25% capacity: only light (sys-admin, incident)
    # Note: "light" jobs are highest priority in the current weight enum

The existing weight_filter() function already gates by max_weight. The capacity floor overrides per-server max_weight downward when fleet health is degraded.

Change C: Discord + GH issue alert on zero healthy servers

if not servers:
    _alert_no_healthy_servers(all_enabled_servers)   # Discord embed + GH issue
    time.sleep(60)
    continue

2.4 Priority mapping (clarification of existing weight enum)

Weight	Meaning	Shed when capacity <
`light`	sys-admin, credential rotation, incident response, short p24-infra tasks	Never
`heavy`	dev-issue implementation, feature PRs	25% of total slots
`playwright`	E2E test runs, visual regression, heavy batch	75% of total slots

2.5 vps-h1 registration

New row in dev_r_server_capacity:

INSERT INTO public.dev_r_server_capacity
  (server_label, max_workers_prime, max_workers_night, max_weight_prime, max_weight_night,
   os_ram_gb, os_vcpus, reserved_ram_gb, worker_ram_gb, enabled, notes)
VALUES
  ('vps-h1', 2, 2, 'heavy', 'heavy',
   8, 2, 2, 3, TRUE,
   'Hostinger VPS — 2nd dispatch node, WAHA decommissioned 2026-06');

max_weight='heavy': vps-h1 has only 2 cores, so playwright E2E (CPU-heavy) is excluded even at full fleet capacity.

2.6 lap1 registration

INSERT INTO public.dev_r_server_capacity
  (server_label, max_workers_prime, max_workers_night, max_weight_prime, max_weight_night,
   os_ram_gb, os_vcpus, reserved_ram_gb, worker_ram_gb, enabled, notes)
VALUES
  ('lap1', 2, 0, 'heavy', 'light',
   8, 4, 3, 2, FALSE,  -- starts disabled; enable manually when laptop is docked
   'Acer laptop — local worker, variable uptime. Enable only when plugged in.');

max_workers_night=0 + enabled=FALSE default: lap1 should not receive night dispatch automatically. Operator enables it manually for daytime work sessions.

3. Files changed

File	Change
`monitoring/supabase/migrations/049_server_heartbeat.sql`	New migration: columns + trigger + pg_cron job
`scripts/server-heartbeat.sh`	New script (bash)
`scripts/server-heartbeat.ps1`	New script (PowerShell, lap1)
`scripts/queue-dispatcher-loop.py`	Three targeted patches (health gate, capacity ratio, alert)
`ansible/roles/claude-runner/tasks/main.yml`	Add heartbeat cron + systemd timer deployment
`docs/playbooks/server-heartbeat.md`	New playbook: setup, troubleshooting, manual override
`docs/environments/vps-h1.md`	Update: new role, dev_r_server_capacity row
`monitoring/grafana/provisioning/dashboards/worker-queue.json`	Add `server_status` panel, capacity ratio gauge

4. Out of scope

CF Worker external health check for vps-i1 monitoring stack → covered in monitoring-standby-vps-h1.md
bms-3 as additional dispatch node → separate issue after this lands
lap1 GitHub runner (lap1-gh-w1) → requires human setup of GH runner token; tracked in docs/environments/lap1.md TODO section

5. Risks

Risk	Mitigation
pg_cron job marks a slow-but-alive server as `down`	15-min window before `down` status; dispatcher retries next cycle
heartbeat script fails silently	Cron stderr → `/var/log/p24-heartbeat.log`; Mezmo ingests this file
capacity ratio calculation counts disabled servers	`total_slots` uses `all_enabled_servers` (enabled=TRUE), not all rows
lap1 heartbeat runs when lid closed / screen locked	Windows: Task Scheduler with “Run whether user is logged on or not”; add `enabled=FALSE` default
Dispatcher sees `degraded` server and fills its slots anyway	Intentional: `degraded` (5–15 min stale) still serves jobs; operator should investigate via Grafana

6. Test plan

Migration 049 applies cleanly on Supabase (supabase db push)
Heartbeat script runs manually on bms-4, row updates in Supabase
server_status transitions: stop heartbeat → wait 6 min → assert degraded; wait 16 min → assert down
Dispatcher skips down server: set bms-4 server_status='down' manually, confirm jobs route to vps-h1 only
Playwright job blocked at 50% capacity: set vps-h1 enabled=FALSE, submit playwright job, confirm it stays queued
Alert fires: set all servers server_status='down', confirm Discord message + GH issue created

p24-infra Docs

Explorer

server-health-priority-dispatch

Server Health Registry + Priority-aware Dispatch

1. Problem statement

What already exists (reuse, don’t rebuild)

2. Design

2.1 Schema change — add heartbeat columns to `dev_r_server_capacity`

2.2 Heartbeat script — runs on each server

2.3 Dispatcher change — health gate before routing

2.4 Priority mapping (clarification of existing weight enum)

2.5 vps-h1 registration

2.6 lap1 registration

3. Files changed

4. Out of scope

5. Risks

6. Test plan

Graph View

Table of Contents

p24-infra Docs

Explorer

server-health-priority-dispatch

Server Health Registry + Priority-aware Dispatch

1. Problem statement

What already exists (reuse, don’t rebuild)

2. Design

2.1 Schema change — add heartbeat columns to dev_r_server_capacity

2.2 Heartbeat script — runs on each server

2.3 Dispatcher change — health gate before routing

2.4 Priority mapping (clarification of existing weight enum)

2.5 vps-h1 registration

2.6 lap1 registration

3. Files changed

4. Out of scope

5. Risks

6. Test plan

Graph View

Table of Contents

2.1 Schema change — add heartbeat columns to `dev_r_server_capacity`