p24-infra Docs

server-heartbeat

10 min read

Server Heartbeat — Setup & Troubleshooting

What it is

A bash script (scripts/server-heartbeat.sh) that runs every 2 minutes on each dispatch server and writes last_heartbeat + current_workers to dev_r_server_capacity in Supabase. The dispatcher uses this to skip dead servers before routing jobs.

Setup on a new server (Linux)

Ansible-managed since #4384. On any host that runs the claude-runner role (vps-i1, vps-h1, and every host provisioned by playbooks/provision-new-vps.yml) all four steps below are performed by the role — script, log file, wrapper and cron. Do them by hand only on a host the role does not cover.

ansible-playbook -i inventory/hosts.yml playbooks/vps-i1.yml --tags claude-runner --check --diff

The role renders /opt/p24-infra/scripts/run-heartbeat.sh from a template and crons only that path, so no credential ever enters the crontab. Per-host secret plumbing is set with heartbeat_env_file / heartbeat_key_var / heartbeat_cron_user in host_vars/<host>.yml — see the env-path table below and roles/claude-runner/README.md.

Before installing its cron the role removes unmanaged heartbeat lines (any heartbeat line not preceded by #Ansible: p24-server-heartbeat) from the target crontab. Hosts set up by hand before #4384 carry exactly such a line, so the first live --apply converges them to a single entry instead of doubling the heartbeat. Verify after applying:

crontab -l -u claude-runner | grep -c heartbeat   # expect: 1

  1. Copy script to server:

    scp scripts/server-heartbeat.sh root@<server>:/opt/p24-infra/scripts/
chmod 750 /opt/p24-infra/scripts/server-heartbeat.sh

  2. Pre-create the log file as root (claude-runner cannot create files in /var/log/):

    touch /var/log/p24-heartbeat.log
chown claude-runner:claude-runner /var/log/p24-heartbeat.log
chmod 640 /var/log/p24-heartbeat.log

  3. On bms-4, the cron uses a wrapper script that sources /opt/p24-infra/bms-4/.env safely. Add cron entry under claude-runner (crontab -e -u claude-runner):

    */2 * * * * /opt/p24-infra/scripts/run-heartbeat-bms4.sh >> /var/log/p24-heartbeat.log 2>&1

    For new servers, create a similar wrapper sourcing the server’s .env.bak or bms-4/.env.

  4. Verify:

    # Run once manually as claude-runner
su -s /bin/bash claude-runner -c '/opt/p24-infra/scripts/run-heartbeat-bms4.sh'
tail -f /var/log/p24-heartbeat.log

Setup on lap1 (Windows)

  1. Copy scripts/server-heartbeat.ps1 to C:\opt\p24-infra\scripts\
  2. Create Task Scheduler task: every 2 min, run PowerShell with SERVER_LABEL=lap1 env set
  3. Secrets from .env.local on the laptop

Troubleshooting

server_status shows degraded or down for a healthy server:

  • Check /var/log/p24-heartbeat.log for PATCH errors (empty log = log file missing or wrong owner — see step 2 of setup)
  • Verify SUPABASE_URL and SUPABASE_SERVICE_KEY are correct and not stale after a key rotation
  • Manual test as claude-runner: su -s /bin/bash claude-runner -c '/opt/p24-infra/scripts/run-heartbeat-bms4.sh'

All heartbeats 401 → down since the exact same second (stale key after a key rotation):

Symptom: one or more servers freeze at an identical last_heartbeat timestamp while bms-4/vps-i1 stay fresh. /var/log/p24-heartbeat.log shows a wall of curl: (22) ... error: 401. This means the Supabase service_role key was rotated but the new value never reached those nodes’ local heartbeat env — every PATCH is rejected 401 so last_heartbeat never advances. (Incident #3486, 2026-07-05: key rotated at 15:36 UTC during the port-22/VPN lockdown; secrets-sync could not SSH to vps-h1 / bms-3 / dev-laptop, so they kept the old key and silently 401’d for ~4 days.)

The heartbeat key does not live in one canonical place across nodes — check the right file per server:

ServerCron/wrapperKey file it readsVar nameKept fresh by
vps-i1run-heartbeat.sh (Ansible-rendered, root cron — #4390, was claude-runner before that)/opt/p24-infra/monitoring/.envSUPABASE_SERVICE_ROLE_KEYsecrets-sync.yml
vps-h1run-heartbeat.sh (Ansible-rendered, root cron)/opt/p24-infra/vps-h1/.envSUPABASE_SERVICE_ROLE_KEYsecrets-sync.yml
bms-3run-heartbeat-bms3.sh/opt/p24-infra/bms-4/.env (local copy on bms-3)SUPABASE_SERVICE_ROLE_KEY⚠️ nothing — hand-placed
bms-4run-heartbeat-bms4.sh/opt/p24-infra/bms-4/.envSUPABASE_SERVICE_ROLE_KEYsecrets-sync.yml

One-file rule (#3575). A node’s heartbeat must read the same .env that secrets-sync.yml already writes for that node — never a dedicated heartbeat.env or any other hand-placed copy. A second copy of the key is a second thing to rotate, and the one nobody remembers is the one that 401s. When provisioning a new node, set heartbeat_env_file / heartbeat_key_var in ansible/inventory/host_vars/<host>.yml to that node’s synced .env; do not create a new file.

bms-3 is the remaining exception and is deliberately still open. It reads a misnamed local copy of /opt/p24-infra/bms-4/.env, there is no sync-bms-3 job in secrets-sync.yml (bms-3 is not a sync target at all), and its SSH is currently locked pending KVM access (#2382). Do not record the one-file rule as fleet-complete while this row stands.

bms-3 and bms-4 are not covered by the claude-runner role (playbooks/bms-4.yml is deliberately minimal-touch), so they keep their hand-maintained per-host wrappers. vps-i1 and vps-h1 are Ansible-managed as of #4384 — the env file / key values in their rows come from heartbeat_env_file / heartbeat_key_var in roles/claude-runner/defaults/main.yml (vps-i1 uses these defaults unchanged); the cron-user column for both now comes from a heartbeat_cron_user: root override in each host’s own inventory/host_vars/<host>.yml — vps-h1 has carried this since 3575, and vps-i1’s was added by #4390 once a live-apply attempt found a hand-installed root cron already doing the real work there (see §vps-i1 cron-user fix below).

Diagnose without ever printing the value — compare SHA256 fingerprints against a known-good node:

# On a healthy node (bms-4) — fingerprint the working key:
K=$(grep -m1 '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"')
printf '%s' "$K" | sha256sum | cut -c1-16; unset K
# On the stale node — fingerprint its heartbeat key file (must match the above):
K=$(grep -m1 '^SUPABASE_SERVICE_KEY=' /opt/p24-infra/scripts/heartbeat.env | cut -d= -f2- | tr -d '"')
printf '%s' "$K" | sha256sum | cut -c1-16; unset K

Fix — push the current key over SSH stdin so the value never appears in a chat/log/arg:

KEY=$(grep -m1 '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"')
printf '%s' "$KEY" | ssh root@<node> '
  NEWKEY=$(cat); f=<KEY_FILE_FOR_NODE>; cp "$f" "$f.bak"; tmp=$(mktemp)
  grep -v "^<VAR_NAME>=" "$f" > "$tmp"; printf "<VAR_NAME>=%s\n" "$NEWKEY" >> "$tmp"
  chmod 600 "$tmp"; mv "$tmp" "$f"; unset NEWKEY'
unset KEY
# Verify: run the node's heartbeat once — expect "heartbeat OK" and HTTP 204:
ssh root@<node> 'set -a; . <KEY_FILE_FOR_NODE>; set +a; /opt/p24-infra/scripts/server-heartbeat.sh'

server_status recomputes to healthy automatically once last_heartbeat is fresh — no manual status write needed.

vps-h1 cutover — retiring heartbeat.env (#3575)

vps-h1’s heartbeat_env_file now points at /opt/p24-infra/vps-h1/.env (written by secrets-sync.yml on every run), replacing the unmaintained /opt/p24-infra/scripts/heartbeat.env. The repo change alone does not converge the host — run these in order, and delete last:

# PLAYBOOK: docs/playbooks/server-heartbeat.md
# 1. Confirm the synced file already carries the key (name + presence only, never the value)
ssh root@72.60.32.61 "grep -c '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/vps-h1/.env"   # expect 1
 
# 2. Re-render the wrapper from the updated host_vars
ansible-playbook -i inventory/hosts.yml playbooks/vps-h1.yml --tags claude-runner --check --diff
ansible-playbook -i inventory/hosts.yml playbooks/vps-h1.yml --tags claude-runner
 
# 3. Observe one green heartbeat before removing anything
ssh root@72.60.32.61 "/opt/p24-infra/scripts/run-heartbeat.sh"        # expect: heartbeat OK ... HTTP 204
ssh root@72.60.32.61 "crontab -l -u root | grep -c heartbeat"          # expect 1
 
# 4. Only now retire the old file
ssh root@72.60.32.61 "mv /opt/p24-infra/scripts/heartbeat.env /opt/p24-infra/scripts/heartbeat.env.retired"

Deleting heartbeat.env before step 3 passes stops vps-h1’s heartbeats outright, which drops it out of dispatch. Keep the .retired copy until a full secrets-sync.yml run has completed after the cutover.

vps-i1 cron-user fix — reconciling the hand-installed root cron (#4390)

Unlike vps-h1’s env-file cutover above, vps-i1 has no env-file migration to do — its heartbeat_env_file / heartbeat_key_var were already correct (they matched the role defaults). The gap #4390 found was narrower: the role’s default heartbeat_cron_user is claude-runner, but vps-i1’s actual working heartbeat has always been a hand-installed cron under root (*/2 * * * * /opt/p24-infra/scripts/run-heartbeat-vps-i1.sh). An infra-task worker that ran the role with the old default would have installed a second, managed heartbeat under claude-runner while leaving root’s hand-cron running untouched — a silent double-heartbeat, not a fix — so it correctly stopped instead of hand-editing root’s crontab.

The fix is a heartbeat_cron_user: root override in inventory/host_vars/vps-i1.yml, mirroring vps-h1’s existing pattern. Once that lands, applying the role targets root’s crontab, where the role’s “reconcile legacy unmanaged cron” task can strip the unmanaged hand-installed line and converge to a single Ansible-managed entry:

# PLAYBOOK: docs/playbooks/server-heartbeat.md
# 1. Preview the change — expect it to touch root's crontab, not claude-runner's
ansible-playbook -i inventory/hosts.yml playbooks/vps-i1.yml --tags claude-runner --check --diff
 
# 2. Apply
ansible-playbook -i inventory/hosts.yml playbooks/vps-i1.yml --tags claude-runner
 
# 3. Verify exactly one heartbeat entry remains, under root
ssh root@217.154.82.162 "crontab -l -u root | grep -c heartbeat"          # expect 1
ssh root@217.154.82.162 "crontab -l -u claude-runner | grep -c heartbeat" # expect 0
ssh root@217.154.82.162 "/opt/p24-infra/scripts/run-heartbeat.sh"          # expect: heartbeat OK ... HTTP 204

No file needs retiring here — the role’s reconcile step removes the unmanaged line as part of the same apply, so there is no ordered “delete last” step like the vps-h1 cutover above.

Manual override (emergency):

-- Force a server back to healthy while investigating
UPDATE dev_r_server_capacity
SET last_heartbeat = NOW(), server_status = 'healthy'
WHERE server_label = 'bms-4';

Disable a server from dispatch without killing workers:

UPDATE dev_r_server_capacity SET enabled = FALSE WHERE server_label = 'vps-h1';

Alert: ServerHeartbeatAuthFailure

Fires when a node’s heartbeat PATCH is rejected (HTTP 401/403) for 10 minutes — the node is alive and cronning, but its Supabase service_role key is stale. This is the specific signal that was missing during #3486, when a rejected heartbeat was indistinguishable from a dead host.

Source metric: p24_server_heartbeat_http_status{server_label}, pushed by scripts/server-heartbeat.sh to the vps-i1 pushgateway (job=server-heartbeat) on every run — directly on localhost from vps-i1, over SSH from every other node. The push is best-effort and never fails the heartbeat itself, so an absent series means “pushgateway unreachable”, not “heartbeat fine”.

Fix: the §“All heartbeats 401” fingerprint-and-push procedure above. ServerHeartbeatStale will usually be firing alongside it — this alert tells you which of the two runbooks to open.

Alert: ServerHeartbeatStale

Fires when last_heartbeat is > 10 min old. Check:

  1. Is the cron running? crontab -l -u claude-runner
  2. Is the server reachable? ping <server-ip>
  3. Is Supabase up? curl https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/

False positive: frozen exporter gauge for a disabled server (incident #4182, 2026-07-16).

Symptom: ServerHeartbeatStale{server_label="X"} fires with the age climbing without bound, but on-server the heartbeat cron is healthy — /var/log/p24-heartbeat.log shows heartbeat OK (HTTP 204) every 2 min and dev_r_server_capacity.last_heartbeat for X is fresh. The alert instance is queue-exporter:9200, so the staleness is on the metric side, not the server.

Root cause: the queue-exporter (monitoring/exporters/queue-exporter/app.py) only refreshes p24_server_last_heartbeat_timestamp for rows returned by dev_r_server_capacity?...&enabled=eq.true. When a server is set enabled=false (e.g. vps-h1, disabled until provisioned for prod jobs), it drops out of that query and its gauge child keeps re-exposing the value it was last set to — frozen at the instant it was disabled — so time() - it grows forever and the rule (time() - p24_server_last_heartbeat_timestamp{server_label!~"dev-laptop|lap1"}) > 600 pages indefinitely.

Diagnose:

# Is the server actually heartbeating? (fresh timestamp = real heartbeat is fine)
curl -s "$SUPABASE_URL/rest/v1/dev_r_server_capacity?server_label=eq.X&select=server_label,last_heartbeat,enabled,server_status" \
  -H "apikey: $KEY" -H "Authorization: Bearer $KEY"   # look for enabled=false + fresh last_heartbeat
# What is Prometheus seeing?
ssh root@217.154.82.162 'curl -s http://127.0.0.1:9200/metrics | grep ^p24_server_last_heartbeat_timestamp'

Fix: the exporter now prunes gauge children for servers no longer in the enabled set each cycle (#4182), so a disabled server produces no series and no page. If you hit this on an exporter build predating that fix, restart it to clear the frozen child: cd /opt/p24-infra/monitoring && docker compose restart queue-exporter. The alert clears once the vps-h1/X series disappears. Do not flip the server to enabled=true just to silence the page — that makes the dispatcher route jobs to an unprovisioned node.

Graph View