p24-infra Docs

server-heartbeat

15 min read

Server Heartbeat — Setup & Troubleshooting

What it is

A bash script (scripts/server-heartbeat.sh) that runs every 2 minutes on each dispatch server and writes last_heartbeat + current_workers to dev_r_server_capacity in Supabase. The dispatcher uses this to skip dead servers before routing jobs.

Setup on a new server (Linux)

Ansible-managed since #4384. On any host that runs the claude-runner role (vps-i1, vps-h1, and every host provisioned by playbooks/provision-new-vps.yml) all four steps below are performed by the role — script, log file, wrapper and cron. Do them by hand only on a host the role does not cover.

ansible-playbook -i inventory/hosts.yml playbooks/vps-i1.yml --tags claude-runner --check --diff

The role renders /opt/p24-infra/scripts/run-heartbeat.sh from a template and crons only that path, so no credential ever enters the crontab. Per-host secret plumbing is set with heartbeat_env_file / heartbeat_key_var / heartbeat_cron_user in host_vars/<host>.yml — see the env-path table below and roles/claude-runner/README.md.

Before installing its cron the role removes unmanaged heartbeat lines (any heartbeat line not preceded by #Ansible: p24-server-heartbeat) from the target crontab. Hosts set up by hand before #4384 carry exactly such a line, so the first live --apply converges them to a single entry instead of doubling the heartbeat. Verify after applying:

crontab -l -u claude-runner | grep -c heartbeat   # expect: 1

  1. Copy script to server:

    scp scripts/server-heartbeat.sh root@<server>:/opt/p24-infra/scripts/
chmod 750 /opt/p24-infra/scripts/server-heartbeat.sh

  2. Pre-create the log file as root (claude-runner cannot create files in /var/log/):

    touch /var/log/p24-heartbeat.log
chown claude-runner:claude-runner /var/log/p24-heartbeat.log
chmod 640 /var/log/p24-heartbeat.log

  3. On bms-4, the cron uses a wrapper script that sources /opt/p24-infra/bms-4/.env safely. Add cron entry under claude-runner (crontab -e -u claude-runner):

    */2 * * * * /opt/p24-infra/scripts/run-heartbeat-bms4.sh >> /var/log/p24-heartbeat.log 2>&1

    For new servers, create a similar wrapper sourcing the server’s .env.bak or bms-4/.env.

  4. Verify:

    # Run once manually as claude-runner
su -s /bin/bash claude-runner -c '/opt/p24-infra/scripts/run-heartbeat-bms4.sh'
tail -f /var/log/p24-heartbeat.log

Setup on lap1 (Windows)

  1. Copy scripts/server-heartbeat.ps1 to C:\opt\p24-infra\scripts\
  2. Create Task Scheduler task: every 2 min, run PowerShell with SERVER_LABEL=lap1 env set
  3. Secrets from .env.local on the laptop

Setup on windows-dev (Windows)

Same script/mechanism as lap1, SERVER_LABEL=windows-dev. SUPABASE_URL/SUPABASE_SERVICE_KEY come from secrets/monitoring.env.sops (developer workstation, has the age key) — decrypt with the safe extraction pattern, never print the value. First manual run (2026-09-01, unblocking the HEALTH-GATE skip on initial windows-dev activation — see 6714):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$decrypted = sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops
$env:SUPABASE_URL = ($decrypted | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1]
$env:SUPABASE_SERVICE_KEY = ($decrypted | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1]
$env:SERVER_LABEL = "windows-dev"
& .\scripts\server-heartbeat.ps1

Recurring schedule (live, 2026-09-01): scripts/windows-dev-heartbeat-task.ps1 wraps the decrypt + invoke steps above into one file, registered as Task Scheduler task P24-WindowsDev-Heartbeat (every 2 min, -RunLevel default/non-elevated — elevated registration hit Odmowa dostępu/Access Denied under this session, non-elevated works fine since the task only needs to reach an HTTPS endpoint). Register it with the repo script (#7686) rather than by hand — it is idempotent (-Force create-or-replace) and encodes the hidden-window + bounded-repetition settings:

.\scripts\register-windows-dev-tasks.ps1
# sops.exe already on PATH on windows-dev; if not: .\scripts\register-windows-dev-tasks.ps1 -SopsBinDir "C:\Users\konar\bin"

(The script uses -WindowStyle Hidden so the task no longer flashes a console window, and a large-but-bounded -RepetitionDuration of 10 years — [TimeSpan]::MaxValue fails XML validation.) Verify: Get-ScheduledTaskInfo -TaskName "P24-WindowsDev-Heartbeat" — LastTaskResult should be 0.

Setup on lap2 (Windows + WSL2 worker host)

lap2 is a WSL2-based worker host (ADR-006). It differs from windows-dev in three ways, all handled by the dedicated wrapper scripts/lap2-heartbeat-task.ps1 (which calls the shared scripts/server-heartbeat.ps1):

  1. Secrets via the SOPS broker, not a raw sops -d pipeline. The wrapper uses Invoke-WithSopsSecret (ADR 003, #5298) to decrypt SUPABASE_URL / SUPABASE_SERVICE_KEY from secrets/monitoring.env.sops, so no credential reaches the task args, crontab, or any file. The lap2-windows age key is already a catch-all recipient of monitoring.env.sops (.sops.yaml general rule, 6909) — no secret-manager delivery is needed; the key is decrypted on-demand on the host that already holds the age key. (Confirm with a key-NAME listing only: sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | ForEach-Object { $_.Split("=")[0] }.)
  2. current_workers counts Docker worker containers inside the WSL2 guest, not native Windows claude processes (lap2’s workers are containers — ADR-006). The wrapper reads the count via wsl.exe -d Ubuntu-26.04 --exec docker ps and passes it to server-heartbeat.ps1 as $env:HEARTBEAT_WORKER_COUNT; it degrades to 0 when WSL/Docker is down.
  3. The status-metric push over SSH degrades gracefully. server-heartbeat.ps1 pushes p24_server_heartbeat_http_status to the vps-i1 pushgateway over SSH (root@217.154.82.162, lap2’s ~/.ssh/id_ed25519) only when $env:HEARTBEAT_PUSHGATEWAY_SSH is set (the lap2 wrapper sets it). The push is best-effort and never fails the heartbeat — a sleeping/offline laptop with no route to vps-i1 simply skips it.

First manual run (seed + verify):

# $env:SOPS_AGE_KEY_FILE defaults to C:\Users\Radek\.age\p24-infra-keys.txt (auto-probed) — set it
# only if your key lives elsewhere. Then:
& .\scripts\lap2-heartbeat-task.ps1
# Expect: "[...] heartbeat OK — lap2 busy=<n>"  and dev_r_server_capacity.last_heartbeat fresh.

Register both lap2 tasks (every 2 min heartbeat + WSL keepalive): use the repo script (#7686) rather than hand-registering — it is idempotent (-Force create-or-replace) and encodes the verified principal settings below. Run it from an elevated PowerShell:

.\scripts\register-lap2-tasks.ps1
# Fully silence the 5-min KeepWSL console flash with the VBS wrapper:
.\scripts\register-lap2-tasks.ps1 -UseVbsWrapperForKeepWsl
# Non-elevated session (no -AtLogOn trigger on KeepWSL): add -NoLogonTrigger
# Verify: Get-ScheduledTaskInfo -TaskName "P24-Lap2-Heartbeat"  → LastTaskResult 0
#         Get-ScheduledTaskInfo -TaskName "P24-Lap2-KeepWSL"    → LastTaskResult 0

The script registers P24-Lap2-Heartbeat (the every-2-min heartbeat wrapper scripts/lap2-heartbeat-task.ps1) and P24-Lap2-KeepWSL (scripts/lap2-keep-wsl-up.ps1, which ensures a named p24-wsl-keepalive sleep infinity process inside Ubuntu-26.04 so the VM does not shut down when idle — at logon + every 5 min).

Principal: S4U for the heartbeat, Interactive for KeepWSL — do NOT unify them (#7686). This is the key finding the registration script encodes:

  • P24-Lap2-Heartbeat → LogonType S4U, RunLevel Limited. It runs on the repeating trigger with no interactive logon session present. Verified: runs OK (result 0), no window.
  • P24-Lap2-KeepWSL → LogonType Interactive (mandatory). Under S4U the task returned 0 but the WSL2 VM stayed Stopped — no interactive logon session holds the VM alive, so S4U silently breaks the keepalive. Re-verified: after reverting to Interactive, WSL is Running.

Console-flash (#7686): both tasks use -WindowStyle Hidden. For the heartbeat (S4U) that is fully silent. For KeepWSL (Interactive) -WindowStyle Hidden still flashes a console for a split second every 5 min; -UseVbsWrapperForKeepWsl routes it through scripts/lap2-keep-wsl-up-hidden.vbs (wscript.exe, window style 0) which suppresses the flash entirely.

lap2 scheduled-task notes (#7676): sops.exe must be on PATH for the heartbeat task to find the broker (on lap2 it lives in C:\Users\Radek\bin) — register-lap2-tasks.ps1 prepends that directory (its -SopsBinDir default) to $env:Path inside the task action before invoking the wrapper, when the directory exists. Also, Register-ScheduledTask with an -AtLogOn trigger needs elevation (admin); pass -NoLogonTrigger to register P24-Lap2-KeepWSL with the repeating 5-min trigger only when running non-elevated.

One-time inside the guest — enable Docker + systemd at boot (so a freshly-started VM is worker-ready). Run once as root in the WSL2 guest:

# Inside WSL: wsl -d Ubuntu-26.04 -u root
systemctl enable --now docker
systemctl is-active docker   # → active

(systemd-in-WSL requires [boot] systemd=true in the guest’s /etc/wsl.conf — already set per ADR-006 Phase 1. Verify with wslinfo --version / systemctl is-system-running.)

Sleep/lock caveat (by design, not a bug). A laptop that sleeps/locks or loses AC power suspends the whole VM, so the heartbeat goes stale and server_status flips degraded/down. The dispatcher already tolerates an intermittently-available lap2 (max_workers_prime=1, light only). To maximise availability, disable sleep on AC power in Windows power settings — a manual, human-reviewed change, deliberately not performed by any script here (the machine is the user’s own workstation).

Troubleshooting

server-heartbeat.ps1 (Windows) fails with Forbidden use of secret API key in browser:

Fixed 2026-09-01. Invoke-RestMethod’s default User-Agent (Mozilla/5.0 (Windows NT ...) WindowsPowerShell/...) looks like a browser to Supabase’s newer sb_secret_* key format, which actively rejects anything that resembles browser-originated traffic as an anti-key-leakage measure. server-heartbeat.ps1 now sends a custom User-Agent: p24-infra-heartbeat/1.0 header (same fix class as priorities.py’s MGMT_API_USER_AGENT, #5422). If this resurfaces on a script that predates this fix, add the same custom User-Agent header to its Invoke-RestMethod/Invoke-WebRequest call.

server_status shows degraded or down for a healthy server:

  • Check /var/log/p24-heartbeat.log for PATCH errors (empty log = log file missing or wrong owner — see step 2 of setup)
  • Verify SUPABASE_URL and SUPABASE_SERVICE_KEY are correct and not stale after a key rotation
  • Manual test as claude-runner: su -s /bin/bash claude-runner -c '/opt/p24-infra/scripts/run-heartbeat-bms4.sh'

All heartbeats 401 → down since the exact same second (stale key after a key rotation):

Symptom: one or more servers freeze at an identical last_heartbeat timestamp while bms-4/vps-i1 stay fresh. /var/log/p24-heartbeat.log shows a wall of curl: (22) ... error: 401. This means the Supabase service_role key was rotated but the new value never reached those nodes’ local heartbeat env — every PATCH is rejected 401 so last_heartbeat never advances. (Incident #3486, 2026-07-05: key rotated at 15:36 UTC during the port-22/VPN lockdown; secrets-sync could not SSH to vps-h1 / bms-3 / dev-laptop, so they kept the old key and silently 401’d for ~4 days.)

The heartbeat key does not live in one canonical place across nodes — check the right file per server:

ServerCron/wrapperKey file it readsVar nameKept fresh by
vps-i1run-heartbeat.sh (Ansible-rendered, root cron — #4390, was claude-runner before that)/opt/p24-infra/monitoring/.envSUPABASE_SERVICE_ROLE_KEYsecrets-sync.yml
vps-h1run-heartbeat.sh (Ansible-rendered, root cron)/opt/p24-infra/vps-h1/.envSUPABASE_SERVICE_ROLE_KEYsecrets-sync.yml
bms-3run-heartbeat-bms3.sh/opt/p24-infra/bms-4/.env (local copy on bms-3)SUPABASE_SERVICE_ROLE_KEY⚠️ nothing — hand-placed
bms-4run-heartbeat-bms4.sh/opt/p24-infra/bms-4/.envSUPABASE_SERVICE_ROLE_KEYsecrets-sync.yml

One-file rule (#3575). A node’s heartbeat must read the same .env that secrets-sync.yml already writes for that node — never a dedicated heartbeat.env or any other hand-placed copy. A second copy of the key is a second thing to rotate, and the one nobody remembers is the one that 401s. When provisioning a new node, set heartbeat_env_file / heartbeat_key_var in ansible/inventory/host_vars/<host>.yml to that node’s synced .env; do not create a new file.

bms-3 is the remaining exception and is deliberately still open. It reads a misnamed local copy of /opt/p24-infra/bms-4/.env, there is no sync-bms-3 job in secrets-sync.yml (bms-3 is not a sync target at all), and its SSH is currently locked pending KVM access (#2382). Do not record the one-file rule as fleet-complete while this row stands.

bms-3 and bms-4 are not covered by the claude-runner role (playbooks/bms-4.yml is deliberately minimal-touch), so they keep their hand-maintained per-host wrappers. vps-i1 and vps-h1 are Ansible-managed as of #4384 — the env file / key values in their rows come from heartbeat_env_file / heartbeat_key_var in roles/claude-runner/defaults/main.yml (vps-i1 uses these defaults unchanged); the cron-user column for both now comes from a heartbeat_cron_user: root override in each host’s own inventory/host_vars/<host>.yml — vps-h1 has carried this since 3575, and vps-i1’s was added by #4390 once a live-apply attempt found a hand-installed root cron already doing the real work there (see §vps-i1 cron-user fix below).

Diagnose without ever printing the value — compare SHA256 fingerprints against a known-good node:

# On a healthy node (bms-4) — fingerprint the working key:
K=$(grep -m1 '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"')
printf '%s' "$K" | sha256sum | cut -c1-16; unset K
# On the stale node — fingerprint its heartbeat key file (must match the above):
K=$(grep -m1 '^SUPABASE_SERVICE_KEY=' /opt/p24-infra/scripts/heartbeat.env | cut -d= -f2- | tr -d '"')
printf '%s' "$K" | sha256sum | cut -c1-16; unset K

Fix — push the current key over SSH stdin so the value never appears in a chat/log/arg:

KEY=$(grep -m1 '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"')
printf '%s' "$KEY" | ssh root@<node> '
  NEWKEY=$(cat); f=<KEY_FILE_FOR_NODE>; cp "$f" "$f.bak"; tmp=$(mktemp)
  grep -v "^<VAR_NAME>=" "$f" > "$tmp"; printf "<VAR_NAME>=%s\n" "$NEWKEY" >> "$tmp"
  chmod 600 "$tmp"; mv "$tmp" "$f"; unset NEWKEY'
unset KEY
# Verify: run the node's heartbeat once — expect "heartbeat OK" and HTTP 204:
ssh root@<node> 'set -a; . <KEY_FILE_FOR_NODE>; set +a; /opt/p24-infra/scripts/server-heartbeat.sh'

server_status recomputes to healthy automatically once last_heartbeat is fresh — no manual status write needed.

vps-h1 cutover — retiring heartbeat.env (#3575)

vps-h1’s heartbeat_env_file now points at /opt/p24-infra/vps-h1/.env (written by secrets-sync.yml on every run), replacing the unmaintained /opt/p24-infra/scripts/heartbeat.env. The repo change alone does not converge the host — run these in order, and delete last:

# PLAYBOOK: docs/playbooks/server-heartbeat.md
# 1. Confirm the synced file already carries the key (name + presence only, never the value)
ssh root@72.60.32.61 "grep -c '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/vps-h1/.env"   # expect 1
 
# 2. Re-render the wrapper from the updated host_vars
ansible-playbook -i inventory/hosts.yml playbooks/vps-h1.yml --tags claude-runner --check --diff
ansible-playbook -i inventory/hosts.yml playbooks/vps-h1.yml --tags claude-runner
 
# 3. Observe one green heartbeat before removing anything
ssh root@72.60.32.61 "/opt/p24-infra/scripts/run-heartbeat.sh"        # expect: heartbeat OK ... HTTP 204
ssh root@72.60.32.61 "crontab -l -u root | grep -c heartbeat"          # expect 1
 
# 4. Only now retire the old file
ssh root@72.60.32.61 "mv /opt/p24-infra/scripts/heartbeat.env /opt/p24-infra/scripts/heartbeat.env.retired"

Deleting heartbeat.env before step 3 passes stops vps-h1’s heartbeats outright, which drops it out of dispatch. Keep the .retired copy until a full secrets-sync.yml run has completed after the cutover.

vps-i1 cron-user fix — reconciling the hand-installed root cron (#4390)

Unlike vps-h1’s env-file cutover above, vps-i1 has no env-file migration to do — its heartbeat_env_file / heartbeat_key_var were already correct (they matched the role defaults). The gap #4390 found was narrower: the role’s default heartbeat_cron_user is claude-runner, but vps-i1’s actual working heartbeat has always been a hand-installed cron under root (*/2 * * * * /opt/p24-infra/scripts/run-heartbeat-vps-i1.sh). An infra-task worker that ran the role with the old default would have installed a second, managed heartbeat under claude-runner while leaving root’s hand-cron running untouched — a silent double-heartbeat, not a fix — so it correctly stopped instead of hand-editing root’s crontab.

The fix is a heartbeat_cron_user: root override in inventory/host_vars/vps-i1.yml, mirroring vps-h1’s existing pattern. Once that lands, applying the role targets root’s crontab, where the role’s “reconcile legacy unmanaged cron” task can strip the unmanaged hand-installed line and converge to a single Ansible-managed entry:

# PLAYBOOK: docs/playbooks/server-heartbeat.md
# 1. Preview the change — expect it to touch root's crontab, not claude-runner's
ansible-playbook -i inventory/hosts.yml playbooks/vps-i1.yml --tags claude-runner --check --diff
 
# 2. Apply
ansible-playbook -i inventory/hosts.yml playbooks/vps-i1.yml --tags claude-runner
 
# 3. Verify exactly one heartbeat entry remains, under root
ssh root@217.154.82.162 "crontab -l -u root | grep -c heartbeat"          # expect 1
ssh root@217.154.82.162 "crontab -l -u claude-runner | grep -c heartbeat" # expect 0
ssh root@217.154.82.162 "/opt/p24-infra/scripts/run-heartbeat.sh"          # expect: heartbeat OK ... HTTP 204

No file needs retiring here — the role’s reconcile step removes the unmanaged line as part of the same apply, so there is no ordered “delete last” step like the vps-h1 cutover above.

Manual override (emergency):

-- Force a server back to healthy while investigating
UPDATE dev_r_server_capacity
SET last_heartbeat = NOW(), server_status = 'healthy'
WHERE server_label = 'bms-4';

Disable a server from dispatch without killing workers:

UPDATE dev_r_server_capacity SET enabled = FALSE WHERE server_label = 'vps-h1';

Alert: ServerHeartbeatAuthFailure

Fires when a node’s heartbeat PATCH is rejected (HTTP 401/403) for 10 minutes — the node is alive and cronning, but its Supabase service_role key is stale. This is the specific signal that was missing during #3486, when a rejected heartbeat was indistinguishable from a dead host.

Source metric: p24_server_heartbeat_http_status{server_label}, pushed by scripts/server-heartbeat.sh to the vps-i1 pushgateway (job=server-heartbeat) on every run — directly on localhost from vps-i1, over SSH from every other node. The push is best-effort and never fails the heartbeat itself, so an absent series means “pushgateway unreachable”, not “heartbeat fine”.

Fix: the §“All heartbeats 401” fingerprint-and-push procedure above. ServerHeartbeatStale will usually be firing alongside it — this alert tells you which of the two runbooks to open.

Alert: ServerHeartbeatStale

Fires when last_heartbeat is > 10 min old. Check:

  1. Is the cron running? crontab -l -u claude-runner
  2. Is the server reachable? ping <server-ip>
  3. Is Supabase up? curl https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/

False positive: frozen exporter gauge for a disabled server (incident #4182, 2026-07-16).

Symptom: ServerHeartbeatStale{server_label="X"} fires with the age climbing without bound, but on-server the heartbeat cron is healthy — /var/log/p24-heartbeat.log shows heartbeat OK (HTTP 204) every 2 min and dev_r_server_capacity.last_heartbeat for X is fresh. The alert instance is queue-exporter:9200, so the staleness is on the metric side, not the server.

Root cause: the queue-exporter (monitoring/exporters/queue-exporter/app.py) only refreshes p24_server_last_heartbeat_timestamp for rows returned by dev_r_server_capacity?...&enabled=eq.true. When a server is set enabled=false (e.g. vps-h1, disabled until provisioned for prod jobs), it drops out of that query and its gauge child keeps re-exposing the value it was last set to — frozen at the instant it was disabled — so time() - it grows forever and the rule (time() - p24_server_last_heartbeat_timestamp{server_label!~"dev-laptop|lap1"}) > 600 pages indefinitely.

Diagnose:

# Is the server actually heartbeating? (fresh timestamp = real heartbeat is fine)
curl -s "$SUPABASE_URL/rest/v1/dev_r_server_capacity?server_label=eq.X&select=server_label,last_heartbeat,enabled,server_status" \
  -H "apikey: $KEY" -H "Authorization: Bearer $KEY"   # look for enabled=false + fresh last_heartbeat
# What is Prometheus seeing?
ssh root@217.154.82.162 'curl -s http://127.0.0.1:9200/metrics | grep ^p24_server_last_heartbeat_timestamp'

Fix: the exporter now prunes gauge children for servers no longer in the enabled set each cycle (#4182), so a disabled server produces no series and no page. If you hit this on an exporter build predating that fix, restart it to clear the frozen child: cd /opt/p24-infra/monitoring && docker compose restart queue-exporter. The alert clears once the vps-h1/X series disappears. Do not flip the server to enabled=true just to silence the page — that makes the dispatcher route jobs to an unprovisioned node.

Graph View

Backlinks