p24-infra Docs

worker-capacity-management

4 min read

Worker Capacity Management — vps-i1 max_workers Guard

What triggers this problem

vps-i1 (IONOS, 8 GB RAM) hosts the full monitoring stack (Prometheus, Thanos, Grafana, Loki, Alertmanager, Caddy, and 11 exporters) which consumes ~4 GB of RAM at steady state. This leaves insufficient RAM for Claude worker jobs (each needs ~3 GB for light weight).

When dev_r_server_capacity.max_workers_prime or max_workers_night is set to a value > 0 for vps-i1, the dispatcher starts sending jobs there. Those jobs spawn worker processes that immediately exhaust available RAM and exit with code 3 (RAM contention / OOM guard).

Known incident: #2979 — 36 review-pr jobs failed between 2026-07-05 and 2026-07-06 after max_workers_prime and max_workers_night were silently reset from 0 to 2.

How to confirm the problem

# Check current vps-i1 capacity settings (Windows dev machine)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:SB_KEY = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
    Select-String "^SUPABASE_SERVICE_ROLE_KEY=").ToString().Split("=",2)[1].Trim()
$headers = @{
    "apikey" = $env:SB_KEY; "Authorization" = "Bearer $env:SB_KEY"
    "User-Agent" = "p24-infra-admin/1.0"
}
Invoke-RestMethod -Uri "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/dev_r_server_capacity?server_label=eq.vps-i1&select=server_label,max_workers_prime,max_workers_night,enabled,updated_at" -Headers $headers | Format-Table -AutoSize
$env:SB_KEY = ""

Expected: max_workers_prime=0, max_workers_night=0.

Symptom that something is wrong: jobs in dev_r_worker_queue failing with failure_reason=spawn_ram or error_message containing exit=3 on server_node=vps-i1.

Step-by-step fix

  1. Restore capacity to 0 immediately (Windows dev):
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:SB_KEY = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
    Select-String "^SUPABASE_SERVICE_ROLE_KEY=").ToString().Split("=",2)[1].Trim()
$headers = @{
    "apikey" = $env:SB_KEY; "Authorization" = "Bearer $env:SB_KEY"
    "Content-Type" = "application/json"
    "User-Agent" = "p24-infra-admin/1.0"
    "Prefer" = "return=minimal"
}
Invoke-RestMethod `
    -Uri "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/dev_r_server_capacity?server_label=eq.vps-i1" `
    -Method Patch `
    -Headers $headers `
    -Body '{"max_workers_prime":0,"max_workers_night":0}'
$env:SB_KEY = ""
Write-Host "vps-i1 max_workers reset to 0"
  1. Requeue any failed jobs on vps-i1 with exit=3:

Run via Supabase SQL editor:

UPDATE dev_r_worker_queue
SET status = 'queued',
    server_node = NULL,
    claimed_at  = NULL,
    started_at  = NULL,
    retry_count = GREATEST(0, retry_count - 1),
    failure_reason = NULL,
    error_message  = NULL
WHERE server_node = 'vps-i1'
  AND status = 'failed'
  AND (failure_reason = 'spawn_ram'
       OR error_message LIKE '%exit=3%');
  1. Verify bms-4 is healthy and picking up the requeued jobs:
# Quick status check
gh run list --repo radieu/p24-infra --workflow dispatch-to-queue.yml --limit 5
  1. File a GitHub issue for root cause investigation if the cause is unknown: Title: [Infra] vps-i1 max_workers regression — <date>

Escalation path if the fix doesn’t work

  • If the PATCH fails (HTTP 4xx): check that SUPABASE_SERVICE_ROLE_KEY in SOPS is current (it may need rotation — see docs/playbooks/static-api-key-incident-rotation.md).
  • If jobs keep routing to vps-i1 after the PATCH: restart the dispatcher on bms-4: ssh root@54.36.123.110 systemctl restart p24-queue-dispatcher.timer
  • If bms-4 is also down: check Grafana → NoHealthyDispatchServers alert.

Root cause of the #2979 incident

The precise trigger that reset max_workers_prime from 0 to 2 for vps-i1 on 2026-07-05 is unknown — the Supabase Management Console has no built-in audit log for DML operations.

Most likely causes (in order of probability):

  1. Migration without proper WHERE clause — a Claude worker session implementing a DB change used an UPDATE without WHERE server_label = 'specific-server', or an INSERT + ON CONFLICT DO UPDATE that clobbered the vps-i1 row. Migration 036_worker_queue_operational_extensions.sql seeds DEFAULT values of 1 (prime) and 2 (night); if the ADD COLUMN ran again with a different column definition, new defaults could overwrite existing values.

  2. Direct REST PATCH in a worker session — a Claude Code worker implementing an issue queried and then PATCHed dev_r_server_capacity without filtering to the correct server label.

  3. Human operator edit — a manual update in the Supabase SQL editor targeting WHERE 1=1 or without a proper server_label filter.

What was NOT the cause:

  • The heartbeat script (server-heartbeat.sh) — it only writes last_heartbeat and current_workers.
  • The dispatcher Python loop — it only patches claude_accounts and max_weight_*.
  • The pg_cron status refresh — it only updates server_status.

Prevention

Three layers are now in place:

  1. Prometheus alert (VpsI1WorkersUnexpectedlyEnabled) — fires within 2 minutes of p24_server_max_workers_prime{server_label="vps-i1"} > 0. Routes to Discord + email via Alertmanager. Implemented in monitoring/prometheus/rules/servers.yml.

  2. Queue-exporter metric — p24_server_max_workers_prime and p24_server_max_workers_night are now scraped every 60 s from dev_r_server_capacity by the queue-exporter. This feeds the alert above and is also visible on any Grafana panel querying that metric.

  3. Idempotent guard migration — supabase/migrations/20260706_vps_i1_max_workers_guard.sql documents the intended zero-worker state and is safe to re-apply after any accidental reset.

If you intentionally need to enable vps-i1 workers (e.g., monitoring stack decommission):

  • Apply the change via a named migration with a clear comment explaining the reason
  • Update this playbook’s “Expected” section
  • Silence the Prometheus alert with a temporary inhibition rule

Graph View