p24-infra Docs

claude-account-depletion-tracking

4 min read

Claude account depletion tracking (#1912)

The queue dispatcher balances work across multiple Claude subscription accounts per server. Each account maps to a Linux worker user (e.g. claude-runner, claude-runner-2 on bms-4). When a subscription hits its usage limit, spawn-worker.sh exits 5 and the dispatcher moves to the next account.

Before #1912 the account list was a hardcoded SUBSCRIPTION_USERS dict in scripts/queue-dispatcher-loop.py with no persisted status — an operator had no way to see which account was depleted, when, or when it was expected to reset.

Where the state lives

dev_r_server_capacity.claude_accounts — a JSONB array, one entry per account:

[
  {"user": "claude-runner",   "depleted": false, "depleted_at": null, "reset_date": null},
  {"user": "claude-runner-2", "depleted": true,  "depleted_at": "2026-06-28T10:30:00Z", "reset_date": "2026-06-30"}
]
FieldTypeMeaning
userstringLinux worker user the dispatcher SSHes in as (su -s /bin/bash <user>)
depletedbooltrue once spawn-worker.sh exited 5 for this account
depleted_atISO-8601 timestampwhen the dispatcher last marked it depleted (auto-set)
reset_dateISO-8601 datetimewhen the subscription limit is expected to reset — auto-populated from API headers on depletion (also operator-settable for legacy rows); dispatcher auto-unlocks when this passes

A NULL column (e.g. vps-i1, single subscription) makes the dispatcher fall back to ["claude-runner"].

How the dispatcher uses it (#2197 — auto-unlock)

spawn_worker() in scripts/queue-dispatcher-loop.py:

  1. _account_order(server) returns the user list non-depleted first, depleted last (depleted tried last as a fallback in case the limit already reset).
  2. On exit 5 → _read_sub_status(server, user) SSH-reads the worker’s ~/.claude/sub-status.json; _mark_account_depleted() sets depleted=true, depleted_at=now, and reset_date from the API header.
  3. On exit 0 → if the account was flagged depleted, _clear_account_depleted clears depleted/depleted_at/reset_date (a successful spawn proves the subscription reset).
  4. Auto-unlock — _auto_unlock_reset_accounts(server) runs at the start of every dispatch cycle: for each depleted account whose reset_date < now, it SSH-verifies current usage_pct < 95%, then calls _clear_account_depleted(). This breaks the dead-lock where a depleted account is never tried because a non-depleted sibling always succeeds.
  5. If all accounts are depleted, the dispatcher still tries every one and logs a warning (the limit may have reset since it was last flagged).

All DB writes are best-effort — wrapped in try/except so a Supabase error never crashes the dispatch loop.

sub-status.json fields (check-sub-usage.py, #2197)

Written to ~/.claude/sub-status.json every ~10 minutes per worker user:

FieldMeaning
usage_pctBinding utilization as a percentage (0–100)
utilization_7dRaw 7-day rolling window utilization (0.0–1.0)
utilization_5hRaw 5-hour rolling window utilization (0.0–1.0)
unified_statusallowed / rejected from the API
representative_claimWhich window is binding: seven_day or five_hour
reset_atBinding reset time (5h if claim=five_hour, else 7d) — used by spawn-worker.sh
reset_at_7dWeekly 7-day window reset datetime
reset_at_5h5-hour window reset datetime (empty when header absent)
last_updatedWhen this file was written

Operator tasks

See current status on bms-4 (read-only, no secret values):

# From bms-4 or via SSH:
cat /home/claude-runner/.claude/sub-status.json
cat /home/claude-runner-2/.claude/sub-status.json

See DB depletion state:

# On bms-4 dispatcher host:
source /opt/p24-infra/bms-4/.env
curl -sf "${DISPATCHER_SUPA_URL}/rest/v1/dev_r_server_capacity?select=server_label,claude_accounts" \
  -H "apikey: ${SUPABASE_SERVICE_ROLE_KEY}" -H "Authorization: Bearer ${SUPABASE_SERVICE_ROLE_KEY}" | python3 -m json.tool

Manually unlock an account immediately — use when you know the subscription has reset but auto-unlock hasn’t fired yet (e.g. scripts not yet deployed):

# On the dev machine (Windows) — Python with SSL verification disabled:
import json, urllib.request, ssl
ctx = ssl.create_default_context(); ctx.check_hostname = False; ctx.verify_mode = ssl.CERT_NONE
# Read SUPA_URL and KEY from SOPS first (safe extraction pattern)
# ...then:
accounts[i]["depleted"] = False; accounts[i]["depleted_at"] = None; accounts[i]["reset_date"] = None
# PATCH dev_r_server_capacity?server_label=eq.bms-4 with updated accounts

Set reset_date for a legacy depleted row (accounts depleted before #2197 have reset_date=null):

accounts[i]["reset_date"] = "2026-07-07T10:00:00+00:00"  # ISO-8601 UTC datetime
# PATCH as above — dispatcher auto-unlocks when this time passes

Add a new account to a server — append an entry to that server’s array with depleted=false and create the Linux worker user (see docs/playbooks/adding-new-worker.md).

  • #1792 — original subscription balancing (exit 5 handling in spawn-worker.sh)
  • #1873 — spend-limit visibility (closed as follow-up by #1912)
  • #2197 — auto-unlock on reset + dual reset tracking (this feature)
  • Migration: supabase/migrations/20260628102631_claude_accounts_depletion.sql

Graph View