p24-infra Docs

grafana-claude-sub-monitoring

7 min read

Playbook — Grafana: Claude Subscription Monitoring

Status: ACTIVE Dashboard: Worker Queue (uid: worker-queue-v1) — Claude Sub section (panels 104–111)

Architecture

bms-4 cron (every 10 min)
  └── /opt/p24-infra/scripts/check-sub-usage.py
       runs as: claude-runner (radieu@gmail.com) AND claude-runner-2 (ecotrans)
       reads: ~/.claude/.credentials.json → OAuth accessToken
       calls: POST https://api.anthropic.com/v1/messages (model: haiku)
       reads headers:
         anthropic-ratelimit-unified-5h-utilization   (0.0–1.0)
         anthropic-ratelimit-unified-7d-utilization   (0.0–1.0)
         anthropic-ratelimit-unified-5h-reset         (Unix timestamp)
         anthropic-ratelimit-unified-7d-reset         (Unix timestamp)
         anthropic-ratelimit-unified-status           (allowed / rejected)
       pushes 6 metrics via SSH to pushgateway on vps-i1:
         p4_claude_sub_usage_pct        — binding window % (100 = full)
         p4_claude_sub_usage_pct_5h     — 5-hour window %
         p4_claude_sub_usage_pct_7d     — 7-day window %
         p4_claude_sub_blocked          — 1 if rejected, 0 if allowed
         p4_claude_sub_reset_5h_seconds — Unix timestamp of 5h reset
         p4_claude_sub_reset_7d_seconds — Unix timestamp of 7d reset
       label: account="radieu" or account="ecotrans"
  └── SSH → root@217.154.82.162 → pushgateway :9091

vps-i1 Prometheus (scrape_interval: 15s)
  └── scrapes pushgateway → stores metrics
  └── Grafana reads via datasource "prometheus"

Cron schedule on bms-4:

  • */10 * * * * claude-runner /opt/p24-infra/scripts/run-sub-usage-runner1.sh
  • */10 * * * * claude-runner-2 /opt/p24-infra/scripts/run-sub-usage-runner2.sh

Logs:

  • /var/log/p24-infra-workers/sub-usage-runner1.log (owned by claude-runner)
  • /var/log/p24-infra-workers/sub-usage-runner2.log (owned by claude-runner-2)

Panel layout (Worker Queue dashboard)

IDAccountMetricxyw
104radieu5h usage %043
106radieu5h reset countdown343
107radieu7d usage %643
108radieu7d reset countdown943
105ecotrans5h usage %1243
109ecotrans5h reset countdown1543
110ecotrans7d usage %1843
111ecotrans7d reset countdown2143

Reset countdown PromQL (radieu 5h example):

clamp_min(p4_claude_sub_reset_5h_seconds{account="radieu",machine="bms-4"} - time(), 0)

Unit: dtdurations (seconds, displays as human-readable “2h 15m”).

1. Diagnosis — panels show “N/A” or no data

Step 1: Check if pushgateway has metrics

ssh root@217.154.82.162
curl -s http://localhost:9091/metrics | grep p4_claude_sub | head -20

If empty → metrics are not being pushed. Jump to Step 2. If populated → check Prometheus scrape and Grafana datasource.

Step 2: Check cron is running

ssh root@54.36.123.110
# Check if cron entries exist
grep sub-usage /etc/cron.d/p24-workers 2>/dev/null || crontab -l -u claude-runner 2>/dev/null
# Check log files
ls -la /var/log/p24-infra-workers/sub-usage-runner*.log

Most common cause: log file owned by root → bash wrapper exits before running python

Fix:

chown claude-runner:claude-runner /var/log/p24-infra-workers/sub-usage-runner1.log
chown claude-runner-2:claude-runner-2 /var/log/p24-infra-workers/sub-usage-runner2.log

Then run manually to verify:

su -s /bin/bash claude-runner -c "python3 /opt/p24-infra/scripts/check-sub-usage.py"
su -s /bin/bash claude-runner-2 -c "python3 /opt/p24-infra/scripts/check-sub-usage.py"

Expected output:

[check-sub-usage] user=claude-runner usage=9.0% unified_status=allowed claim=seven_day reset_7d=... reset_5h=...

Step 3: Confirm metrics in pushgateway after manual run

# IMPORTANT: pushgateway port 9091 is NOT exposed externally (firewalled).
# Always check from INSIDE vps-i1 via SSH — do NOT curl from bms-4 or your workstation.
ssh root@217.154.82.162
curl -s http://localhost:9091/metrics | grep p4_claude_sub_usage_pct

Expected — both accounts present after a successful push:

p4_claude_sub_usage_pct{account="ecotrans",...} 26
p4_claude_sub_usage_pct{account="radieu",...} 14
p4_claude_sub_usage_pct_5h{account="ecotrans",...} 16
p4_claude_sub_usage_pct_5h{account="radieu",...} 69
...

Step 4: Confirm Prometheus has the metrics

ssh root@217.154.82.162
curl -s 'http://localhost:9090/api/v1/query?query=p4_claude_sub_usage_pct_5h'

If Prometheus returns empty result but pushgateway has data → Prometheus hasn’t scraped yet (wait 15s) or pushgateway scrape is broken.

1a. Diagnosis — one account “N/A” (no data at all), the other shows zeros

These are two DIFFERENT root causes that look similar on the dashboard but require different fixes. Observed together 2026-07-08.

“N/A” (account missing entirely from pushgateway, script logs show valid usage%)

The script computes real data locally but the SSH push (_push_prometheus() in check-sub-usage.py, uses ssh root@217.154.82.162 ... with no -i flag) never reaches vps-i1. Root cause: the worker user’s private key file has a non-default name (e.g. vps_root_key instead of id_ed25519) with no ~/.ssh/config entry pointing to it — the SSH client silently only tries default-named identity files and gives up with Permission denied (publickey). This can persist even after the matching public key IS present in authorized_keys on vps-i1, because the private key is never offered in the first place.

Confirm:

ssh root@54.36.123.110 "su -s /bin/bash <linux_user> -c 'ssh -v -o BatchMode=yes root@217.154.82.162 echo OK' 2>&1 | tail -20"
# Look for: "Trying private key: ..." lines — if your key file isn't listed, that's the bug

Fix — add an SSH config entry for that user (do NOT rename/move the existing private key):

cat > /home/<linux_user>/.ssh/config << 'EOF'
Host 217.154.82.162
  IdentityFile ~/.ssh/<actual_key_filename>
  IdentitiesOnly yes
EOF
chown <linux_user>:<linux_user> /home/<linux_user>/.ssh/config
chmod 600 /home/<linux_user>/.ssh/config

Also verify the public key derived from that private key (ssh-keygen -y -f <keyfile>) is actually present in root@vps-i1:/root/.ssh/authorized_keys — if it was rotated at some point without updating vps-i1, append it there too (back up the file first).

Zero values + blocked=1 (script logs show API HTTP 401)

Check the token-expiry alert first (#3420). Since 2026-07-09 this exact failure mode is alerted proactively: ClaudeRunnerTokenExpired (critical) / ClaudeRunnerTokenExpiringSoon (warning) fire off claude_runner_token_expires_seconds{account,machine} (pushed by check-sub-usage.py). If that alert is already firing for this account, skip straight to re-auth — no manual diagnosis needed. Query in Grafana/Prometheus: claude_runner_token_expires_seconds{account="<radieu|ecotrans>"} — a value <= 0 confirms the token is expired.

The account’s OAuth token has expired — this is the scenario covered by claude-oauth-reauth.md / claude-runner-oauth-rotation.md. Confirm with the expiry check script in that playbook. Fix requires the OAuth browser authorize step — either full Playwright MCP automation (needs Playwright MCP connected in the active Claude Code session — it is NOT available in every session/environment, check via ToolSearch before assuming Option A is possible) or the semi-manual flow: start claude auth login in a background process as the target Linux user, capture the printed OAuth URL, have a human open it and authorize, then inject the returned CODE#STATE back into the waiting process. See claude-runner-oauth-rotation.md for the auth_v4.py script pattern (works per-account by giving each account its own CODE_FILE/log path, e.g. /tmp/auth_code_v3_<account>).

1b. Diagnosis — panel layout overlaps (panels render on top of each other)

If server stats panels (BMS-4, VPS-I1 at y=15) or GH Actions Runners stat (y=23) are hidden under other panels:

# Check panel gridPos in the deployed Grafana dashboard
ssh root@217.154.82.162
python3 /tmp/list-panels.py | sort -t= -k2 -n

Expected clean layout (no y overlaps between panel groups):

  • y=0: queue stats
  • y=4: subscription panels (8 panels w=3 each)
  • y=7: time series (Queue Depth, GH Actions, BMS RAM) h=8
  • y=15: BMS-4 + VPS-I1 server stats h=4
  • y=19: BMS-3 + DEV-LAPTOP server stats h=4
  • y=23: GH Actions Runners stat h=3
  • y=26: Claude Workers RSS (full width w=24, h=8) ← was wrongly at y=15 w=16
  • y=34: Active Workers | RAM per Server
  • y=40: Running | Dispatched/Queued
  • y=50: Failed | Done
  • y=60: Active Sessions (full width)

If P102 (Claude Workers RSS) appears at y=15 with w=16 → it overlaps server stat panels. Fix: edit JSON, set P102 to y=26, w=24, then deploy via gf_push.py (see Section 2).

2. Updating the dashboard JSON

CRITICAL: This dashboard uses allowUiUpdates: true in provisioning config.

This means Grafana’s internal DB version takes precedence over the filesystem file. A simple git pull + reload does NOT update the dashboard. You MUST push via the Grafana API.

Dashboard JSON location

  • Git: monitoring/grafana/provisioning/dashboards/worker-queue.json (on dev branch, PR #2287 + #2311)
  • Server path: /opt/p24-infra/monitoring/grafana/provisioning/dashboards/worker-queue.json

Workflow for JSON changes

  1. Edit JSON locally (use worktree for feature branch, or on current branch)
  2. Validate JSON: 
    $content = [System.IO.File]::ReadAllText("path/to/worker-queue.json")
$content | ConvertFrom-Json  # throws if invalid
  3. SCP to vps-i1: 
    # PLAYBOOK: monitoring-stack-operations.md
scp -i ~/.ssh/id_ed25519 worker-queue.json root@217.154.82.162:/opt/p24-infra/monitoring/grafana/provisioning/dashboards/worker-queue.json
  4. Push via Grafana API (reads credentials from server .env, never prints them): 
    # Write to /tmp/gf_push.py on vps-i1, then:
ssh root@217.154.82.162 'python3 /tmp/gf_push.py'
    See gf_push.py template below.

gf_push.py template

import json, urllib.request, urllib.error, base64
 
def get_env_val(path, key):
    for line in open(path).read().splitlines():
        if line.startswith(key + '='):
            return line.split('=', 1)[1].strip().strip('"').strip("'")
    return ''
 
env_path = '/opt/p24-infra/monitoring/.env'
user = get_env_val(env_path, 'GRAFANA_ADMIN_USER') or 'admin'
pw = get_env_val(env_path, 'GRAFANA_ADMIN_PASSWORD')
 
dash = json.load(open('/opt/p24-infra/monitoring/grafana/provisioning/dashboards/worker-queue.json'))
payload = json.dumps({'dashboard': dash, 'overwrite': True, 'message': 'your change description'}).encode()
creds = base64.b64encode((user + ':' + pw).encode()).decode()
req = urllib.request.Request('http://localhost:3000/api/dashboards/db', data=payload,
    headers={'Content-Type': 'application/json', 'Authorization': 'Basic ' + creds}, method='POST')
try:
    with urllib.request.urlopen(req) as r:
        d = json.loads(r.read())
        print('OK version=' + str(d.get('version')) + ' uid=' + str(d.get('uid')))
except urllib.error.HTTPError as e:
    print('ERR', e.code, e.read().decode()[:300])

Credential key names in .env:

  • GRAFANA_ADMIN_USER (not GF_SECURITY_ADMIN_USER)
  • GRAFANA_ADMIN_PASSWORD (not GF_SECURITY_ADMIN_PASSWORD)

3. Check script version on server vs repo

The deployed script on bms-4 may differ from what’s in the repo. Verify:

ssh root@54.36.123.110
head -60 /opt/p24-infra/scripts/check-sub-usage.py | grep "def _push_prometheus"
# Should show signature with utilization_5h, utilization_7d, reset_ts_5h_raw, reset_ts_7d_raw

If the signature is def _push_prometheus(usage_pct: float, unified_status: str) only → deploy the updated version from the feature branch.

4. Prevention

  • Log file ownership must be maintained after any OS update or crontab change
  • When adding a new log file for a cron job, always chown immediately after creation
  • Never modify the dashboard via Grafana UI and assume the JSON file is the source of truth — with allowUiUpdates: true, the API/UI version wins
  • Always update both the JSON file AND push via API in the same deployment step

Graph View

Backlinks