gh-runner-metrics-approach

GH Actions Runner Metrics — Architecture Playbook

Problem

Self-hosted runners on bms-4 expose only process-level metrics (alive/RAM) via gh-runner-metrics.sh + node_exporter textfile collector. GitHub’s own view of whether a runner is online and busy is only available via the GitHub API — but the repo-level runners registry endpoint needs a token scope GH_TOKEN does not have (see below).

Current state: Layer-1 only (#4369)

Offline detection is served entirely by the node_exporter textfile layer. The GitHub-API busy/status layer was removed from the queue-exporter in #4369 because it returned HTTP 403 on every scrape (scope gap, not rate limit — see the correction below). Since #4361 re-keyed the GHRunnerOffline alert off the Layer-1 gh_runner_up metric, the API layer added no alerting value and only burned an already ~430%-oversubscribed request budget (#4180).

Layer 1 — OS process (node_exporter textfile, bms-4)

• Script: /usr/local/bin/gh-runner-metrics.sh (cron every 1 min, root)
• Output: /var/lib/prometheus/node-exporter/gh-runners.prom
• Metrics: gh_runner_up{runner}, gh_runner_rss_bytes{runner}
• Labels: carries codename="bms-4" from Prometheus scrape relabeling
• Alerting: GHRunnerOffline fires off gh_runner_up (self-monitoring.yml, re-keyed #4361)

Layer 2 — GitHub API (REMOVED from the exporter, #4369)

• Was: gh_runner_busy{runner, repo}, gh_runner_status{runner} in monitoring/exporters/queue-exporter, fed by GET /repos/radieu/{repo}/actions/runners.
• Removed because the call 403’d deterministically (scope gap).
• Accepted loss: the API-sourced busy signal has no Layer-1 equivalent. gh_runner_status was redundant with gh_runner_up after #4361, so only busy is a real loss.

Scope correction — actions:read is NOT sufficient (#4369)

A prior version of this playbook claimed GH_TOKEN “has actions:read scope … so no new token is needed” and concluded the runners layer would work. That premise is wrong and is what produced the blind spot:

• actions:read grants the workflow-run endpoints (runs, jobs, artifacts).
• The repo-level runners registry — GET /repos/{repo}/actions/runners — requires administration:read on a fine-grained PAT (or manage_runners:* on an org token).

GH_TOKEN has the former, not the latter, so the runners path returned 403 on every call while the issues path on the same token in the same process stayed healthy — confirming scope, not rate-limit budget (distinct from #4180). The same endpoint is also called by Grafana’s Infinity datasource in grafana/provisioning/dashboards/nightly-jobs.json with the same GH_TOKEN, so those panels 403 too; #4369 (option 2) does not fix them — that is tracked separately.

Restoring the busy signal later (option 1 — needs secret-manager)

If the API-sourced busy signal (and the two nightly-jobs.json panels) are wanted back, a credential change is required — this is secret-manager territory, not a dev-issue:

1. Grant administration:read to GH_TOKEN in secrets/monitoring.env.sops, or
2. Migrate to the GitHub App already in secrets/n8n-bms4.env.sops (GITHUB_APP_ID + GITHUB_APP_PRIVATE_KEY_B64 + GITHUB_APP_INSTALLATION_ID): add those credentials to monitoring.env.sops and implement the JWT + installation-token flow in the exporter. Preferred for new monitoring integrations if GH_TOKEN rotation is burdensome — which it currently is (behind the P0 SOPS rebuild #2835).

Either path then re-adds the _GH_RUNNER_REPOS loop and the gh_runner_busy / gh_runner_status gauges to monitoring/exporters/queue-exporter/app.py, and restores worker-queue panel-92 targets C/D plus the nightly-jobs.json runner panels.

Runner registry (bms-4)

Runner name (GitHub)	Registered repo	OS path
bms4-p24	radieu/p24-infra	/opt/actions-runner-bms4-p24
bms4-etop	radieu/et-operational-platform	/opt/actions-runner-bms4-etop
bms4-android	radieu/whatsup-android-chat-puller	/home/claude-runner/gh-runner

Note: the OS path basename (gh-runner) differs from the GitHub-registered name (bms4-android). gh_runner_up uses the path-derived name.

Grafana — worker-queue dashboard, panel 92

Panel 92 (“GH Actions Runners — BMS-4”) is a stat panel. After #4369 only the Layer-1 targets carry data:

Target	Metric	Meaning	State
A	gh_runner_up{codename="bms-4"}	Process alive (node_exporter)	live
B	gh_runner_rss_bytes{codename="bms-4"} / 1048576	RAM MB (node_exporter)	live
C	gh_runner_busy	Busy/idle (GitHub API)	no data (layer removed #4369)
D	gh_runner_status	Online/offline (GitHub API)	no data (layer removed #4369)

Targets C/D no longer receive data. Their removal from the panel JSON is deliberately not bundled into #4369 — worker-queue.json has an unrelated malformed-JSON blocker (},, at line 3167, filed as #4371) and the panel-cleanup rides with that fix.

Deployment (Layer 1 only)

# on vps-i1
cd /opt/p24-infra/monitoring
docker compose up -d --build queue-exporter
# verify: only the node_exporter-sourced Layer-1 series remain; no gh_runner_busy/status
curl -s http://localhost:9200/metrics | grep gh_runner   # → no output from the exporter
docker logs monitoring-queue-exporter-1 2>&1 | grep -i "runners" # → no 403 lines

Troubleshooting

Symptom	Check
gh_runner_up missing	gh-runner-metrics.sh cron on bms-4 + node_exporter textfile dir
Exporter still 403ing on /actions/runners	stale image — rebuild; #4369 removed the call
Want busy/status back	credential change (option 1 above) — route to secret-manager