gh-runner-assignment-policy

GH Runner Assignment Policy — vps-i1 is Light/Infra-Only

Rule

vps-i1 (ionos label) runs only light, infra-specific jobs. It must never run npm ci, pip install, Playwright, Docker builds, or any job that takes > 5 minutes.

Heavy or generic jobs → ubuntu-latest (GH-hosted) or [self-hosted, bms4, heavy].

Why

vps-i1 (6 vCPU, 8 GB RAM) simultaneously runs the entire monitoring stack: Prometheus, Grafana, Alertmanager, Thanos, 25+ exporters, Loki, Caddy, pdf-service, traccar.

A single npm ci + Playwright run consumes ~2 GB RAM and spikes CPU, triggering OOM kills on monitoring containers. This took the whole observability stack offline on 2026-06-24 when the rc-smoke-tests ran on the vps-i1 runner.

Runner label map

Label	Host	RAM	Allowed workloads
[self-hosted, ionos] or [self-hosted, vps-i1, light]	vps-i1	8 GB shared with monitoring	Infra ops: Prometheus reload, Grafana backup, SOPS decrypt, git pull, curl, short scripts (< 2 min)
[self-hosted, bms4, heavy]	bms-4	32 GB dedicated	Playwright E2E, npm ci, Docker builds, long-running tests
ubuntu-latest	GH-hosted	7 GB ephemeral	Unit tests, linting, pure CI

Workflow authoring rules

1. Never use bare runs-on: self-hosted — without labels this matches ANY self-hosted runner including vps-i1. Always add discriminating labels.

2. Never use runs-on: [self-hosted, Linux] in et-oper or other app repos. Linux is a generic label present on vps-i1. Use [self-hosted, bms4] (or ubuntu-latest) for app CI.

3. For Playwright / browser tests → always [self-hosted, bms4, heavy]

4. For npm ci + app tests → [self-hosted, bms4] (GH-hosted minutes are exhausted; do not rely on ubuntu-latest for et-op CI)

5. vps-i1 ionos label is reserved for p24-infra infra workflows only (alertmanager escalation, Prometheus triage, Grafana backup, WAHA ops, secret sync, etc.)

Registered runners on vps-i1

After the 2026-06-24 incident the following runners were removed from vps-i1:

Runner service	Repo	Action
actions.runner.radieu-et-operational-platform.ionos	et-operational-platform	REMOVED 2026-06-24. et-op CI is pinned to [self-hosted, bms4] → runs on bms4-etop (p24-infra#1154 / et-op#1099)
actions.runner.radieu-amazon-kdp-tango.kdp-ionos-runner	amazon-kdp-tango	Assess: stop if kdp workflows can use ubuntu-latest

p24-infra runner (actions.runner.radieu-p24-infra.*, label set includes ionos) stays — it runs infra-only light jobs that need localhost access to Prometheus (port 9090) and Alertmanager (port 9093). Therefore vps-i1 is not expected to reach zero runners — the light p24-infra ionos runner is intentional, and et-op auto-issue-pipeline.yml / nightly-orchestrator.yml still target [self-hosted, ionos].

secrets-deploy shared label — bms-4/vps-i1 failover for secrets-sync (2026-07-08, #3216)

secrets-sync.yml’s sync-pinbox24-w3 / sync-pinbox24-w4 / sync-pinbox24-backends jobs were pinned to [self-hosted, bms4] only. When bms4-p24infra went offline, these jobs silently queued forever instead of failing over — the resulting manual SSH workaround on bms-1 drifted from git and caused an incident (docs/pinbox24/incident-2026-07-08-w4-signed-url-broken.md).

These three jobs are SOPS-decrypt + SCP/SSH only — no local npm ci/pip install/Playwright/Docker build on the runner itself (the actual docker-compose work happens remotely over SSH on bms-1). This matches the “secret sync” workload already listed as approved for vps-i1 above. Added a custom label secrets-deploy to both bms4-p24infra (id 26) and ionos-2 (id 28) via POST /repos/{owner}/{repo}/actions/runners/{runner_id}/labels (no runner restart needed), and changed those three jobs’ runs-on to [self-hosted, secrets-deploy]. GitHub Actions now dispatches to whichever of the two runners is online, instead of queuing indefinitely when bms-4 is down.

Do not add secrets-deploy to any job that runs heavy local compute — this label intentionally spans vps-i1, so the same OOM risk from the 2026-06-24 incident applies if it’s misused. Only add it to jobs that are pure SOPS-decrypt/SCP/SSH-orchestration, same as the three above.

To re-check current labels: gh api repos/radieu/p24-infra/actions/runners --jq '.runners[] | {id,name,labels:[.labels[].name]}'.

Extended to all remaining light jobs (2026-07-08, same day)

A full audit of secrets-sync.yml found zero heavy local-compute commands (npm ci, pip install, playwright install, docker build, or any other build tooling) anywhere in the file — every job is SOPS-decrypt + SCP/SSH or a plain REST API call (Vercel, GitHub Secrets), with the actual heavy lifting (docker-compose, etc.) always happening remotely over SSH, not on the runner itself. After per-job confirmation, secrets-deploy was also added to:

• sync-vps-h1 — SSH to vps-h1 (git pull, gh CLI). No self-targeting risk.
• sync-bms-4 — SSH to bms-4 itself. bms4-p24infra already self-targets this host today as normal behavior (unchanged); adding the fallback only lets ionos-2 reach it cross-host if bms4-p24infra is down — no new risk introduced.
• sync-brandpilot, sync-et-operational-platform, sync-et-lager, sync-art-agency — Vercel REST API only, no SSH at all. The lightest class of job in this file.
• sync-bms-1 — SSH to bms-1, but scoped to a single key (MEZMO_INGESTION_KEY) — narrower than the already-fixed sync-pinbox24-w3/w4/backends jobs which target the same host.
• sync-gh-secrets — no SSH to any server at all, just gh secret set against the GitHub API directly. No deploy target exists to self-target against.

Only sync-vps-i1 needed the self-targeting guard (see above) — it’s the sole job where the deploy target (vps-i1) is simultaneously the physical host of one of the two pool runners AND hosts a service with prior OOM fragility (the monitoring stack). None of the jobs in this section share that combination.

After this change, every job in secrets-sync.yml carries the secrets-deploy fallback — a bms-4 outage no longer stalls any secrets deployment.

4th job added — sync-vps-i1 needed a self-targeting guard, unlike the other three

sync-vps-i1 (decrypts secrets/monitoring.env.sops + role credential files, SCPs them to vps-i1, then SSHes in to docker compose up -d --force-recreate the entire monitoring stack) was also pinned to [self-hosted, bms4] only and suffered the exact same single-point-of-failure as the three pinbox24 jobs above. Repointed it at runs-on: [self-hosted, secrets-deploy] too.

Why this job needed extra handling that the three pinbox24 jobs did not: sync-pinbox24-w3 / -w4 / -backends deploy to bms-1 — a completely different, unrelated host from either runner in the secrets-deploy pool, so there is no self-targeting risk no matter which of the two runners picks the job up. sync-vps-i1 deploys to vps-i1, and ionos-2 is a runner physically hosted on vps-i1. If bms4-p24infra is offline and sync-vps-i1 lands on ionos-2, the job would SSH from vps-i1 to itself (loopback) to recreate its own monitoring stack — decrypting secrets and running docker compose up -d --force-recreate against the same containers the runner process depends on. That’s a qualitatively different risk than “job waits for bms-4,” so it gets an explicit refusal instead of silent execution:

• Every real deploy step (Install sops, all Decrypt * steps, Ship to vps-i1) carries if: runner.name != 'ionos-2', so none of them execute when the job is dispatched to the vps-i1 runner itself.
• A dedicated Guard — refuse self-targeting deploy on ionos-2 step (if: runner.name == 'ionos-2') runs instead, emitting an ::warning:: annotation plus a plain-text explanation of why the deploy was refused and what to do next (wait for bms-4, or deploy manually per this playbook).
• The Log secrets deploy to vps-i1 step (still if: always()) records result: "skipped_self_target" with a distinct detail message when this happens, so the Supabase infra_operations audit trail doesn’t misreport a no-op run as a successful deploy.
• checkout, the logging step, and Scrub plaintext secrets are left unguarded — they’re no-ops or safe cleanup regardless of which runner picked up the job.

Net effect: sync-vps-i1 still fails over to ionos-2 when bms-4 is down for job-scheduling purposes (it no longer queues forever), but the actual monitoring-stack redeploy only ever runs when dispatched to bms4-p24infra. A genuine vps-i1 monitoring-secrets update during a bms-4 outage still requires either waiting for bms-4 to come back, or an operator running the Ship to vps-i1 steps manually per this playbook.

Capacity-tier shared labels — light / super-light (2026-07-17, #4247)

Problem

et-operational-platform’s light orchestration jobs (Nightly Orchestrator, Auto Issue Pipeline) were pinned to runs-on: [self-hosted, ionos]. The ionos label exists on only vps-i1’s runner, so when that single systemd unit died (2026-07-17, #4221) the jobs stranded with no failover and queued 6h+. The GH Actions side had no capacity-tier failover, while the Claude worker-queue already uses host-agnostic capacity tiers (super-light / light / heavy — dev_r_worker_queue, plan-2427). The two dispatch systems spoke different vocabularies.

Fix — mirror the worker-queue tiers onto GH runner labels

Apply the worker-queue capacity vocabulary as shared runner labels (same mechanism as secrets-deploy above — one label on multiple hosts; GitHub dispatches to whichever is online):

Tier label	Hosts carrying it	Allowed workloads
super-light	vps-i1, bms-4, vps-h1, dev-laptop (when up)	Tiny orchestration only — issue reads, gh calls, short scripts (< 2 min)
light	vps-i1, bms-4	Light infra/orchestration jobs (heavier than super-light, still no build tooling)
heavy / bms4 / playwright	bms-4 only	npm ci, pip install, Playwright, Docker builds (unchanged — stays off vps-i1 per this policy)

Ordering is super-light ⊂ light ⊂ heavy by resource cost. vps-h1 (2 cores / 8 GB) and the intermittently-available dev-laptop carry only super-light — never light or heavy — because a heavier job on them would starve or OOM. bms-4 (32 GB) carries light + super-light in addition to its heavy labels. vps-i1 carries light + super-light but the OOM guardrail above still forbids npm ci/Playwright there — never add heavy to vps-i1.

Where the labels are declared (config-as-code)

• ansible/inventory/host_vars/vps-i1.yml → github_runner_extra_labels: [secrets-deploy, light, super-light]

• ansible/inventory/host_vars/bms-4.yml → github_runner_extra_labels: [light, super-light]

• ansible/inventory/host_vars/vps-h1.yml → github_runner_extra_labels: [super-light]

• .github/workflows/reregister-ionos-runners.yml → pending paired edit (needs a workflow-scoped token — the queue-worker OAuth token has only repo scope and GitHub refuses workflow-file pushes without it). Change the et runner’s reregister_one label argument from self-hosted,ionos to self-hosted,ionos,light,super-light so an auto-heal re-registration re-applies the tiers:

  -              "ionos" "self-hosted,ionos" \
  +              "ionos" "self-hosted,ionos,light,super-light" \

Same inert-until-applied caveat as secrets-deploy: setting github_runner_extra_labels in host_vars only re-applies on the next run of the github-runner role for that host’s github_runner_repo_url, and bms-4’s runners are hand-registered (the role does not manage them), so those entries are declarative intent. The labels take effect live via one of the two paths below.

Live-apply (sys-admin / infra-task — NOT a dev-issue step)

Applying a label to an already-registered runner needs no restart. From any host with a GitHub-App/admin token (or an operator gh auth login with Administration:write on the repo):

# 1. Find the runner id (per repo — the label lands on the runner, which is repo-scoped)
gh api repos/radieu/et-operational-platform/actions/runners \
  --jq '.runners[] | {id,name,labels:[.labels[].name]}'
 
# 2. Add the tier label(s) — additive, does not remove existing labels
gh api --method POST repos/radieu/et-operational-platform/actions/runners/<ID>/labels \
  --input - <<<'{"labels":["light","super-light"]}'

Apply per capable host’s runner in the target repo:

• et-op ionos runner (vps-i1) → light, super-light
• bms4-etop (bms-4) → super-light (and light if it should absorb the heavier light tier)
• vps-h1 runner (if/when registered for the repo) → super-light

Once the pending reregister-ionos-runners.yml edit above lands, a re-registration of the et runner is the other supported apply path.

Paired change — et-operational-platform repo (do NOT edit from p24-infra)

Retarget the et-op workflows by weight, not host:

• Nightly Orchestrator / Auto Issue Pipeline (tiny) → [self-hosted, super-light]
• Anything heavier → [self-hosted, light]
• Playwright / build CI stays [self-hosted, bms4] (this policy still forbids heavy CI on vps-i1 — it OOM-kills the monitoring stack)

Result

GitHub dispatches et orchestration to any online host carrying the tier label, so intermittently-available hosts (vps-h1, dev-laptop) absorb load and no single dead systemd unit can strand the queue again. Do not add light/heavy to a small host, and never add heavy to vps-i1 — same OOM class as the 2026-06-24 incident. This is the forward-fix (prevents recurrence); it does not drain jobs already stuck behind a dead runner — that is a separate svc.sh start / re-register (#4221, and the auto-heal workflow above).

Registered runners on bms-4

bms-4 (8 vCPU, 32 GB RAM) is the dedicated heavy-CI host. Two repo-scoped self-hosted runners are online (verified 2026-06-25, #1154):

Runner name	Repo	Labels	Serves
bms4-etop	radieu/et-operational-platform	self-hosted, Linux, X64, bms4, heavy, playwright	et-op ci.yml (npm ci + unit tests + coverage), Playwright
bms4-p24infra	radieu/p24-infra	self-hosted, Linux, X64, bms4	p24-infra heavy jobs

Naming note (vs #1153): #1153 proposed a single {server}-{type}-{index} runner name per host (e.g. bms4-gw-1). In practice bms-4 runs one runner per repo (bms4-etop, bms4-p24infra), which is more descriptive and avoids a single runner being shared across repos. Issue #1154 originally asked for bms4-gw-1; the functional goal (move et-op CI off vps-i1 onto bms-4) is already met by bms4-etop. Renaming a self-hosted runner requires root on bms-4 (re-run config.sh), so any rename to the -gw- convention is a human-gated decision — not recommended given the per-repo scheme works.

Checklist when adding a new workflow

• Does the job run npm ci / pip install / playwright install? → [self-hosted, bms4] (or bms4, heavy)
• Does the job download large binaries (>50 MB)? → not on ionos
• Does the job run > 5 minutes? → not on ionos
• Does the job need localhost access to Prometheus/Alertmanager/Grafana on vps-i1? → ionos

Live runner-label audit — 2026-06-25 (#1154)

Audit of radieu/et-operational-platform/.github/workflows/* runs-on values:

Workflow	runs-on	Verdict
ci.yml	[self-hosted, bms4] (was [self-hosted, Linux])	✅ fixed in et-op#1099
rc-smoke-tests.yml	bare self-hosted	⚠️ violates Rule 1 — still picks up any self-hosted runner incl. vps-i1. The 2026-06-24 incident “Fix” below was never actually applied to the file. Follow-up needed.
trello-sync.yml	bare self-hosted	⚠️ violates Rule 1 — light job; pin to ubuntu-latest or [self-hosted, bms4]. Follow-up needed.
auto-issue-pipeline.yml, nightly-orchestrator.yml	[self-hosted, ionos]	OK — light infra/orchestration jobs intended for vps-i1
playwright-nightly.yml, sentry-bridge.yml, supabase-migrations.yml, technician-suite.yml	ubuntu-latest	OK

The two bare-self-hosted workflows are tracked as a follow-up to #1154 (they are out of scope for the runner-registration task but are the same OOM-risk class). Do not close the OOM-prevention work until both are pinned.

Removing a runner from vps-i1

ssh root@217.154.82.162
# List runner services
systemctl list-units 'actions.runner.*' --all
 
# Stop and disable a specific runner
systemctl stop  actions.runner.radieu-et-operational-platform.ionos
systemctl disable actions.runner.radieu-et-operational-platform.ionos
 
# Remove runner registration (needs runner token from GH Settings > Actions > Runners)
su -s /bin/bash <runner-user> -c '/opt/actions-runner/et-oper/config.sh remove --token <TOKEN>'

Incident reference

• Date: 2026-06-24
• Cause: rc-smoke-tests.yml used bare runs-on: self-hosted, picked up vps-i1 runner, ran npm ci + playwright install chromium → OOM → monitoring stack offline ~12h
• Intended fix: rc-smoke-tests.yml → [self-hosted, bms4, heavy]; et-oper runner removed from vps-i1
• ⚠️ As of 2026-06-25 the rc-smoke-tests.yml label change was NOT applied (file still bare self-hosted) — see the live audit above. The et-oper ionos runner removal did take effect.