gh-runner-restart-offline

GH Actions self-hosted runner — restart offline runner on vps-i1

Trigger: health-check runner-et or runner-kdp step fails (systemctl is-active returns non-active). The runner process is not running on vps-i1 (217.154.82.162), causing queued jobs to stay queued indefinitely.

Before you restart — rule out a job-level failure (not a runner outage)

“Many workflows stuck queued for 20–30 min” does not by itself mean the runner is down. A healthy runner can leave jobs queued while a job-level step fails on a permission gap. If a job reached in_progress (even if it then failed) or the runner API reports status: online, this is not a runner outage and a restart is wasted — see docs/playbooks/gh-actions-failed-jobs-taxonomy.md Group K (queued + EACCES: permission denied, stat '/root/.local/bin/git' = drift sudoers, 3344). Confirm the runner is genuinely offline before proceeding:

gh api repos/radieu/p24-infra/actions/runners --jq '.runners[] | {name, status, busy}'   # offline?
gh run list --repo radieu/p24-infra --limit 10 --json status,conclusion,name              # any in_progress → runner is live

Before you restart — rule out a GitHub-side platform incident (#5785, 2026-08-06)

The /actions/runners status/busy fields and job dispatch are themselves part of GitHub’s Actions control plane. During a GitHub-wide Actions incident, that API can report runners offline (or busy: true with no matching in_progress run) while the runner process is completely healthy — because the platform-side status reporting/dispatch is what’s degraded, not the runner. Restarting or re-registering a runner in this state fixes nothing (the runner was never actually down) and wastes time that’s better spent waiting out the incident.

Check this BEFORE touching any runner service, every time gh api .../actions/runners disagrees with what you can see on the box:

# 1. Is GitHub Actions itself degraded right now?
curl -fsS https://www.githubstatus.com/api/v2/status.json | jq .status.description
# Also check https://www.githubstatus.com/ directly for an active "Actions" incident —
# the JSON summary can lag the incident page by a few minutes.
 
# 2. Ground-truth the runner locally — trust systemd + journalctl over the GH API during an incident.
#    "active running" + a job completing in the last few minutes = the runner is fine.
ssh root@<host> "systemctl is-active actions.runner.<unit>; \
  journalctl -u actions.runner.<unit> -n 15 --no-pager"

Diagnostic signature confirmed 2026-08-06 (issue #5785): GH status page showed an active Actions incident (start 15:22 UTC, “some workflow runs failed to start/complete, Actions REST API errors, impact later extended to hosted runners”) lasting several hours. During that window:

• gh api .../actions/runners reported bms4-p24infra, bms4-p24infra-extra1/2/3, and i1-gw-1 as offline (flickering to online and back within minutes, no config change on our side).
• SSH to bms-4/vps-i1 showed every actions.runner.* systemd unit active running, the Runner.Listener process alive with a multi-day uptime, and journalctl showing a continuous stream of Running job: X → Job X completed with result: Succeeded pairs — including jobs that finished 1-2 minutes before the API said the runner was offline.
• A plain runs-on: ubuntu-latest job (deploy-meta-dispatcher.yml, no self-hosted runner involved at all) sat queued with runner_id: 0 for 35+ minutes — proof the delay was GitHub’s own hosted-runner dispatch, not anything wrong with our self-hosted fleet.
• No zombie process, no OOM, no disk/CPU exhaustion on either host (df -h, free -m, docker ps all normal) — resource exhaustion was ruled out, not assumed.

Action in this case: do nothing to the runners. Comment on the incident issue with the githubstatus.com link and the ground-truth evidence above, leave the issue open, and re-check the backlog once GitHub’s status page marks the incident resolved. See also docs/playbooks/github-unavailability.md for the broader GitHub-outage response plan (git mirror, manual deploys, escalation) if the incident is prolonged enough to need it.

Confirmation

From any host with SSH to vps-i1:

ssh root@217.154.82.162 "systemctl is-active actions.runner.radieu-et-operational-platform.ionos; \
  systemctl is-active actions.runner.radieu-amazon-kdp-tango.kdp-ionos-runner"
# Expect: active / active
# If inactive / failed / dead — restart required

Or check via health-check GH Actions run: health-runners job → Check GH runner — et-operational-platform/ionos.

Fix — restart runner services

SSH to vps-i1 and restart both runner systemd services:

ssh root@217.154.82.162  # PLAYBOOK: gh-runner-restart-offline.md
 
# Check current status
systemctl status actions.runner.radieu-et-operational-platform.ionos --no-pager
systemctl status actions.runner.radieu-amazon-kdp-tango.kdp-ionos-runner --no-pager
 
# Restart both
systemctl restart actions.runner.radieu-et-operational-platform.ionos
systemctl restart actions.runner.radieu-amazon-kdp-tango.kdp-ionos-runner
 
# Verify
systemctl is-active actions.runner.radieu-et-operational-platform.ionos    # expect: active
systemctl is-active actions.runner.radieu-amazon-kdp-tango.kdp-ionos-runner # expect: active

Verify resolution

After restart, wait ~60s, then re-run the health-check workflow manually:

gh workflow run health-check.yml --repo radieu/p24-infra

Or check GH → p24-infra → Actions → Infrastructure Health Check → Run workflow. runner-et and runner-kdp steps should show ✓.

If restart “succeeds” but the runner stops again — registration deleted server-side

Failure signature (most common — restart is NOT enough): systemctl restart reports active for ~2 seconds, then the service deactivates on its own and is-active returns inactive again. journalctl -u actions.runner.* -n 15 shows:

runsvc.sh[…]: Failed to create a session. The runner registration has been deleted from the
              server, please re-configure. Runner registrations are automatically deleted for
              runners that have not connected to the service recently.
runsvc.sh[…]: Runner listener exit with terminated error, stop the service, no retry needed.
systemd[…]:   <service>.service: Deactivated successfully.

GitHub auto-purges a self-hosted runner’s registration after it has been disconnected for ~14 days. The local .runner/.credentials files survive on disk, but the server-side registration is gone, so the listener can no longer authenticate. A systemctl restart cannot fix this — the runner must be re-registered with a fresh registration token. (Seen 2026-06-30, issue #2181: both ionos runners purged simultaneously after a long disconnection.)

Permission requirement (why an autonomous worker usually cannot self-heal this)

Minting a registration token requires administration:write on the target repo:

gh api -X POST repos/<owner>/<repo>/actions/runners/registration-token --jq .token

The standard worker PAT and the n8n-io-pull-request-flow GitHub App both lack this scope (the App’s administration:write is pending human re-auth — see priorities.md). When the token cannot be minted, escalate to human-action with the re-registration steps below; do not loop on systemctl restart.

Re-register (needs a fresh registration token — human or admin-scoped token)

Runner → working-directory map on vps-i1 (from the systemd unit WorkingDirectory, all run as user github-runner):

Service	Repo	Runner name	Working dir
actions.runner.radieu-et-operational-platform.ionos	radieu/et-operational-platform	ionos	/home/github-runner/actions-runner
actions.runner.radieu-amazon-kdp-tango.kdp-ionos-runner	radieu/amazon-kdp-tango	kdp-ionos-runner	/opt/actions-runner-kdp

# Example for the et-operational-platform runner — substitute repo/name/dir for kdp from the table.
REPO=radieu/et-operational-platform; NAME=ionos; DIR=/home/github-runner/actions-runner
TOKEN=$(gh api -X POST repos/$REPO/actions/runners/registration-token --jq .token)  # needs admin
systemctl stop actions.runner.radieu-et-operational-platform.ionos
sudo -u github-runner bash -c "cd $DIR && ./config.sh remove --token $TOKEN" || \
  sudo -u github-runner bash -c "cd $DIR && ./config.sh remove --local"   # if server reg already gone
sudo -u github-runner bash -c "cd $DIR && ./config.sh --url https://github.com/$REPO \
  --token $TOKEN --name $NAME --labels self-hosted,ionos --unattended --replace"
systemctl start actions.runner.radieu-et-operational-platform.ionos
systemctl is-active actions.runner.radieu-et-operational-platform.ionos   # expect: active (and stays active)

After re-registering, confirm the runner shows Idle (green) in GH → repo → Settings → Actions → Runners, then re-run the health check (see “Verify resolution”).

Mandatory follow-up if the re-registered runner is ionos-2 or bms4-p24infra — re-apply secrets-deploy

Both ionos-2 (id 28, on vps-i1) and bms4-p24infra (id 26, on bms-4) additionally carry a custom secrets-deploy label (added out-of-band via the GitHub API, 3233) so secrets-sync.yml’s sync-pinbox24-w3/sync-pinbox24-w4/sync-pinbox24-backends jobs fail over between the two instead of queuing forever when one host is offline. config.sh in the steps above does not carry this label over — a re-registered runner comes back with only self-hosted,ionos (or self-hosted,bms4), silently dropping secrets-deploy until someone notices the next time the other host goes down. Full background: docs/playbooks/gh-runner-assignment-policy.md §secrets-deploy shared label.

For ionos-2: the github-runner Ansible role (ansible/roles/github-runner/tasks/main.yml) now re-applies github_runner_extra_labels via the GitHub API on every run, so re-registering through Ansible instead of raw config.sh carries the label over automatically — pass it as an extra-var so it reaches the p24-infra secondary runner slot:

ansible-playbook playbooks/provision-new-vps.yml -i "217.154.82.162," \
  --extra-vars "github_runner_label=ionos github_runner_token_infra=<NEW> github_runner_extra_labels=[\"secrets-deploy\"]"

(playbooks/vps-i1.yml’s single github-runner role instance targets the decommissioned et-operational-platform slot, not ionos-2 — always go through provision-new-vps.yml with github_runner_token_infra for this runner, same as the role README’s documented re-registration steps.) If you instead re-register by hand with raw config.sh (as in the example above), run the one-liner below.

For bms4-p24infra: the Ansible role is not wired into ansible/playbooks/bms-4.yml for this runner (it’s registered outside Ansible’s management entirely) — re-registration is always manual, so this step is always required, not just a fallback.

Either way — verify: after any re-registration (Ansible or manual), confirm the label actually landed before closing out the incident (see “Verify” below); don’t assume the Ansible path worked silently.

Copy-paste recovery (needs an admin-scoped gh session — the same one used to mint the registration token above; run right after ./svc.sh start and the is-active check):

# Substitute REPO/NAME for whichever runner you just re-registered:
#   ionos-2         → REPO=radieu/p24-infra NAME=ionos-2
#   bms4-p24infra   → REPO=radieu/p24-infra NAME=bms4-p24infra
REPO=radieu/p24-infra; NAME=ionos-2
RUNNER_ID=$(gh api repos/$REPO/actions/runners --jq ".runners[] | select(.name==\"$NAME\") | .id")
gh api --method POST "repos/$REPO/actions/runners/$RUNNER_ID/labels" \
  --input - <<< '{"labels":["secrets-deploy"]}'
 
# Verify:
gh api repos/$REPO/actions/runners --jq '.runners[] | {id,name,labels:[.labels[].name]}'

Confirm secrets-deploy appears in the output before considering the re-registration complete.

If restart fails — runner token expired (legacy 401 case)

Symptoms: journalctl -u actions.runner.* -n 30 shows Unauthorized or 401 (rather than the “registration has been deleted” line above). Same fix — re-register with a fresh token using the steps above.

Root cause notes

• Runners on vps-i1 can go offline after: OS reboot (if service not set to restart), OOM kill, process crash. These are recoverable with a plain systemctl restart.
• Server-side registration deletion (the “If restart succeeds but stops again” section above) is a different failure: GitHub purges the registration after a prolonged disconnection, so a restart cannot recover it — re-registration is mandatory. Both services are enabled, so a reboot alone would not cause this; the underlying trigger is the service having been stopped/disconnected for ~2 weeks.
• The health-runners job runs on [self-hosted, bms4] — it will still run even when vps-i1 runners are down.
• vps-i1 is monitoring-only (no Claude workers); runner outage does not block Supabase queue dispatch. It DOES stall queued jobs on radieu/et-operational-platform and radieu/amazon-kdp-tango.
• A GitHub-side Actions platform incident (see “Before you restart — rule out a GitHub-side platform incident” above) can make gh api .../actions/runners report healthy runners as offline/falsely busy, and can stall even GH-hosted ubuntu-latest jobs. Always check githubstatus.com and ground-truth via SSH before restarting on this symptom alone.

Related

• docs/playbooks/adding-new-worker.md — initial runner registration
• docs/playbooks/gh-runner-assignment-policy.md — which jobs use which runners
• Issue #1829 — 2026-06-28 incident: both ionos runners offline simultaneously after Traefik + runner crash
• Issue #2181 — 2026-06-30 incident: both ionos runner registrations purged server-side; restart insufficient, re-registration required (added the “registration deleted server-side” section).
• docs/playbooks/github-unavailability.md — broader GitHub-outage response plan
• Issue #5785 — 2026-08-06 incident: gh api .../actions/runners reported healthy bms-4/vps-i1 runners offline/falsely-busy during a GitHub-wide Actions platform incident; no restart needed, added the “rule out a GitHub-side platform incident” section above.

---

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="gh-runner-ionos",
    result="success",  # "success" | "failed" | "skipped"
    detail="GH Actions self-hosted runner restarted or re-registered on vps-i1",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'gh-runner-ionos', 'success', 'GH Actions self-hosted runner restarted or re-registered on vps-i1', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''