p24-infra Docs

gitlab-runner-bms1-reregister

8 min read

GitLab CI runners on bms-1 — re-register after token purge or rotation

Issue: #2078, #4829 Server: bms-1 (OVH ns367522, 94.23.26.113) — Pinbox24 production Status: runbook — autonomously executable (secret-manager + sys-admin) with GITLAB_ADMIN_PAT. Corrected 2026-08-01 (#4829): this was previously marked human/UI-only. That premise was wrong — see “Why this is autonomous” below. bms-1 is still Pinbox24 production, so treat every step with the same care as any live-CI change: verify before deleting old runners, keep the config.toml backup, confirm gitlab-runner verify is green before moving on.

Symptom

gitlab-runner on bms-1 cannot pick up CI jobs. All registered runner entries return 403 Forbidden from GitLab. CI/CD for the Pinbox24 microservices (s3-v2, mailgun) no longer auto-deploys; deploys fall back to the manual procedure.

Typical trigger: the gitlab-runner binary was upgraded across a major version boundary (e.g. 13.8.0 → 19.1.1 on 2026-06-29) and/or the old registration tokens were purged/expired in GitLab. Legacy registration tokens are deprecated in modern GitLab; new runners use the runner authentication token created per “New project runner”.

Confirmation

From bms-1 (# PLAYBOOK: gitlab-runner-bms1-reregister.md required for any SSH that modifies the host):

gitlab-runner --version          # confirm the installed binary version
gitlab-runner verify             # 403 on every runner == tokens purged/expired
systemctl is-active gitlab-runner

⚠️ Never run gitlab-runner list from an LLM/agent session. Unlike verify, its output includes each runner’s authentication token (Token=glrt-...) in plaintext — printing it into a Claude Code tool-result transcript is a full token exposure (incident: #4829, 2026-08-01). Use grep -E '^\s*name =' on config.toml (below) to get runner names/count instead.

Check the runner config WITHOUT printing token values (config.toml holds runner auth tokens):

# Count stale runner blocks only — never cat the file (it contains token values)
grep -c '^\[\[runners\]\]' /etc/gitlab-runner/config.toml
grep -E '^\s*concurrent' /etc/gitlab-runner/config.toml   # expect: concurrent = 4
grep -E '^\s*name =' /etc/gitlab-runner/config.toml       # runner descriptions only, no tokens

Why this is autonomous (corrected 2026-08-01, #4829)

Modern GitLab (the glrt-… runner-authentication-token format) manages runners through a real REST API — this is NOT the legacy shared “registration token” model that really was UI-only:

  1. Runner creation/deletion is API-driven — POST /api/v4/user/runners (or the project/group variants) creates a runner and returns its auth token in the response body; DELETE /api/v4/runners/:id revokes it immediately. No UI step required.
  2. GITLAB_ADMIN_PAT (secrets/administration.env.sops) already has full api scope — verified live via GET /api/v4/personal_access_tokens/self before relying on it. This scope covers runner management.
  3. Production SSH to bms-1 is a normal sys-admin operation, not a human-only gate — same trust level as any other secrets-sync/deploy step this repo already automates.
  4. Secret values still apply the usual rules: the new token is used once (piped straight from the API response into the SSH registration call) and never displayed, logged, or committed. Multi-project runners (bms-1-autodeploy spans 3 projects) need explicit attach/detach calls in addition to create/delete — GitLab blocks a direct DELETE /runners/:id on a runner still attached to more than one project (403); unassign the extra projects first (DELETE /projects/:id/runners/:runner_id).

Procedure

1. Resolve project IDs + confirm PAT scope (read-only)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:GLPAT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops | Select-String "^GITLAB_ADMIN_PAT=").ToString().Split("=",2)[1]
$headers = @{ "PRIVATE-TOKEN" = $env:GLPAT }
 
# Confirm scope (should include "api")
Invoke-RestMethod -Uri "https://gitlab.com/api/v4/personal_access_tokens/self" -Headers $headers | Select-Object name, scopes
 
# List current runners to identify old IDs + confirm project association per runner
Invoke-RestMethod -Uri "https://gitlab.com/api/v4/runners?per_page=100" -Headers $headers | Select-Object id, description, status
$env:GLPAT = ""

2. Create + register each new runner (one call per runner — the token is shown only once)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:GLPAT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops | Select-String "^GITLAB_ADMIN_PAT=").ToString().Split("=",2)[1]
$headers = @{ "PRIVATE-TOKEN" = $env:GLPAT }
 
$body = @{ runner_type = "project_type"; project_id = <PROJECT_ID>; description = "<NAME>"; tag_list = "<TAGS>"; run_untagged = $false; locked = $false } | ConvertTo-Json
$r = Invoke-RestMethod -Uri "https://gitlab.com/api/v4/user/runners" -Method Post -Headers $headers -ContentType "application/json" -Body $body
Write-Output ("Created runner id " + $r.id)
 
# Multi-project runner only: attach to each additional project BEFORE registering
# Invoke-RestMethod -Uri ("https://gitlab.com/api/v4/projects/" + $extraProjectId + "/runners") -Method Post -Headers $headers -ContentType "application/json" -Body (@{ runner_id = $r.id } | ConvertTo-Json)
 
# Register on bms-1 — pass the token as a plain command argument (values with no spaces need no
# quoting, which avoids the nested-quote/bash -c escaping trap that broke this exact step on
# 2026-08-01: PowerShell -> ssh -> bash -c '...' with 3 layers of quotes is extremely fragile).
$cmd = "gitlab-runner register --non-interactive --url https://gitlab.com --token " + $r.token + " --executor shell --description <NAME> --tag-list <TAGS> --run-untagged=false --locked=false"
ssh -o BatchMode=yes root@94.23.26.113 $cmd 2>&1 | Select-String -Pattern "Runtime platform|successfully|error|ERROR"
$env:GLPAT = ""
$r = $null   # token is gone once this call ends — GitLab will not show it again; if registration
              # fails, delete the orphaned runner via API and create a fresh one, don't retry the
              # same token

Repeat once per runner. If a register call fails (e.g. a quoting bug), the token is unrecoverable — delete the orphaned runner object (DELETE /api/v4/runners/:id) and create a new one rather than retrying with a broken command against the same token.

3. Clean stale runner blocks from config.toml

gitlab-runner register appends — it does not replace. After registering all replacements, config.toml will have both old and new blocks, often with identical names, so match by position (old blocks first, new ones appended at the end) or by counting [[runners]] occurrences — never by name alone.

ssh root@94.23.26.113   # PLAYBOOK: gitlab-runner-bms1-reregister.md
cp /etc/gitlab-runner/config.toml /etc/gitlab-runner/config.toml.bak-pre-rotation   # keep, don't delete immediately
grep -n '\[\[runners\]\]\|name = ' /etc/gitlab-runner/config.toml   # no tokens shown, safe
# Old blocks are the first N; new ones are the last N (registration order). Remove only the old
# block line-ranges, e.g.: sed -i '7,50d' /etc/gitlab-runner/config.toml
grep -E '^\s*name =' /etc/gitlab-runner/config.toml   # confirm exactly the expected N new names remain

4. Restart and verify

systemctl restart gitlab-runner
systemctl is-active gitlab-runner        # expect: active
gitlab-runner verify                     # all new runners "is alive", no 403 — NEVER use `gitlab-runner list`

Trigger a pipeline on each project (or push a no-op MR) and confirm a job is picked up by the autodeploy / prod-deploy-p24 tagged runner.

5. Delete the old runner objects (after verify is green)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:GLPAT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops | Select-String "^GITLAB_ADMIN_PAT=").ToString().Split("=",2)[1]
$headers = @{ "PRIVATE-TOKEN" = $env:GLPAT }
try {
  Invoke-RestMethod -Uri ("https://gitlab.com/api/v4/runners/" + $oldId) -Method Delete -Headers $headers
} catch {
  # 403 on a multi-project runner means unassign the extra projects first:
  # Invoke-RestMethod -Uri ("https://gitlab.com/api/v4/projects/" + $extraProjectId + "/runners/" + $oldId) -Method Delete -Headers $headers
}
$env:GLPAT = ""

Do this only after step 4’s gitlab-runner verify is green — deleting old runners while new ones are still unverified would leave zero working runners if something’s wrong.

6. Add the GITLAB_API_TOKEN CI/CD variable (human, only needed if not already set)

Auto-MR creation needs an API token exposed to the pipelines.

  • For both projects: Settings → CI/CD → Variables → Add variable
    • Key: GITLAB_API_TOKEN
    • Value: a GitLab personal/project access token with api scope (generate in GitLab; treat as a secret — never commit or log the value)
    • Flags: Masked = on, Protected = on (if pipelines run on protected branches only)

Reference the variable in CI as $GITLAB_API_TOKEN; never inline the value in .gitlab-ci.yml.

After completion

  • Update docs/playbooks/pinbox24-bms1-manual-deploy.md (referenced by #2078; currently missing — create it if the manual workaround is still in use) to note auto-deploy is restored.
  • If dev_r_services tracks the bms-1 GitLab runner as an element, confirm its row is current.
  • Close #2078 only after a real pipeline auto-deploys on both projects.

Secret-handling reminder

Runner authentication tokens (glrt-…), the resulting config.toml, and GITLAB_API_TOKEN are secret values. Never cat config.toml, never echo a token, never paste a token into a commit, comment, or log. If a token value is exposed, rotate it (regenerate the runner / revoke the access token) and follow docs/playbooks/static-api-key-incident-rotation.md.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="deploy",
    resource="gitlab-runner-bms1",
    result="success",  # "success" | "failed" | "skipped"
    detail="GitLab runner on bms-1 re-registered with fresh registration token",
    env="bms-1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'deploy', 'gitlab-runner-bms1', 'success', 'GitLab runner on bms-1 re-registered with fresh registration token', 'bms-1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks