adding-new-worker

Playbook: Adding a New Claude Code Worker

Scope: Register a new AI worker (Claude Code agent or GitHub Actions runner) so every session and task it runs is traceable in Supabase (agent_sessions.worker_env, agent_tasks.worker_id).

---

Naming Convention

{server-label}-{worker-type}-{index}

Segment	Meaning	Examples
server-label	Short identifier of the physical host	i1 (vps-i1), bms4 (bms-4), h1 (vps-h1), lap1 (Acer laptop)
worker-type	Role of the worker process	cw = Claude Code worker, gw = GitHub Actions runner, que = Supabase queue worker
index	1-based slot on that host	1, 2, 3 …

Examples:

• i1-cw-1 — first Claude Code worker on vps-i1
• bms4-cw-1 — first Claude Code worker on bms-4
• i1-gw-1 — first GitHub Actions runner on vps-i1
• bms4-gw-1 — first GitHub Actions runner on bms-4
• lap1-que-w1 — first Supabase queue worker on the local Acer laptop (lap1)
• lap1-gh-w1 — GitHub Actions runner on the local Acer laptop (lap1)

que worker-type — used for Supabase queue pollers on non-Linux hosts (e.g. Windows laptop) where cw (Claude Code worker via SSH) is not applicable. These workers poll p24_worker_queue outbound over HTTPS without needing inbound access.

---

Registered Workers

worker_id	Host	IP	Type	Status
i1-cw-1	vps-i1 (AI-Dev-IO1)	217.154.82.162	Claude Code worker	active
bms4-cw-1	bms-4 (AI-Dev-BMS4-1)	54.36.123.110	Claude Code worker	active
bms3-cw-1	bms-3 (OVH ns3129867)	51.68.155.224	Claude Code worker	pending provisioning (#2147)
bms3-cw-2	bms-3 (OVH ns3129867)	51.68.155.224	Claude Code worker	pending provisioning (#2147)
i1-gw-1	vps-i1	217.154.82.162	GH Actions runner	active (renamed from ionos-2, 2026-07-08, #1810)
bms4-gw-1	bms-4	54.36.123.110	GH Actions runner	active
lap1-que-w1	lap1 (Acer laptop, Windows)	<lap1-ip> (LAN)	Queue worker (Supabase poll)	pending activation
lap1-que-w2	lap1 (Acer laptop, Windows)	<lap1-ip> (LAN)	Queue worker (Supabase poll)	pending activation
lap1-gh-w1	lap1 (Acer laptop, Windows)	<lap1-ip> (LAN)	GH Actions runner	pending installation
wsl1-que-w1	wsl1 (WSL2 Ubuntu on dev-workstation)	CF Tunnel: wsl1-ssh.tunnel.zintegrowana.online	Queue worker (Supabase poll)	pending activation — see docs/environments/wsl1.md
windev-cw-1	windows-dev (WSL2 Docker on dev-workstation)	CF Tunnel: windows-dev-ssh.tunnel.zintegrowana.online	Claude Code worker (Docker ephemeral)	provision spec, ships DISABLED (#6110) — see docs/environments/windows-dev.md

bms-3 dispatch node (#2147): bms-3 is being brought online as a 3rd dispatch node. The dev_r_server_capacity row ships enabled = false; full bring-up (install, activation, soak, rollback) is in provision-bms-3-dispatch-node.md. Worker slots stay pending provisioning until the soak test passes.

windows-dev (WSL2/Docker) node (#6110 / ADR-005 §5b): the dev workstation’s WSL2 instance run as a Docker ephemeral-worker host (one docker run per job), reached over CF Tunnel inbound (ADR-005 §3) rather than the bare ssh … spawn-worker.sh used for Linux hosts. It is materially different from the Linux nodes above, so it gets its own subsection:

• Image: docker/windows-dev-worker/ (Dockerfile + entrypoint.sh) — bakes p24-infra + claude-code and runs spawn-worker.sh with WORKER_ISOLATION=docker (foreground, no systemd; ADR-005 §6 “reuse the contract, don’t replace it”).
• Host wrapper: scripts/run-windows-dev-worker.sh — the CF-Tunnel dispatch target. Applies the -m/--cpus caps (light → 4 GB / 2 CPUs, matching weight_ram_gb) and translates the RAM-gate/exit-code contract: a Docker OOM-kill (exit 137 / .State.OOMKilled) → the dispatcher’s exit 2; 3/4/5/6 pass through; docker infra failure (125/126/127) → 255.
• FS rule: all code under the WSL2 Linux FS (~/projects), never the C:\ mount.
• Secrets (ADR-005 §8): per-role at the mount level — age key mounted only for secret-manager containers; dev-coder/others get none.
• claude_accounts: existing accounts only (claude-runner/claude-runner-2/radieu) — no new seats; concurrent same-account use across hosts is an accepted risk (ADR-005 §4/§4a).
• Runs concurrent with interactive use — hard caps protect the foreground session; start at 1 light worker and validate empirically before raising (ADR-005 §5b).
• Ships DISABLED; full bring-up is in provision-windows-dev-docker-worker.md.

---

Steps: Register a New Claude Code Worker

1. Set CLAUDE_WORKER_ID on the server

SSH to the target server and add the env var to the claude-runner user’s shell profile:

# SSH to server
ssh root@<server-ip>
 
# Add to claude-runner's bashrc (persists across tmux sessions)
echo 'export CLAUDE_WORKER_ID=<worker-id>' >> /home/claude-runner/.bashrc
 
# If using a systemd service for claude-runner, add to the [Service] block:
# Environment=CLAUDE_WORKER_ID=<worker-id>
# Then: systemctl daemon-reload && systemctl restart claude-runner

Why .bashrc alone is not enough — and how the id actually reaches a worker (#1810). The dispatcher spawns workers via su -s /bin/bash claude-runner -c ..., a non-login, non-interactive shell. Such a shell never sources ~/.bashrc (the stock guard case $- in *i*) ;; *) return;; returns first), so the exported value does not reach the worker environment on its own. spawn-worker.sh therefore resolves the canonical id by reading the export CLAUDE_WORKER_ID= line straight out of ~/.bashrc (last occurrence wins) — so the .bashrc line above is still the source of truth, it is just read by grep, not by sourcing. The queue job ref is exported separately as CLAUDE_JOB_REF.

# Verify the value spawn-worker.sh will resolve (grep, since the shell won't source .bashrc):
grep -hoE '^[[:space:]]*export[[:space:]]+CLAUDE_WORKER_ID=[^[:space:]]+' \
  /home/claude-runner/.bashrc | tail -1
# NOTE: `su -s /bin/bash claude-runner -c 'echo $CLAUDE_WORKER_ID'` prints EMPTY here —
# that is expected (non-interactive shell skips .bashrc), not a misconfiguration.

2. Verify worker-issue.md reads the env var

The worker-issue.md session setup block reads CLAUDE_WORKER_ID and writes it into agent_sessions.worker_env as JSON:

{"worker_id": "<worker-id>", "host": "<ip>", "hostname": "<hostname>"}

And into agent_tasks.worker_id when claiming a task.

Confirm this works by running a test session and checking:

SELECT session_id, worker_env FROM agent_sessions ORDER BY started_at DESC LIMIT 3;
SELECT id, worker_id FROM agent_tasks WHERE worker_id IS NOT NULL ORDER BY created_at DESC LIMIT 3;

3. Register in dev_r_services

Run the following SQL via mcp__claude_ai_Supabase__execute_sql or Supabase dashboard:

INSERT INTO dev_r_services (
  project_id, project_name, office_id, ws_id, app_id, owner,
  service_name, service_type, status, host,
  description, compliance_workbook, workbook_url, compliance_notes
)
SELECT
  'p24-infra', 'p24-infra', 'p24-devops', '99', 'et-app', 'radieu',
  '<worker-id>', 'automation', 'active', '<ip>',
  'Claude Code worker slot <n> on <server-label> (<GH-user>). Runs as claude-runner. Identifies itself via CLAUDE_WORKER_ID=<worker-id> env var.',
  'yes', 'docs/playbooks/adding-new-worker.md',
  'Set CLAUDE_WORKER_ID=<worker-id> in /home/claude-runner/.bashrc.'
WHERE NOT EXISTS (SELECT 1 FROM dev_r_services WHERE service_name = '<worker-id>');

4. Update this playbook

Add the new worker to the Registered Workers table above.

---

Steps: Register a New GitHub Actions Runner

1. Install the runner on the server

Follow the standard GitHub Actions self-hosted runner installation. Use the naming convention for the runner name:

# During ./config.sh, set:
# Runner name:   <worker-id>    e.g. i1-gw-2
# Runner labels: self-hosted,linux,<server-label>   e.g. self-hosted,linux,i1

2. Register in dev_r_services

INSERT INTO dev_r_services (
  project_id, project_name, office_id, ws_id, app_id, owner,
  service_name, service_type, status, host,
  description, compliance_workbook, workbook_url, compliance_notes
)
SELECT
  'p24-infra', 'p24-infra', 'p24-devops', '99', 'et-app', 'radieu',
  '<worker-id>', 'automation', 'active', '<ip>',
  'GitHub Actions self-hosted runner on <server-label>. Runner name: <worker-id>.',
  'yes', 'docs/playbooks/adding-new-worker.md',
  'Runner labels: self-hosted,linux,<server-label>. GH token stored in runner service.'
WHERE NOT EXISTS (SELECT 1 FROM dev_r_services WHERE service_name = '<worker-id>');

3. Reference in GitHub Actions workflows

Use the runner label in .github/workflows/*.yml:

jobs:
  deploy:
    runs-on: [self-hosted, linux, i1]  # targets all i1 runners
    # OR target a specific runner:
    runs-on: i1-gw-1

---

---

SSH Access for Cross-Server Push (pushgateway, etc.) — MANDATORY when a worker pushes over SSH

Some worker scripts push data to another server over SSH while running as a dedicated Linux worker user. The canonical example is scripts/check-sub-usage.py, which runs as claude-runner / claude-runner-2 on bms-4 and pushes Claude-sub metrics to the vps-i1 pushgateway with a plain ssh root@217.154.82.162 ... (no -i flag).

Whenever a worker Linux user needs SSH access to another server, both of the following must be provisioned — and verified, not assumed:

1. The private key must be resolvable by the SSH client

ssh (without -i) only offers default-named identity files (~/.ssh/id_rsa, id_ecdsa, id_ed25519, …). A key with any other filename (e.g. vps_root_key) is never offered unless an explicit ~/.ssh/config entry points to it — the client silently gives up with Permission denied (publickey). So the worker user’s key must EITHER:

• use a default filename (~/.ssh/id_ed25519), OR
• have an explicit ~/.ssh/config IdentityFile entry for the target:

# As the worker user's home (do NOT rename/move an existing key — add config instead):
cat > /home/<worker-user>/.ssh/config << 'EOF'
Host <target-ip>
  IdentityFile ~/.ssh/<actual_key_filename>
  IdentitiesOnly yes
EOF
chown <worker-user>:<worker-user> /home/<worker-user>/.ssh/config
chmod 600 /home/<worker-user>/.ssh/config

2. The matching public key must be present in the target’s authorized_keys

Do not assume it — verify it during provisioning. A prior key rotation may have dropped it:

# Derive the public key from the worker user's private key and confirm it is authorized:
PUB=$(ssh-keygen -y -f /home/<worker-user>/.ssh/<actual_key_filename>)
ssh root@<target-ip> "grep -qF \"${PUB%% *} ${PUB#* }\" /root/.ssh/authorized_keys && echo present || echo MISSING"
# If MISSING → back up authorized_keys first, then append the public key.

3. Verify end-to-end — as the worker user, not as root

The dispatcher spawns workers via non-login shells, and root’s own key is not the worker user’s key, so always test as the worker user:

su -s /bin/bash <worker-user> -c 'ssh -o BatchMode=yes -o ConnectTimeout=5 root@<target-ip> echo OK'
# Expect: OK   (anything else — especially "Permission denied (publickey)" — is a provisioning gap)

Why this matters (#3419, 2026-07-08 incident). When this step is skipped, the SSH push fails silently: the metric simply stops arriving and the Grafana panel goes N/A with no error anywhere but the raw cron log. See grafana-claude-sub-monitoring.md §1a for the full diagnosis (N/A = missing SSH push vs. zero-value = expired OAuth).

4. Register the pair in the drift check

scripts/check-ssh-push-connectivity.py probes every known (worker-user → target) SSH-push pair and raises a Discord alert + GitHub issue (and pushes p4_ssh_push_ok{user,target} to the pushgateway) when one breaks — so a silent auth break surfaces as an alert instead of a slowly darkening dashboard. When you add a new SSH-push worker, add its pair to PUSH_PAIRS in that script. It runs nightly via nightly-infra-check (Phase 12) and can also be run standalone:

# On bms-4, as root (cron-friendly — non-fatal, always exits 0):
python3 /opt/p24-infra/scripts/check-ssh-push-connectivity.py
# Recommended standalone cron (belt-and-suspenders alongside the nightly agent):
#   17 */6 * * * root python3 /opt/p24-infra/scripts/check-ssh-push-connectivity.py >> /var/log/p24-infra-workers/ssh-push-check.log 2>&1

---

Steps: Register a Local/Laptop Node (Windows, outbound-only)

Local nodes such as lap1 (Acer laptop, Windows 10) differ from VPS-based workers in several ways. Use this section instead of the SSH-based steps above.

Key differences vs. VPS nodes

Aspect	VPS nodes (i1, bms4, h1)	Local/laptop node (lap1)
OS	Linux (Ubuntu / AlmaLinux)	Windows 10
SSH access	root SSH from dev machine	Not applicable — no inbound
Claude worker shell	bash + CLAUDE_WORKER_ID in .bashrc	PowerShell / env var in launcher script
Docker	Yes — Docker Compose stacks	No Docker for workers
Uptime	Near-100% (VPS)	Variable — laptop may be off
Job priority	P0–P2	P1–P2 only (never P0)
Queue polling	spawn-worker.sh via dispatcher	Direct outbound HTTPS to Supabase

1. Set CLAUDE_WORKER_ID on the laptop

In the worker launcher script (PowerShell) or Windows system environment:

$env:CLAUDE_WORKER_ID = "lap1-que-w1"   # or lap1-que-w2
# Verify (never print the value — just confirm it is set)
[bool]$env:CLAUDE_WORKER_ID

For a persistent Windows service, add CLAUDE_WORKER_ID as a system environment variable via System Properties → Advanced → Environment Variables.

2. Install GitHub Actions runner (lap1-gh-w1 only)

Download the Windows runner package from https://github.com/radieu/p24-infra/settings/actions/runners/new?arch=x64.

During ./config.cmd configuration:

Runner name:   lap1-gh-w1
Runner labels: self-hosted,windows,lap1

Install as a Windows service so it persists across reboots:

./svc.cmd install
./svc.cmd start

3. Register in dev_r_services

Use the SQL INSERT blocks in docs/environments/lap1.md §dev_r_services Registration.

4. Priority filter — IMPORTANT

lap1 must never accept P0 jobs. P0 = incident response, sys-admin operations, any task that requires guaranteed availability. If the worker capacity table (dev_r_server_capacity) has a row for lap1, set max_jobs low (1–2) and add a note in description about variable uptime.

5. Update this playbook

Add the new worker IDs to the Registered Workers table above.

---

Done: Renamed ionos-2 to i1-gw-1 (2026-07-08, #1810)

The IONOS vps-i1 GitHub Actions runner (registered directory /opt/actions-runner-ionos-infra-2, systemd unit actions.runner.radieu-p24-infra.ionos-localhost-2.service) had already been renamed once from ionos-localhost to ionos-2 — still off the naming convention. Renamed to i1-gw-1 directly via SSH:

ssh root@217.154.82.162
cd /opt/actions-runner-ionos-infra-2
systemctl stop actions.runner.radieu-p24-infra.ionos-localhost-2.service
sudo -u claude-runner ./config.sh remove --token <REMOVE_TOKEN>
sudo -u claude-runner ./config.sh --url https://github.com/radieu/p24-infra --token <REGISTER_TOKEN> \
  --name i1-gw-1 --labels self-hosted,linux,ionos,secrets-deploy,i1 --unattended --replace
systemctl start actions.runner.radieu-p24-infra.ionos-localhost-2.service

New GH agent id 29, dev_r_services row i1-gw-1 inserted. The systemd unit/service name still says ionos-localhost-2 (cosmetic only — matches the working directory, not worth a service rename/re-link for a label that’s internal-only).

---

How Worker Identity Flows into Supabase

claude-runner env: CLAUDE_WORKER_ID=i1-cw-1
        │
        ▼
worker-issue.md Step 0 — session registration:
  INSERT INTO agent_sessions (..., worker_env)
  VALUES (..., '{"worker_id":"i1-cw-1","host":"217.154.82.162","hostname":"vps-i1"}')
        │
        ▼
worker-issue.md Step 5 — task claim:
  UPDATE agent_tasks SET worker_id = 'i1-cw-1'
  WHERE claimed_by = '<session-id>' AND github_issue_id = <n>

Wired entry points (both resolve CLAUDE_WORKER_ID → worker_env at session registration):

Entry point	Table written	Identity columns
infra/agent-prompts/worker-issue.md (Step 0)	agent_sessions + dev_r_agent_sessions, agent_tasks	worker_env, server_node, worker_id
infra/agent-prompts/infra-task-request-worker.md (Step 3)	dev_r_agent_sessions	worker_env, server_node

Both use the same fallback: if CLAUDE_WORKER_ID is unset or still carries a job ref (contains :, /, or a space), they write <server-label>-cw-unknown so worker_env is never NULL. Set the canonical CLAUDE_WORKER_ID per machine (criterion below) to populate real ids.

Both tables are queryable in Grafana via grafana_readonly:

-- Which worker ran each session in the last 24h?
SELECT session_id, worker_env->>'worker_id' as worker, started_at
FROM agent_sessions
WHERE started_at > NOW() - INTERVAL '24h'
ORDER BY started_at DESC;
 
-- Which worker claimed each task?
SELECT id, github_issue_id, worker_id, status, claimed_at
FROM agent_tasks
WHERE worker_id IS NOT NULL
ORDER BY claimed_at DESC LIMIT 20;

---

Schema Reference

Table	Column	Type	Purpose
agent_sessions	worker_env	jsonb	{worker_id, host, hostname} — set at session start
agent_tasks	worker_id	text	worker_id string — set at task claim
dev_r_agent_sessions	worker_env	jsonb	Same as above for the audit log table
dev_r_services	service_name = <worker-id>	—	Registry row for the worker slot

---

*Written 2026-06-23 — Implements 1153

---

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="deploy",
    resource="claude-worker",
    result="success",  # "success" | "failed" | "skipped"
    detail="New Claude worker deployed and registered — runner configured on target server",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'deploy', 'claude-worker', 'success', 'New Claude worker deployed and registered — runner configured on target server', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''