p24-infra Docs

provision-windows-dev-docker-worker

11 min read

Playbook: Provision the windows-dev Docker Worker Node

Status update (2026-08-11): active guidance again. Briefly marked superseded by Pane (#6132); that pivot was reverted the same day after Pane’s headless daemon setup hit an unresolved architectural hang (see #6132’s final comment). This playbook is the real path forward. Go-live is tracked in #6163: the dispatcher-loop CF-Tunnel/docker run integration (prerequisite 5) is now implemented — the soak procedure below applies once the remaining on-host prerequisites (1–4) pass; don’t flip enabled=true before a clean soak.

Scope: Bring the windows-dev (WSL2) Docker ephemeral-worker node online (#6110 / ADR-005 §5b). Mirrors the discipline of provision-bms-3-dispatch-node.md: the capacity row ships DISABLED; activation (enabled=true) is the last step, by a human, after the live install and a soak.

Role: the artifacts (image, wrapper, migration, docs) are dev-coder deliverables and land via PR. Every step below marked [sys-admin] is a live server/Docker/CF-Tunnel operation on the windows-dev host and must be executed by sys-admin (or the infra-task worker), not inline by a dev-coder session. Any credential mount is a secret-manager prerequisite (ADR-005 §8) — a dev-coder/sys-admin session never writes SOPS.

What ships in the PR (already done by #6110)

ArtifactPurpose
docker/windows-dev-worker/Dockerfile + entrypoint.shthe maintained worker image
scripts/run-windows-dev-worker.shhost wrapper: -m/--cpus caps + 137→exit-2 translation
scripts/spawn-worker.sh (WORKER_ISOLATION=docker branch)foreground, no-systemd in-container launch
supabase/migrations/…_windows_dev_docker_worker_capacity_6110.sqldev_r_server_capacity row (DISABLED) + claude_accounts
docs/environments/windows-dev.mdhardware + capacity + contract spec
scripts/queue-dispatcher-loop.py (spawn_worker() docker-run branch) — #6163CF-Tunnel ProxyCommand + detached run-windows-dev-worker.sh launch

Prerequisites (before enabling)

  1. [sys-admin] WSL2 Ubuntu healthy on the dev workstation, with Docker Engine (docker-ce) installed natively inside the WSL2 distro — the standard apt-based Linux install, not Docker Desktop (owner decision on #6110, 2026-08-11). All code lives under the Linux FS (~/projects), never the C:\ mount (ADR-005 §5b).

    Why native Docker Engine, not Docker Desktop. Docker Desktop’s paid-tier requirement (250+ employees or $10M+ revenue) applies only to Docker Desktop itself — the Windows GUI app that manages its own hidden WSL2 distro. Docker Engine (docker-ce, the daemon) is free and open-source (Apache 2.0) with no company-size restriction. windows-dev is a headless, systemd-enabled Ubuntu instance (SSH + CF Tunnel, no GUI), so a native Engine install is both the correct fit and sidesteps the licensing question entirely — it is not a workaround.

    Install (once, [sys-admin] inside the WSL2 distro — Docker’s official Ubuntu instructions):

    # Docker Engine apt repo + install (docs.docker.com/engine/install/ubuntu)
sudo apt-get update && sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] \
  https://download.docker.com/linux/ubuntu $(. /etc/os-release && echo "$VERSION_CODENAME") stable" \
  | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin
# systemd-enabled WSL2 distro → the daemon runs as a normal service:
sudo systemctl enable --now docker
sudo usermod -aG docker "$USER"   # re-login so `docker` works without sudo
docker run --rm hello-world       # verify the daemon is up

  2. [sys-admin] CF Tunnel windows-dev-ssh.zintegrowana.online provisioned and reachable (same pattern as lap1/wsl1; see server-ssh-connectivity.md and p24-network-admin.md). Extend it to carry the docker run invocation of run-windows-dev-worker.sh, not just a bare SSH shell (ADR-005 §3).

  3. [secret-manager] Per-account claude auth login on the windows-dev host for each account in claude_accounts (claude-runner, claude-runner-2 — radieu deliberately excluded, 2026-09-01 owner decision: it’s the developer’s own daily-driver account and would compete with interactive use) — its own independent login, never a copied .credentials.json (ADR-002 Option C, banned). Concretely, inside WSL2 on the host (not in a container):

    mkdir -p /root/.claude-configs/claude-runner /root/.claude-configs/claude-runner-2
CLAUDE_CONFIG_DIR=/root/.claude-configs/claude-runner claude auth login
CLAUDE_CONFIG_DIR=/root/.claude-configs/claude-runner-2 claude auth login
# claude auth login needs a real pty (won't work piped) — run inside tmux for remote/scripted use:
#   tmux new-session -d -s claude_auth -x 220 -y 50
#   tmux send-keys -t claude_auth 'CLAUDE_CONFIG_DIR=... claude auth login' Enter
#   tmux capture-pane -t claude_auth -p   # read the printed OAuth URL
#   tmux send-keys -t claude_auth '<pasted code>' Enter
chown -R 1001:1001 /root/.claude-configs/claude-runner /root/.claude-configs/claude-runner-2

    1001:1001 is the claude user’s uid:gid baked into the image (docker run --rm --entrypoint id p24-windows-dev-worker:latest to confirm against a rebuilt image — this is not guaranteed stable across base-image changes). run-windows-dev-worker.sh (§6163 scope 4, implemented) reads CLAUDE_ACCOUNT (forwarded by the dispatcher) and mounts /root/.claude-configs/${CLAUDE_ACCOUNT} → /home/claude/.claude in the container — fails fast (exit 1) if either is missing rather than launching an unauthenticated container.

  4. [secret-manager] For secret-manager-role dispatch only: mount the SOPS age key into that role’s container. dev-coder/other roles get no age key mounted at all (ADR-005 §8).

  5. Dispatcher docker-dispatch path — DONE (#6163). queue-dispatcher-loop.py’s spawn_worker() now detects a windows-dev/docker-run host (_uses_docker_run_dispatch), wraps SSH in a cloudflared access ssh ProxyCommand for the CF Tunnel hostname (_is_cf_tunnel_host), and detached-launches run-windows-dev-worker.sh instead of spawn-worker.sh. Tested in scripts/tests/test_spawn_windows_dev_dispatch.py. This unblocks activation — the row is still left disabled until the on-host prerequisites 1–4 + the soak below pass.

  6. CF Access service token — REQUIRED before enabling dispatch (#6740). The windows-dev-ssh CF Tunnel’s Access application uses an owner-only browser-OTP policy (the fleet default, fleet-backup-ssh-cf-tunnel.md). The dispatcher is non-interactive and can never complete that OTP flow, so every cloudflared access ssh attempt is rejected at the Access layer with websocket: bad handshake before SSH auth starts — a 100% dispatch failure, each attempt costing the job a 5-min backoff before it falls back to bms-4. The fix is a CF Access service token (machine-to-machine credential, not the browser flow):

    • [secret-manager / p24-network-admin] Create a service token on the windows-dev-ssh Access application and add it as an allowed identity on the windows-dev-ssh-owner-only policy (Cloudflare dashboard/API). Store the pair in secrets/administration.env.sops under CF_ACCESS_WINDOWS_DEV_CLIENT_ID / CF_ACCESS_WINDOWS_DEV_CLIENT_SECRET, and deploy it into the dispatcher’s environment (the p24-queue-dispatcher.service EnvironmentFile) via the normal secrets-sync path — these are the env var names the dispatcher reads.
    • Dispatcher wiring — DONE (#6740). queue-dispatcher-loop.py reads those two env vars (CF_ACCESS_SERVICE_TOKEN_ID_VAR / CF_ACCESS_SERVICE_TOKEN_SECRET_VAR, overridable) and, for a CF-Tunnel host only, exports them to the ssh subprocess as TUNNEL_SERVICE_TOKEN_ID / TUNNEL_SERVICE_TOKEN_SECRET — the names cloudflared reads. They pass via the process environment, never an argv, so the token cannot leak into ps or the dispatcher log. The change is inert until the token is deployed: with the source vars unset the overlay is empty and behaviour is identical to the pre-#6740 interactive-OTP path. The same overlay + ProxyCommand now also covers the _read_sub_status account-headroom SSH (previously it did a direct connect to the tunnel hostname and hung until ConnectTimeout). Tested in scripts/tests/test_spawn_windows_dev_dispatch.py.
    • [sys-admin] Re-test from bms-4 before flipping dev_r_server_capacity.enabled back to true: ssh -o ProxyCommand="cloudflared access ssh --hostname %h" root@windows-dev-ssh.zintegrowana.online with the token env vars exported must connect without an OTP prompt.

Build the image [sys-admin, on the windows-dev host, inside WSL2]

cd ~/projects/p24-infra          # the Linux-FS checkout, never /mnt/c/...
# Build context is the repo root (the Dockerfile COPYs . into /opt/p24-infra). Add a .dockerignore
# to keep node_modules/.git/secrets out of the context.
docker build \
  -f docker/windows-dev-worker/Dockerfile \
  --build-arg GIT_SHA="$(git rev-parse --short HEAD)" \
  --build-arg BUILD_DATE="$(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --build-arg SERVER_LABEL=windows-dev \
  -t p24-windows-dev-worker:latest .

On any other WSL2 worker host under this same class (ADR-006 Phase 3 — e.g. lap2), pass that host’s own label instead, e.g. --build-arg SERVER_LABEL=lap2. SERVER_LABEL defaults to windows-dev for backward compatibility, but every host must build with its own — the image tag name (p24-windows-dev-worker:latest) can stay identical across hosts since Docker images are host-local, but .server-label baked inside the image is what spawn-worker.sh/ queue-dispatcher.sh use for worker IDs and audit trails; leaving it at the default on a non-windows-dev host silently misattributes every job.

Rebuild the image whenever spawn-worker.sh / agent prompts change — treat it as version-pinned infra. (CI build + checksum verification of the pinned sops/age binaries is a follow-up hardening.)

Smoke test [sys-admin]

# Dry cap check — confirm the wrapper builds the expected docker run args for a light job.
DOCKER_BIN=echo scripts/run-windows-dev-worker.sh 6110 dev-issue light | tr ' ' '\n' | grep -E 'memory|cpus'
# Expect (light tier, weight-tier update 2026-08-11): --memory 2g / --memory-swap 2g / --cpus 1
 
# Real single-job run against a throwaway test issue (subscription config mounted):
scripts/run-windows-dev-worker.sh <test-issue> dev-issue light radieu/p24-infra 0 dev_r_worker_queue dev-coder
echo "exit=$?"    # 0 = ok; 2 = OOM (bump caps / lower concurrency); 5 = subscription; 255 = infra

Confirm under a concurrent interactive-use test (run a light worker while actively using the machine): the foreground session must not visibly degrade (ADR-005 §5b UX requirement). Tune --cpus / reserved_ram_gb down if it does.

Activate [sys-admin / secret-manager — human, last step]

Only after all prerequisites pass and the dispatcher docker-path is merged:

UPDATE public.dev_r_server_capacity SET enabled = true WHERE server_label = 'windows-dev';

Then watch one dispatch cycle: confirm a light job dispatches over the tunnel, the caps hold, and the container reclaims cleanly (docker ps empty after exit). Update the dev_r_services rows to status='active'.

Rollback

UPDATE public.dev_r_server_capacity SET enabled = false WHERE server_label = 'windows-dev';

Disabling stops all new dispatch immediately; in-flight containers finish or are docker rm -f’d. No other host is affected — unclaimed rows fall back to bms-4.

Full environment restore — WSL2 distro reinstalled or lost

When this applies: the WSL2 Ubuntu distro itself is gone or reset (manual wsl --unregister, a failed wsl --uninstall/reinstall attempted to work around #6185’s instability, or any other event that wipes the guest filesystem). This is distinct from the “Rollback” section above, which only flips the dispatch flag — this section rebuilds the underlying host from nothing.

Nothing here is irreplaceable data. As of 2026-08-12 the distro held: Docker Engine + 2 images (p24-windows-dev-worker:latest, rebuildable from docker/windows-dev-worker/Dockerfile in this repo; hello-world, a smoke-test pull), cloudflared (config lives in Cloudflare, not locally — reprovisioning just needs the existing token), 1 SSH authorized_keys entry, and standard Ubuntu packages. No unique local state to back up before a reinstall, but wsl --export is cheap insurance if you want a rollback point anyway:

wsl --shutdown
wsl --export Ubuntu C:\Users\konar\wsl-backups\ubuntu-pre-reinstall-$(Get-Date -Format yyyyMMdd-HHmmss).tar

Steps to rebuild from a fresh WSL2 Ubuntu install

  1. Install/reinstall the distro (Microsoft Store or wsl --install -d Ubuntu). Confirm the WSL platform version matches what you intend — see docs/environments/windows-dev.md and #6185 for why the exact version matters here (a Store auto-update mid-instability-investigation is the live open question, not just cosmetic).

  2. Enable systemd — required for cloudflared, docker, and any future systemd-managed worker service:

    sudo tee /etc/wsl.conf > /dev/null << 'EOF'
[boot]
systemd=true
EOF

    Then wsl --shutdown from PowerShell and reopen the distro so it takes effect.

  3. Install Docker Engine — see Prerequisite 1 above (apt-based docker-ce, not Docker Desktop). Same commands, unchanged.

  4. Reinstall cloudflared and reconnect the existing tunnel — the Cloudflare Tunnel object (windows-dev-ssh) and its DNS CNAME already exist and don’t need recreating; only the local connector needs to be reinstalled with the existing run-token:

    curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo gpg --dearmor -o /usr/share/keyrings/cloudflare-main.gpg
echo "deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt-get update && sudo apt-get install -y cloudflared
    # On the Windows side — safe extraction pattern, never print the token
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:CF_TOKEN = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops |
  Select-String "^CF_TUNNEL_WINDOWS_DEV_TOKEN=").ToString().Split("=",2)[1]
wsl -d Ubuntu -u root -- bash -lc "cloudflared service install `$env:CF_TOKEN"
$env:CF_TOKEN = ""

    Verify: systemctl is-active cloudflared (expect active), then confirm DNS resolves and the tunnel shows status: healthy per docs/playbooks/fleet-backup-ssh-cf-tunnel.md’s verification steps.

  5. Re-add the SSH key — the same radieu-root-2026-06-27 public key (or current admin key) into /root/.ssh/authorized_keys, matching the pattern already used for lap1/wsl1 (docs/environments/lap1.md §Remote Access, docs/environments/wsl1.md §Setup Done).

  6. Rebuild the worker image — see “Build the image” above (docker build -f docker/windows-dev-worker/Dockerfile ...).

  7. Re-run claude auth login for each account in claude_accounts (claude-runner, claude-runner-2, radieu) — independent logins, never copy .credentials.json from another host (standing rule, CLAUDE.md §Claude Code OAuth Token Refresh).

  8. Confirm dev_r_server_capacity.ssh_host for windows-dev still points at the correct single-level hostname (windows-dev-ssh.zintegrowana.online) — a reinstall doesn’t touch the DB row, so this should already be correct, but verify rather than assume (see 6216-class regression history for why silent config drift is the recurring failure class here).

  9. Re-run the Smoke test section above before considering the host ready again. The row stays enabled=false until a fresh soak per ADR-006’s Phase 1 checklist.

Sequencing guard

Do not scale total fleet concurrency (raise max_workers* above 1–2, add heavy weights) until the bms-3 RAM-weight dispatch bug (#4717/#4728) is resolved — ADR-005 §5a. windows-dev has no operating history as a worker host, so this applies with extra force here.

Graph View

Backlinks