p24-infra Docs

ondemand-burst-vps

10 min read

On-demand burst VPS — scaling workers when the queue backs up

Issue: #1244 Status: Plan / human-action — pick one option before any implementation Author of deliverable: worker agent (bms-4), 2026-06-25 Labels: human-action, plans-review

This is a planning/decision document, not an implementation. It analyses the market for short-lived paid compute, then lays out three architecture options (with strengths and weaknesses each) for renting compute on demand to absorb worker-queue backlog — including the specific “rent a box just for one e2e test session, then destroy it” use case. No code ships until a human picks an option.

1. Problem statement

Our worker fleet is fixed-size:

HostMax parallel agents
AI-Dev-IO1 (vps-i1)2–3
AI-Dev-BMS4-1 (bms-4)4

When dev_r_worker_queue fills faster than that fleet drains it (a wave of issues, a nightly batch, or an e2e-test run that needs a clean dedicated box), jobs sit in status=queued for a long time. We want to temporarily rent extra paid compute, run workers on it for a short window, then destroy it so we only pay for the minutes used. The cheapest credible way to do that is hourly/per-second-billed cloud compute that we provision and tear down via API.

What we already have (reuse, don’t rebuild)

The burst mechanism is mostly glue on top of assets that already exist:

AssetRole in burst flow
ansible/playbooks/provision-new-vps.ymlBrings a bare host to “useful state”: common, docker, claude-runner, github-runner, audit-daemon roles. Already parameterised by github_runner_label + tokens.
dev_r_server_capacity (Supabase)Server registry the dispatcher reads — server_label, ssh_host, ssh_user, max_workers_{prime,night}, max_weight_{prime,night}. Add a row → dispatcher uses the host. Delete the row → it stops.
scripts/queue-dispatcher-loop.pyLeader (vps-i1) already “fills free slots on all enabled servers”. It reads dev_r_server_capacity and dispatches dev_r_worker_queue rows. This is the natural place to read backlog depth.
dev_r_worker_queue statusesqueued/claimed/running/done/failed/oom_killed/cancelled — re-dispatch + failure handling already exist, which matters for evictable/spot compute.
SOPS (secrets/*.env.sops)Source for the new host’s claude-runner OAuth creds, GH runner token, age key.

So every option below is fundamentally: (a) a trigger that watches backlog, (b) a provisioner that creates a cloud box, runs the existing playbook, and registers it in dev_r_server_capacity, and (c) a reaper that destroys the box and de-registers it when the queue drains. The options differ in what kind of compute we rent and how much glue each needs.

2. Market analysis — short-lived paid compute (June 2026)

Selection criteria: billed by the hour or finer, EU region available (GDPR — our data is EU-resident), scriptable provisioning API, cheap.

ProviderBilling granularityEU regionAPI qualityIndicative price (4 vCPU / 8 GB, shared)Notes
Hetzner CloudHourly, with monthly capYes (Falkenstein/Nuremberg DE, Helsinki FI)Good (hcloud CLI + Terraform provider)CPX line; but CPX/CCX prices rose 113–209 % effective 2026-06-15Was the price-performance king; the June 2026 hike materially changes the math. Still cheap on the new shared CX plans.
VultrHourly on every planYes (Frankfurt, Amsterdam, Paris, Warsaw)Cleanest provisioning API on the market (per the 2026 comparisons)~5–24/mo equiv; e.g. ~0.03–0.06/hr smallBest fit for deploy → use → destroy. Warsaw region is latency-close to our OVH/IONOS EU footprint.
DigitalOceanPer-second, 60 s minimumYes (Frankfurt, Amsterdam)Good (Terraform, doctl)Basic droplet ~$24/mo equivFinest billing granularity — ideal for very short e2e sessions. Pricier per-hour than Vultr/Hetzner.
OVH Public CloudHourlyYes (we already use OVH for bms-1..4)OK (OpenStack/Terraform)CompetitiveSame vendor as our bare-metal — one invoice, one support relationship, EU. Public Cloud is separate from the Kimsufi dedicated boxes.
AWS EC2 Spot / GCP Spot (preemptible)Per-secondYes (eu-central-1, europe-west)ExcellentCheapest compute available (60–90 % off on-demand)Can be reclaimed with ~2 min notice. Only viable because our queue already re-dispatches failed/oom_killed jobs. Highest integration + IAM complexity.
Fly.io / Northflank / Cloud Run Jobs / FargatePer-second, scale-to-zeroYesExcellentPay only while a job runsContainer-native; no VM to provision/tear down, but our worker currently assumes a full host (SSH, claude-runner user, GH runner).

Key market takeaways

  1. Hetzner’s June 2026 price increase removes its automatic “cheapest” crown for the dedicated-vCPU lines. Re-benchmark before defaulting to it.
  2. Vultr is the strongest “spin up / destroy” general-purpose VM: hourly billing on everything, the cleanest API, and a Warsaw EU region.
  3. DigitalOcean wins for very short sessions (per-second / 60 s minimum) — a single e2e run that lasts 12 minutes costs ~12 minutes, not a rounded-up hour.
  4. OVH Public Cloud is the lowest-friction operationally — same vendor, same invoice, same EU jurisdiction as the rest of our fleet.
  5. Hyperscaler spot is the cost floor but only pays off at volume and adds real IAM/eviction complexity.

3. Three architecture options

Option A — Ephemeral full VM, queue-depth autoscaler (Terraform + existing Ansible)

One-line: A small autoscaler watches dev_r_worker_queue backlog; when it crosses a threshold it terraform applys a new hourly VM (Vultr or OVH Public Cloud, EU), the existing provision-new-vps.yml brings it to worker-ready state, a row is inserted into dev_r_server_capacity, and the existing dispatcher starts feeding it. A reaper destroys the VM and deletes the row once the queue is empty and the box has been idle for N minutes (or a hard TTL elapses).

queue backlog > threshold for T min
        │
        ▼
 burst-provisioner  ──terraform apply──▶  Vultr/OVH VM (EU, hourly)
        │                                      │
        │            ansible-playbook provision-new-vps.yml (common/docker/claude-runner/github-runner/audit-daemon)
        │                                      │
        └──INSERT dev_r_server_capacity──▶ dispatcher fills slots ──▶ workers drain queue
                                               │
 queue empty + idle N min OR TTL ──▶ reaper: DELETE row, deregister GH runner, terraform destroy

Strengths

  • Maximum reuse — provision-new-vps.yml + dev_r_server_capacity + dispatcher already do 80 % of the work. Smallest conceptual leap from today’s design.
  • Workers run in the identical environment to the permanent fleet (full host, claude-runner OAuth, SSH) → no behaviour drift, no per-job surprises.
  • Provider-portable: Terraform module swap (Vultr ↔ OVH ↔ Hetzner) is a few lines.
  • Hourly billing + hard TTL = predictable, capped spend.

Weaknesses

  • Cold-start latency: create VM + run full Ansible playbook + Claude auth = ~5–10 min before the first job runs. Bad for very short bursts; fine for sustained backlog.
  • Stateful provisioning steps (claude-runner OAuth credential copy, GH runner registration) must be idempotent and auto-revoked on teardown or we leak runners/credentials.
  • Need a small new autoscaler service + reaper (where does it run? leader = vps-i1) and Terraform state management.

Best for: sustained queue backlog and the general “add capacity for an hour or two” case. This is the recommended default.

Option B — Container burst / scale-to-zero (Cloud Run Jobs, Fargate, or Northflank)

One-line: Package the worker as a container image; instead of renting a VM, the trigger launches one short-lived container per job (or a small pool) on a scale-to-zero, per-second platform. No host to provision or tear down.

Strengths

  • No VM lifecycle — no Ansible run, no SSH, no GH-runner registration churn. Fastest cold start (seconds, image-pull-bound) and finest billing.
  • Scale to zero literally — you pay nothing between bursts; the platform handles concurrency.
  • Cleanest possible “one e2e session” story: a job is a container that exits when done.

Weaknesses

  • Largest re-architecture. Our worker currently assumes a full host: claude-runner Linux user, SSH-based n8n→claude pattern, GH self-hosted runner, local clones under /home/claude-runner. Containerising it means rebuilding that contract (Claude auth as a secret/env, git auth, no SSH execute pattern).
  • Claude Code OAuth in an ephemeral container is awkward — credentials must be injected as secrets and refreshed; no persistent ~/.claude.
  • Per-vendor lock-in to the container platform’s job model; harder to keep “identical to permanent fleet”.

Best for: a future where workers are stateless and containerised. High payoff, high upfront cost — not the fastest path from where we are today.

Option C — Spot / preemptible VMs with re-dispatch on eviction (cost floor)

One-line: Same shape as Option A, but rent AWS Spot / GCP preemptible instances for 60–90 % off. Accept that the provider can reclaim the box with ~2 min notice; rely on the queue’s existing failed/oom_killed → re-dispatch behaviour so an evicted job simply re-runs on another worker.

Strengths

  • Cheapest compute by a wide margin — the natural choice if burst volume grows enough that VM cost dominates.
  • Reuses the same Terraform + Ansible + dev_r_server_capacity glue as Option A (just a different instance type/lifecycle).
  • Our queue is already eviction-tolerant (re-dispatch loop, terminal failed status) — the hardest prerequisite is partly met.

Weaknesses

  • Eviction mid-job wastes the partial work and the minutes spent on it; jobs must be idempotent and safely re-runnable (true for most issue-implementation workers, risky for anything that pushes commits or mutates external state mid-run).
  • Highest operational complexity: hyperscaler IAM, VPC, spot-capacity availability, and a 2-minute drain handler (graceful worker stop on eviction notice).
  • AWS/GCP are new vendors for us — new accounts, new billing, new security surface, and we must keep compute in EU regions for GDPR.

Best for: a later cost-optimisation phase once burst usage is high and steady. Over-engineered as a first step.

4. Comparison & recommendation

DimensionA — Ephemeral VMB — Container burstC — Spot VM
Reuse of current stack★★★ high★ low★★☆ high (glue), new vendor
Cold-start latency~5–10 minseconds~5–10 min
Cost per burstlow (hourly)lowest for tiny jobslowest overall
Eviction risknonenoneyes (~2 min notice)
Implementation effortsmalllargemedium-large
”One e2e session, then destroy” fitgoodexcellentgood (if idempotent)
New vendor / security surfaceminimal (Vultr/OVH)mediumhigh (AWS/GCP)

Recommendation: Start with Option A on Vultr (EU/Warsaw) or OVH Public Cloud, then benchmark Hetzner’s post-hike CX plans. It is the smallest, safest delta from today’s architecture, reuses provision-new-vps.yml + dev_r_server_capacity + the dispatcher almost wholesale, and gives predictable hourly-capped spend. Revisit Option C (spot) only if burst volume grows enough that VM cost becomes the dominant line item, and treat Option B (containers) as the longer-term target if/when workers are made stateless.

For the specific “rent a box just for one e2e test session” ask: a thin variant of Option A — a manual/CI-triggered burst-up / burst-down pair (provision → run the e2e session → destroy), without the autoscaler — delivers that immediately and is the natural first milestone. DigitalOcean’s per-second billing is the cheapest for very short sessions if minutes matter more than per-hour rate.

5. Cross-cutting concerns (apply to whichever option is chosen)

  • Cost guardrails (mandatory): max concurrent burst servers, max daily burst spend, and a hard TTL on every ephemeral host so a stuck reaper can never run a box for days. Emit a Discord cost alert on each provision/destroy.
  • Teardown must be complete: destroy the VM and (1) deregister the GitHub self-hosted runner, (2) DELETE the dev_r_server_capacity row, (3) mark any in-flight dev_r_worker_queue rows for re-dispatch, (4) revoke/rotate any short-lived creds copied to the box. A leaked runner or orphaned capacity row causes silent failures.
  • Secrets on ephemeral hosts: claude-runner OAuth (.credentials.json), GH runner token, and SOPS age key must be injected at provision time and never baked into an image. Provisioning pulls them from SOPS, not from a snapshot.
  • EU AI Act traceability: ephemeral workers still write dev_r_agent_sessions with a real server_node (e.g. burst-vultr-<id>), so every agent action remains traceable to a physical host per Art. 13.
  • GDPR / region: all burst compute must be provisioned in an EU region.
  • No MongoDB exposure: burst workers do not need MongoDB; do not add their (rotating) IPs to the bms-2/bms-3 ufw 27017 allowlist.
  • Trigger signal: extend queue-dispatcher-loop.py (leader) to compute backlog = count(status=queued) + age of oldest queued_at; fire burst-up when both exceed thresholds for T minutes; fire burst-down when backlog = 0 and burst hosts idle for N minutes.

6. Open decisions for the human (this is why it’s human-action / plans-review)

  1. Pick an option (A / B / C — recommendation is A).
  2. Pick a provider for the first implementation (Vultr vs OVH Public Cloud vs re-benchmarked Hetzner; DigitalOcean if very-short-session billing matters most).
  3. Set the spend ceiling — max concurrent burst hosts and max €/day.
  4. Autoscaler vs manual e2e burst-up/down — do we want the full queue-depth autoscaler first, or just the manual one-session provision/destroy pair?
  5. Approve opening new vendor accounts (Vultr / OVH Public Cloud project / AWS-GCP) — this is a human-gated step (billing + account creation).

Once a human answers these, a follow-up implementation issue can be created against the chosen option with concrete tasks (Terraform module, autoscaler/reaper service, dispatcher backlog signal, teardown hardening).

Graph View