p24-infra Docs

plan-6645-native-windows-video-production-runner

10 min read

Plan #6645 — Native-Windows video-production runner on windows-dev

Status: Design (Code-change-design ready for /review-plan) Owner-issue: #6645 Branch: plan/6645-native-windows-video-runner Scope of this PR: this design document only. No poller, no Task Scheduler entry, no dispatcher code, and no schema is built here — standing any of it up requires physical access to the windows-dev workstation, which cannot be done from bms-4/Telegram (see §9). Follow-up implementation issues are opened only after this design is reviewed and approved. Relates: ADR 005 §5b (windows-dev = WSL2/Docker ephemeral lane, CF-Tunnel inbound), ADR 006 (WSL worker host class), docs/environments/windows-dev.md, 6163 (windows-dev go-live). Cross-repo target: radieu/radekkonarski-personal-brand produce.js; content tracker radek #212 (VIDEO-11…26), radek #209 (LinkedIn path-fix), 211 (Wasabi canonical path + manifest).

1. Goal (one sentence)

Give windows-dev a second, independent execution lane — a native-Windows poller keyed to a new job_type=video-production — that runs produce.js in the native Windows environment it was written for (residential IP, warm HeyGen web-UI session, YouTube OAuth token.json, headless:false

  • native msedge.exe), so that the 16 approved video scenarios can be produced end-to-end autonomously, without disturbing or being disturbed by the existing WSL2/Docker ephemeral-worker lane on the same machine.

2. Why this lane must be native-Windows — the constraints (verified)

produce.js (repo radieu/radekkonarski-personal-brand) is bound to the native-Windows environment. Neither existing execution path can deliver it:

Execution path todayWhy it fails for produce.js
bms-4 worker (datacenter IP, Linux, headless)Cold HeyGen web-UI login from a datacenter IP → high CAPTCHA risk; no warm session, no token.json, no native msedge.exe.
windows-dev WSL2/Docker ephemeral lane (#6110 / ADR-005 §5b)Runs Linux inside a container (docker run --rm, code under the WSL2 Linux FS). Reproduces the same headless/cold problem: no access to the native-Windows warm HeyGen cookies or the YouTube token.json that live on the native C:\ filesystem; headless:false needs a real native browser + desktop session.
HeyGen API path (RADEK_BRAND_HEYGEN_API_KEY, heygen-api-key-rotation.md)Paid API-credits path; explicitly not the route this design chooses (out of scope, §7).

The three artifacts produce.js depends on are human-seeded local files on the native Windows FS, not CI-distributed SOPS secrets:

  1. Warm HeyGen web-UI session — browser cookies/profile on the native FS (Edge profile), created by a one-time human login. This is not RADEK_BRAND_HEYGEN_API_KEY and is not in any SOPS file.
  2. YouTube OAuth token.json — written once after a human OAuth consent flow, refreshed silently by the Google client until the refresh token is revoked/expired.
  3. Residential IP + native msedge.exe — properties of the physical workstation, unreproducible in a container or a datacenter host.

Consequence for the architecture: because the trust anchors are native-FS artifacts and a residential IP, this lane cannot be an SSH/CF-Tunnel inbound-dispatched Docker container like the existing windows-dev lane. It must be a native-Windows outbound-poll process. This is the same outbound-poll shape ADR-005 §3 superseded for the generic WSL2 worker (because that had no public-IP problem the CF Tunnel didn’t already solve) — but here it is required, not a reintroduced mesh: only a native process can reach msedge.exe, the native Edge profile, and token.json.

3. Architecture — two isolated lanes on one machine, keyed by job_type

windows-dev (one physical workstation)
├── Lane A — WSL2/Docker ephemeral worker (EXISTING, #6110/#6163, ADR-005 §5b)
│     dispatch:  CF-Tunnel INBOUND  ← bms-4 queue-dispatcher-loop.py  → docker run --rm
│     claims:    dev-issue / infra-alert / review-* (light weight)  [NOT video-production]
│     env:       Linux in a container, code under WSL2 Linux FS
│
└── Lane B — Native-Windows video-production poller (NEW, this plan)
      dispatch:  OUTBOUND self-poll  → claims dev_r_worker_queue rows directly
      claims:    job_type=video-production ONLY
      env:       native Windows, msedge.exe, warm HeyGen profile, YouTube token.json
      runs:      produce.js  (radieu/radekkonarski-personal-brand, native `node`)

The two lanes never contend: Lane A only ever receives what bms-4 dispatches to it, Lane B only ever claims job_type=video-production, and (per §3.2) the bms-4 dispatcher is taught to skip video-production entirely so it is never SSH/CF-dispatched to Lane A or to bms-4.

3.1 New job_type=video-production

  • Add video-production to the job_type whitelist in scripts/spawn-worker.sh (the case around L106–109) and to the dev_r_worker_queue.job_type CHECK constraint (migration via the per-repo ledger runner — never apply_migration, 5645). Without both, an inserted row is rejected/mis-handled.
  • Do NOT add it to _HARD_PREF_JOB_TYPES for the dispatcher’s benefit; instead handle it via the skip in §3.2. (_HARD_PREF_JOB_TYPES governs soft-preference release for SSH-dispatched jobs; Lane B is not SSH-dispatched, so that set is the wrong lever.)

3.2 The bms-4 dispatcher must never claim video-production

The core routing requirement (“route to windows-dev, not bms-4”) is met not by server-affinity alone but by excluding the job_type from the bms-4 dispatch loop, so Lane B is its sole claimer:

  • Add video-production to a small “self-serviced job types” exclusion consulted by scripts/queue-dispatcher-loop.py’s queue query (the status=eq.queued&… filter around L1335–1346), so bms-4 leaves those rows queued for the native poller to claim. This mirrors the existing pattern where certain job types are handled outside the normal SSH-spawn path.
  • Belt-and-braces: enqueue every video-production row with server_preference='windows-dev' too, so even if the exclusion regresses, the row will not match bms-4’s server_preference filter.

3.3 The native poller (Lane B) — outbound claim loop

A native-Windows process (Task Scheduler entry running a PowerShell/Node poller, or a persistent service) that:

  1. Polls dev_r_worker_queue via PostgREST for status=queued AND job_type=video-production AND (server_preference=windows-dev OR is null), ordered by priority,queued_at, limit 1.
  2. Claims atomically with a guarded PATCH (…&status=eq.queued → set status=claimed, server_node='windows-dev', session_id, claimed_at) — the same race-safe single-row-UPDATE mutex supa_claim_task uses; concurrent claimers serialize on the row lock.
  3. Heartbeats dev_r_worker_queue.heartbeat_at every ~60s while running (the stale-worker reset in queue-dispatcher.sh then reclaims a dead run) and PATCHes dev_r_server_capacity.windows-dev.last_heartbeat on the same cadence via the existing scripts/server-heartbeat.ps1 mechanism.
  4. Checks out radieu/radekkonarski-personal-brand (native git, GitHub App token / PAT) and runs produce.js for the requested video via native node, in the native Windows session (not WSL2).
  5. Marks terminal — status=done|failed, completed_at, result_summary — mirroring supa_complete_queue_item.

The poller reuses the existing Supabase REST claim/complete contract (Blocks A/B/C of worker-issue.md) — it is a new host process, not a new queue table or a new reporting schema (ADR-005 §2 forbids a second jobs table).

3.4 Registration in dev_r_server_capacity

windows-dev already has a row (added for Lane A, #6163). Lane B needs no new capacity semantics in the dispatcher because Lane B is self-poll, not SSH-dispatched — the RAM bin-packer never books a slot for it. Concretely:

  • Keep the existing windows-dev row’s Lane-A fields as-is.
  • The native poller updates last_heartbeat so the row shows live, and (design decision for the follow-up build issue) records its own concurrency via a metadata key rather than current_workers, so it does not confuse Lane-A RAM accounting.
  • Do not rely on enabled/max_weight_prime to route Lane B — those gate the SSH-dispatch loop only, which never touches video-production after §3.2.

4. Warm-session / token requirements (human-anchored, documented, not eliminable)

These are prerequisites the poller cannot self-provision; the design records them explicitly and the follow-up build issue must ship a renewal runbook:

ArtifactSeeded byLives atRenewal when it expires
HeyGen warm web-UI sessionone-time human login in native Edge on windows-devnative Edge profile (native FS)cyclic human re-login when the session drops — cannot be automated (this is the whole reason the API path was declined).
YouTube OAuth token.jsonone-time human OAuth consentnative FS path used by produce.js (name it in the build issue)silent refresh by the Google client until the refresh token is revoked/expired → then a human re-consent.
Residential IP + msedge.exeproperty of the workstation—n/a (inherent to the host).
  • Monitoring hook: the poller should emit a clear failed result with a distinguishable failure_reason (e.g. heygen_session_expired, youtube_token_expired) so a human is paged to re-auth, rather than silently retrying into CAPTCHA loops.
  • RADEK_BRAND_HEYGEN_API_KEY (SOPS: radekkonarski-personal-brand/secrets/radekkonarski-brand.env.sops, the repo’s own SOPS file — p24-infra spawn-worker.sh intentionally does not map this repo, #4573) is not used by this lane; named here only to disambiguate it from the warm-session cookie.

5. Cross-repo dispatch (how a video-production row gets created)

produce.js is in radieu/radekkonarski-personal-brand, so the queue row carries repo='radieu/radekkonarski-personal-brand' (the existing dev_r_worker_queue.repo column → spawn-worker.sh arg 4 mechanism). For Lane B the poller does the checkout itself, so the only new requirement is that the producers of the row learn the new job_type:

  • Telegram / bms-4 manual dispatch: the insert path (meta-dispatcher / Telegram orchestrator / /new-issue-style enqueue) must accept job_type=video-production and set repo='radieu/radekkonarski-personal-brand', server_preference='windows-dev', weight='light'. This is a routing/whitelist addition on the producer side, symmetric to §3.1’s consumer-side whitelist.
  • No SOPS mapping change in p24-infra spawn-worker.sh is needed for this lane (Lane B does not run spawn-worker.sh; it runs produce.js natively with native-FS artifacts).

6. Definition of Done — per video

A video-production job is done only when all of:

  1. MP4 produced by produce.js for the requested scenario.
  2. Uploaded to Wasabi at the canonical path videos/video-NN.mp4 (radek 211).
  3. YouTube: uploaded and set public via the YouTube OAuth token.
  4. LinkedIn auto-post left to the existing cron + gate (radek #209 path-fix, #211 manifest) — the poller writes the Wasabi/YouTube result into the manifest that cron consumes; it does not re-implement LinkedIn posting.

The result_summary records the Wasabi key + YouTube URL so the callback/manifest is traceable.

7. Out of scope (deliberate)

  • Producing VIDEO-11…26 content — tracked in radek #212. This plan builds the mechanism, not the content.
  • HeyGen API path (paid API-credits) and bms-4 headless path (fragile, CAPTCHA-prone) — this design deliberately chooses the third route: the original windows-dev native environment, run on-demand instead of by hand.
  • Any change to Lane A (the WSL2/Docker ephemeral worker) — the two lanes are independent.

8. Regression risks

  • job_type CHECK-constraint drift (#4293/#4605 class): adding video-production must land in the per-repo migration ledger and the spawn-worker.sh whitelist together, or rows are silently rejected/mishandled. Mitigate by shipping both in the same follow-up PR and testing an end-to-end insert→claim→done on a dummy row before enabling producers.
  • bms-4 accidentally claiming Lane-B rows if the §3.2 skip regresses → job runs headless on the wrong host and CAPTCHAs. Mitigated by the belt-and-braces server_preference='windows-dev'.
  • Silent CAPTCHA/expiry loops if session/token renewal is not surfaced → covered by the distinguishable failure_reason in §4.
  • Lane collision on capacity accounting if Lane B writes current_workers → covered by using a metadata key instead (§3.4).

9. Hard floor — what a human must do, and why this is plan-only

  • Standing up the poller, the Task Scheduler entry, and confirming the warm HeyGen session + YouTube token.json requires access to the windows-dev workstation itself — it cannot be done from Telegram or bms-4. This plan produces a complete, execution-ready design for that dev-access window.
  • The one-time HeyGen login and YouTube OAuth consent are human actions, renewed cyclically; they cannot be eliminated (this is precisely why the automated API path was declined, §7).

10. Follow-up issues to open (only after this design is approved — not this PR)

  1. Build spec — native-Windows video-production poller (sys-admin/dev-coder, executed in a windows-dev dev-access window): implement the outbound claim loop (§3.3), the Task Scheduler entry, the produce.js invocation, the Definition-of-Done manifest write (§6), and the session/token-expiry failure_reason surfacing (§4). Create the runbook for cyclic HeyGen/YouTube re-auth.
  2. Routing changes (dev-coder): add video-production to spawn-worker.sh’s whitelist, the dev_r_worker_queue.job_type CHECK (per-repo ledger migration), the bms-4 dispatcher’s self-serviced-skip list (§3.2), and the producer-side enqueue paths (§5).
  3. Ops doc — docs/environments/windows-dev.md gains a “Lane B — native video-production” section; dev_r_services row + compliance entry for the new runner.

Auto-generated by worker agent for #6645. Plan-only; no implementation in this PR.

Graph View