bms4-proxy-operations

bms4-proxy-operations.md

STATUS — Phases 1–2 BUILT (code merged), Phase 3 INERT scaffold, NOT yet deployed (#6168). This document is the ops doc required by ADR-005 §7/§9 and the concrete build spec for issue #6112 (ADR-005 Follow-up 2, Partially implements: #6085). As of #6168 the build lives at infra-src/bms4-proxy/: Purpose 1 (egress) and Purpose 2 (routing) are built with tests, and Purpose 3 (session custody) is built inert / disabled by design (BMS4_PROXY_SESSION_CUSTODY_ENABLED=0). No part of the service is deployed or running yet — the code is shippable via PR with normal review (§7), but installing the systemd unit and starting it is a sys-admin / infra-task action, and Purpose 3 stays disabled until a human enables it after the §5d test plan passes. Per ADR-005 §9, its circuit-breaker must be validated by deliberately breaking the production refresh path, which requires human-supervised, staged rollout. When the service is actually deployed, flip this banner to LIVE and complete the deploy-time items in §9 Compliance.

1. Purpose & scope

The bms4 proxy centralizes three concerns onto bms-4 (54.36.123.110), the primary dispatch host, so that ephemeral/remote workers — chiefly windows-dev’s Docker containers (#6110, a consumer of Purpose 3) — do not each reimplement account routing or run their own interactive claude auth login.

#	Purpose	Novelty	Risk	Build order
1	Centralized egress IP — single static outbound IP for worker→Anthropic traffic, for firewall / allowlist / logging	Low (bms-4 already has a static IP)	Low	First
2	Centralized account/task routing — bms-4 decides which Anthropic account handles which job	None (refactor of existing _account_order headroom ranker)	Low	Second
3	Centralized OAuth session custody — bms-4 holds authenticated sessions and injects them into ephemeral workers	High (new)	High — deliberately accepted repeat of the openclaw-gateway failure class	Last, human-gated

1a. Explicitly OUT OF SCOPE — declined geo-evasion framing

ADR-005 §9 declined outright any proxy behavior whose purpose is masking multi-worker / multi-location usage from Anthropic’s own abuse detection (the original external proposal’s “preventing ‘Geographic Travel’ detection” framing). No version of this build implements that.

This matters because Purpose 1’s mechanism (route worker Anthropic egress through one bms-4 IP) is technically indistinguishable from that declined framing — only the stated intent and the configuration boundary separate them. Therefore the egress proxy is scoped strictly to:

• firewall/allowlist — so a corporate/upstream allowlist need only permit bms-4’s IP;
• logging/observability — one egress point to log outbound volume;

and is explicitly not scoped to make N workers on M residential IPs appear to Anthropic as one origin in order to sidestep anti-abuse detection. A reviewer must confirm the deployed config matches only Purposes 1–3 above (ADR-005 §Verification). If a future need blurs this line, it is a new human decision, not an implementation detail.

2. Architecture

                         bms-4 (54.36.123.110) — primary dispatch host
  ┌──────────────────────────────────────────────────────────────────────┐
  │  (P1) egress-proxy      forward HTTP(S) proxy → api.anthropic.com only │
  │        :<egress_port>   allowlisted upstream; access log               │
  │                                                                        │
  │  (P2) routing endpoint  reuse _account_order() headroom ranker         │
  │        (in-dispatcher)  → "which account for this job/host?"           │
  │                                                                        │
  │  (P3) session-custody   holds per-account authenticated session state; │
  │        broker           injects into ephemeral workers; refresh-       │
  │                         failure monitor + circuit-breaker (§5)         │
  └──────────────────────────────────────────────────────────────────────┘
        ▲ CF Tunnel inbound (ADR-005 §3 — no Tailscale, no new inbound port)
        │
  windows-dev Docker containers (#6110)   ── consumers of P1 + P3
  lap1 native workers                     ── consumers of P2 routing (unchanged path)

Networking: dispatch reaches remote nodes over the existing CF Tunnel inbound pattern (ADR-005 §3), extended to carry docker exec/docker run. No Tailscale, no mesh VPN (#3294 removal stands), no new inbound port on the public-IP+SSH-key servers.

Relationship to the old claude-proxy. infra-src/claude-proxy/ (an n8n→claude -p HTTP wrapper, removed from bms-4 2026-06-24) is a different service and is not the basis for this build. Its README documents the forbidden .credentials.json copy-vector (scp a local .credentials.json onto bms-4 and cp it into /home/claude-runner/.claude/) — that is precisely the openclaw-gateway / refresh-token-race anti-pattern (docs/playbooks/claude-oauth-reauth.md Option C, banned). Purpose 3 here must not reuse that mechanism; see §5.

3. Purpose 1 — centralized egress IP

Component: a forward proxy on bms-4 (e.g. a minimal proxy such as tinyproxy/squid, or a small purpose-built Python forwarder) that:

• listens on a loopback/tunnel-reachable port on bms-4;
• allowlists a single upstream: api.anthropic.com (443). No open forward-proxying — an allowlist keeps the component from becoming a general-purpose open relay;
• writes an access log (timestamp, upstream host, bytes) for the logging purpose;
• is consumed by remote workers via HTTPS_PROXY/HTTP_PROXY env in the container/worker scope.

Config source: the port and any proxy auth secret live in SOPS (secrets/monitoring.env.sops or a dedicated file at build time) — never hardcoded. secret-manager role adds the key; this doc references key names only.

Boundary check (build acceptance): the allowlist MUST be api.anthropic.com only, and the component MUST NOT be configured to rewrite/obscure per-worker identity beyond the single-IP egress that is an inherent property of any forward proxy. Document the config verbatim in the PR that builds it so the §1a boundary is reviewable.

4. Purpose 2 — centralized account/task routing

This is a refactor/centralization of existing logic, not new logic (ADR-005 §9.2). The headroom ranker already lives in scripts/queue-dispatcher-loop.py:

Function	Role
_read_sub_status(server, worker_user) (~L1285)	SSH-reads ~/.claude/sub-status.json for one account
_headroom_from_sub_status(...) (~L1564)	headroom = 100 − max(utilization_5h, utilization_7d)
_account_headroom(...) (~L1591)	cached per (server_label, worker_user), TTL _ACCOUNT_USAGE_TTL_S=300
_rank_non_depleted(...) (~L1613)	stable sort, most-idle first, unknown-headroom accounts last
_account_order(server) (~L1642)	non-depleted (ranked) + depleted-last; reads dev_r_server_capacity.claude_accounts

Centralization design:

• Extract the ranker (_account_order + its helpers) into a shared module (e.g. scripts/lib/account_routing.py) imported by both queue-dispatcher-loop.py and the proxy, so the ranking exists in exactly one place — remote nodes (windows-dev) do not reimplement it.
• Expose a thin “which account for this (host, job)?” query the remote/ephemeral worker path can call, backed by the same dev_r_server_capacity.claude_accounts + sub-status.json inputs and the same depletion/exit 5 semantics (spawn-worker.sh exit 5 → _mark_account_depleted).
• No new selection logic, no growing-account-pool assumption — accounts are the existing, redistributed set (claude-runner, claude-runner-2, radieu); ADR-005 §4 authorizes no new seat purchases.

Acceptance: the extracted module must be import-compatible with the current dispatcher (the dispatcher keeps calling _account_order(server) and gets identical ordering); add unit tests mirroring the existing ranker behavior before wiring the proxy to it.

5. Purpose 3 — OAuth session custody (accepted-risk, human-gated)

5a. The accepted risk (do not treat as an oversight)

Purpose 3 is architecturally the same pattern as the openclaw-gateway incident (ADR-002 §Phase 0): a single held credential/session state, injected into a consumer that cannot write back the rotated refresh token, silently breaking that consumer’s own refresh path. The repo’s established safe default is the opposite — each host/process does its own independent claude auth login, and .credentials.json is never copied between hosts (docs/playbooks/claude-oauth-reauth.md Option C, banned; per-account hourly claude-token-refresh cron 4210).

The owner was shown this exact comparison on #6085 and chose to proceed anyway for the operational benefit of not needing per-container interactive login (ADR-005 §9, §4a). This spec records that as the owner’s deliberate, informed decision — and makes the two ADR-mandated safeguards hard build gates, not nice-to-haves.

5b. Mandatory safeguard 1 — refresh-failure monitoring (shared-session signature)

Reuse the existing per-account refresh-token telemetry rather than inventing a parallel one:

• credential-exporter already emits p4_claude_rt_expires_days{machine,account} from /var/lib/p24/claude-token-state.json (refresh_token_expires_at_ms), with alert rules at < 7 and < 3 days in monitoring/prometheus/rules/self-monitoring.yml.
• Add a signal scoped to the openclaw-gateway failure signature: refresh succeeds server-side but the rotated token is never persisted back into the held/injected session. Concretely, track, per custodied account, the age/staleness of the session token the proxy is actually injecting vs. the freshly-refreshed token on the custody host — a divergence (proxy still handing out an old access token after a refresh occurred) is the exact silent-break indicator. Emit it as a new gauge (e.g. p4_claude_shared_session_stale_seconds{account}) or as an added label dimension on the existing series, and alert when it exceeds a threshold well under the access-token lifetime (~8–12 h).

5c. Mandatory safeguard 2 — circuit-breaker

When the shared-session refresh path is detected broken (5b signal trips, or an injected session returns 401/refresh-failed), the system MUST NOT keep routing jobs through the stale session. One of these must exist, be the wired default, and be tested before Purpose 3 is load-bearing:

• (a) fail over to independent per-container/per-worker claude auth login (mirrors the existing per-host default) — recommended default, degrades to the known-safe posture; or
• (b) halt dispatch through the proxy entirely (fail closed).

State machine:

CLOSED  ──(5b stale signal OR injected-session 401/refresh-fail)──▶ OPEN
OPEN    ── default action = (a) failover to per-worker login  [recommended]
        ──            or   = (b) halt proxy dispatch (fail closed)
OPEN    ──(operator resolves + custody session re-auth verified fresh)──▶ HALF-OPEN
HALF-OPEN ──(one canary job succeeds on injected session)──▶ CLOSED
          ──(canary fails)──▶ OPEN

The implementer chooses (a) vs (b) as the wired default in the build issue; this spec requires that one exists and is exercised. “It worked once in testing” is explicitly insufficient (owner’s own phrasing, ADR-005 §9).

5d. Mandatory test plan

• In a non-production test, deliberately break the shared-session refresh path (e.g. freeze the injected token / revoke the custody session mid-run) and confirm the system either fails over (a) or halts (b) — never silently continues on the stale session.
• Confirm the 5b metric trips and the corresponding alert fires.
• Confirm recovery: CLOSED after a verified re-auth + canary success.

6. Failure signals & alerting summary

Signal	Source	Meaning	Action
p4_claude_rt_expires_days < 7 / < 3	credential-exporter (existing)	refresh token nearing expiry	reauth per claude-oauth-reauth.md
shared-session stale gauge (new, §5b)	proxy custody host	rotated token not persisted back into injected session	circuit-breaker → OPEN
injected-session 401 / refresh-fail	proxy runtime	live break	circuit-breaker → OPEN

7. Build phases — human-supervised rollout

Sequence so risk rises only after each floor is proven:

1. Phase 1 — Purpose 1 (egress). Build the allowlisted forward proxy; verify §1a boundary; no credential handling. Shippable with normal review.
2. Phase 2 — Purpose 2 (routing). Extract the ranker to a shared module + unit tests; wire the “which account?” query; identical dispatcher behavior. Shippable with normal review.
3. Phase 3 — Purpose 3 (session custody) — HUMAN-GATED. Do not carry real dispatch traffic until 5b monitoring + 5c circuit-breaker exist and the 5d test plan has passed under human supervision. Registered enabled=false / test-only until then, mirroring how windows-dev is gated enabled=false until its cap strategy exists (ADR-005 §5).

Sequencing prerequisite (ADR-005 §5a note): do not scale total fleet concurrency through this proxy until the bms-3 RAM-weight dispatch bug (#4717/#4728) is resolved.

8. Operational runbook

Build artifacts (as of #6168, in infra-src/bms4-proxy/):

Concern	File	systemd unit	Default listen
P1 egress proxy	egress_proxy.py	bms4-egress-proxy.service	127.0.0.1:8888 (CONNECT)
P2 routing query	routing_endpoint.py	(co-hosted; run under the same unit or a sibling)	GET /route?server=<label>, /health
P3 session custody	session_custody.py	inert — no unit until enabled (§7)	—

# service status / logs (unit installed at deploy time from bms4-egress-proxy.service)
systemctl status bms4-egress-proxy
journalctl -u bms4-egress-proxy -n 50 -f
 
# access log (Purpose 1 logging) — timestamp \t client \t upstream host:port \t status \t sent/recv
tail -f /var/log/bms4-proxy/egress-access.log   # path = $BMS4_EGRESS_PROXY_ACCESS_LOG
 
# egress proxy reachability (Purpose 1) — expect a proxied 4xx from Anthropic w/o creds, NOT a DNS/refused error
#   (confirms the allowlisted upstream is reachable; do not send real credentials in this smoke test)
curl -sS -o /dev/null -w '%{http_code}\n' -x http://127.0.0.1:8888 https://api.anthropic.com/v1/messages
#   a forbidden upstream MUST be refused by the allowlist (expect 403 from the proxy itself):
curl -sS -o /dev/null -w '%{http_code}\n' -x http://127.0.0.1:8888 https://example.com/   # → 403
 
# routing query (Purpose 2) — which account for a host? (same ranking the dispatcher uses)
curl -sS 'http://127.0.0.1:<routing_port>/route?server=bms-4'   # → {"server":"bms-4","account":"..."}
 
# circuit-breaker / staleness (Purpose 3, only when enabled) — p4_claude_shared_session_stale_seconds
#   is exported for scrape; OPEN state + a stale gauge over BMS4_PROXY_STALE_THRESHOLD_S must alert (§6).

Config env vars are documented in infra-src/bms4-proxy/README.md. Secrets (BMS4_EGRESS_PROXY_AUTH_TOKEN and any P3 custody config) are SOPS-managed and referenced by NAME only — never hardcoded in the unit or this doc.

• Break-glass: if the custody path misbehaves, force circuit-breaker OPEN (fail over to per-worker login, or halt) — never “fix” it by copying .credentials.json between hosts.
• Errors: every failure path sends the Discord embed (P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL) + a bug-labelled GH issue, per CLAUDE.md §Error Notification Standard (egress_proxy.notify_error()).

9. Compliance & deployment checklist

Build-time (done in #6168):

• dev_r_services row registered for bms4-proxy (workbook_url → this doc), with a built-not-deployed status so the compliance registry reflects the code exists but is not live.
• §8 filled with the real unit name (bms4-egress-proxy.service) / default port (127.0.0.1:8888).
• Status banner updated to “Phases 1–2 built, Phase 3 inert, not deployed” (LIVE is deploy-time).

Deploy-time (a sys-admin / infra-task action — NOT part of #6168, not before the service runs):

• Install bms4-egress-proxy.service, deliver BMS4_EGRESS_PROXY_* via secrets-sync.yml, start it.
• Flip this doc’s status banner to LIVE; update the dev_r_services row status to active.
• Secrets (proxy auth, any custody config) in SOPS+age only — secret-manager role adds them.
• Update the topology tables in docs/queue-dispatcher-operations.md and docs/playbooks/queue-dispatch-cross-project.md from “planned” to “live”.
• Confirm reviewer sign-off that the deployed config matches Purposes 1–3 only (not §1a).
• Purpose 3 only: run the §5d test plan under supervision, then set BMS4_PROXY_SESSION_CUSTODY_ENABLED=1. Do not enable before the circuit-breaker is proven.

10. Cross-references

• ADR-005 §3 (networking), §4 (account load-balancing), §9 (this proxy, accepted risk + safeguards), §Verification
• ADR-002 — openclaw-gateway precedent, .credentials.json copy-vector ban, per-account topology
• scripts/queue-dispatcher-loop.py — _account_order ranker (Purpose 2 source)
• monitoring/exporters/credential-exporter/app.py + monitoring/prometheus/rules/self-monitoring.yml — p4_claude_rt_expires_days (Purpose 3 monitoring base)
• docs/playbooks/claude-oauth-reauth.md — banned copy-vector, reauth procedure
• #6085 (design), #6110 (windows-dev Docker workers — Purpose 3 consumers), #6112 (this build spec)