p24-infra Docs

rc-smoke-tests-credential-and-config-debugging

5 min read

RC smoke tests — credential vs config debugging (et-operational-platform)

Trigger: rc-smoke-tests.yml fails, and the failure looks like a broken credential (Invalid login credentials, 401, not set) but the credential was just verified as correct.

Context: this playbook documents four distinct, stacked issues found and fixed in one 2026-08-09 session (et-op issues #1609, #1621, #1664). Read the whole thing before assuming you’ve found “the” bug — there was never just one.

1. PowerShell’s Invoke-RestMethod/Invoke-WebRequest can silently mangle auth headers

Symptom: a Supabase (or any Bearer-token) API call returns 401 from PowerShell, but the exact same key works from curl or the app’s own SDK.

Confirm it: run the identical request two ways and compare status codes:

# PowerShell — may return 401 even with a correct key
Invoke-WebRequest -Uri $url -Headers @{ apikey = $key; Authorization = "Bearer $key" }
# curl — authoritative
curl -s -o body.json -w "%{http_code}" "$url" -H "apikey: $key" -H "Authorization: Bearer $key"

If curl succeeds and PowerShell fails with the same key, PowerShell is the bug — not the credential. Do not rotate, re-mint, or “fix” the key. Verify with curl (or the app’s actual client library, e.g. @supabase/supabase-js, to also rule out library-specific quirks) before concluding anything is broken server-side.

Prevention: when a Windows-session investigation needs to hit an HTTP API with custom auth headers, prefer curl.exe (via the Bash tool) or Node/the target SDK over Invoke-RestMethod/Invoke-WebRequest for anything beyond a simple unauthenticated GET.

2. A test account needs BOTH a working password AND a user_roles row

Symptom: global-setup.ts signs in successfully (no “Invalid login credentials”), but the Playwright test still times out waiting for page content — the app silently redirects to /dashboard because useAuth({ requiredRoles: [...] }) didn’t find the role.

Confirm it:

select * from user_roles where user_id = '<test-account-uuid>';

An empty result means the account authenticates but has zero authorization — useAuth (src/hooks/useAuth.ts) checks requiredRoles against getUserRoles() (a plain user_roles query) after sign-in succeeds, and redirects away on mismatch with no error surfaced to Playwright other than a timeout on whatever element was supposed to render next.

Fix: insert a user_roles row modeled on an existing account with the same role_type (check office_id, resource_type, assigned_resources — these vary by role; e.g. per-queue technician accounts need resource_type='service_queues', assigned_resources=['W'|'Z'|'M']).

Prevention: whenever provisioning or resetting a Supabase test account (via Admin API, dashboard, or Playwright), always check user_roles for that user_id in the same pass — a password reset alone is not sufficient to make a role-gated page testable.

3. Every Playwright project with @smoke-tagged specs needs the Vercel bypass header

Symptom: global-setup.ts reports zero auth failures, but a specific role’s smoke test(s) still time out waiting for page content that should render immediately after auth.

Root cause (et-op #1664): playwright.config.ts computes vercelBypassHeaders once and wires it into extraHTTPHeaders per-project. When the Vercel Deployment Protection bypass header was added (et-op PR #1610), it was wired into the 7 dedicated smoke-* projects but missed any other project whose testMatch also picks up an @smoke-tagged spec file living outside tests/e2e/smoke/. Without the header, requests against a protected preview URL hit Vercel’s SSO wall instead of the real app — the page never renders, so every locator times out.

Confirm it:

# List every file with an @smoke tag
grep -rl "@smoke" tests/e2e/
 
# List every project name + whether it has extraHTTPHeaders wired
grep -n "name:\|extraHTTPHeaders" playwright.config.ts

Any @smoke file not under tests/e2e/smoke/ is matched by a role-specific project (via testMatch), not one of the smoke-* projects — check that role’s project has extraHTTPHeaders: vercelBypassHeaders too.

Fix: add extraHTTPHeaders: vercelBypassHeaders to the affected project’s use block.

Prevention: when adding a new @smoke-tagged spec to a role directory (not tests/e2e/smoke/), or adding a brand-new role project, explicitly check it has the bypass header wired — this is not automatic and CI will not flag a missing header as a config error, only as an opaque content-not-found timeout.

4. net::ERR_NETWORK_CHANGED on the bms4 self-hosted runner is transient

Symptom: global-setup.ts’s page.goto(BASE_URL) fails with net::ERR_NETWORK_CHANGED, consistently around the same elapsed time into the run.

Confirm it: check Supabase auth logs (mcp__supabase__get_logs service=auth) for the same time window — if there are no invalid_credentials/429 entries at all (not even failed ones), the run never got far enough to attempt auth; it’s a pure network-layer failure on the runner, not an app or credential issue.

Fix: just retry (gh workflow run rc-smoke-tests.yml --ref rc or re-push). Observed twice in a row on 2026-08-09 then resolved on the third attempt with no configuration change.

Escalation: if this recurs frequently (more than one retry needed per run, or happens on unrelated workflows on the same runner), open an infra-task-request issue in radieu/p24-infra — bms4 runner network stability is out of et-operational-platform’s role boundary.

General debugging order for “smoke tests fail with an auth-shaped error”

  1. Check global-setup.ts’s own log output first — does it report per-account auth failures, or does it succeed and the failure happens later in the actual test? These are different bugs (credential/role vs config/bypass-header) with different fixes.
  2. If global-setup reports a failure: verify the credential with curl/the real SDK before touching GH Secrets or SOPS — see §1.
  3. If global-setup succeeds but a specific role’s test times out: check user_roles for that account (§2), then check the Playwright project’s extraHTTPHeaders wiring (§3).
  4. If the failure is a raw network error before any app-level assertion: it’s very likely runner flakiness (§4) — retry before debugging further.

Related: et-op issues #1609, #1621, #1664. et-op PR #1610 (original bypass-header wiring), 1625 (per-role secret wiring), #1672 (plac project bypass-header fix).

Graph View