Playbook: Supabase Project API Key Rotation

Service: Supabase project mwkqmgadqnkkihjdeqsi Keys covered: SUPABASE_SERVICE_ROLE_KEY, SUPABASE_ANON_KEY (and per-component variants) Rotation frequency: 90 days (or immediately on suspected exposure) Management token required: SUPABASE_ACCESS_TOKEN in secrets/administration.env.sops

Where the management token actually lives (#4585). It is in secrets/administration.env.sops, NOT monitoring.env.sops — it was deliberately moved off the monitoring surface on 2026-07-05 (560fb17e, #2075, least-privilege isolation) with the last in-stack consumer removed by 9abcbaa (#2620), and must not be put back (PR #4700 was rejected for exactly that: administration.env.sops is excluded from secrets-sync.yml, whereas monitoring.env.sops is deployed to vps-i1 and read wholesale via env_file: .env by alertmanager, caddy, loki and mezmo-exporter). administration.env.sops is developer-workstation only — CI never deploys it, so the PowerShell snippets below are developer-machine procedures.

⚠️ bms-4 worker token source (#5319). A standard bms-4 secret-manager worker (age key ~/.age/p24-infra-keys.txt) cannot decrypt role-secret-manager.env.sops or administration.env.sops — so it cannot read ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN. The only worker-reachable copy of an equivalent Supabase management token is SUPABASE_MANAGEMENT_TOKEN in secrets/brandpilot.env.sops (which the worker can decrypt). On bms-4, read that key — see § bms-4 worker — token source for the exact code path. This matches the matrix’s own #4400 Master-key deliverability re-check (secret-rotation-access-matrix.md).

⚠️ The write path is currently broken — see #4626. The token authenticates and lists keys fine, but keys it mints return 401 Invalid API key against the project data API. Do not treat a successful GET as proof that rotation will work; the liveness gate in Step 2.5 is mandatory.

⚠️ reveal=true tested and disproven — root cause still unknown (2026-08-06). Supabase support (Cemal Kilic, ticket SU-439714) proposed that adding ?reveal=true to the create/read-back calls would fix the 401. A secret-manager agent tested this: minted a throwaway key with ?reveal=true, re-fetched it with ?reveal=true, waited for propagation, and it still returned 401 against dev_r_services (test key deleted afterward). The create/read-back calls in this playbook now include reveal=true per Supabase support’s suggestion — it is harmless but does not by itself resolve #4626. Do not skip the Step 2.5 liveness gate on the theory that reveal=true fixes anything.

Related:

supabase-secrets-management-gaps.md — full gap analysis and rationale

n8n/n8n-supabase-credential-rotation.md — n8n credential sync (incident recovery)

supabase-access-token-rotation.md — rotating the management token itself (sbp_…)

static-api-key-incident-rotation.md — incident response wrapper (compromised key)

Emergency: “Legacy API keys disabled” — Supabase forced migration

Symptom: Login and all Supabase calls return {"message":"Legacy API keys are disabled","hint":"Your legacy API keys (anon, service_role) were disabled on ..."}.

Root cause: Supabase deprecated legacy JWT-format keys (eyJ...). New format: sb_publishable_... (anon) and sb_secret_... (service_role). When Supabase enforces the cutover, all running deployments with old keys break immediately.

Key format difference:

Format	Example prefix	Length	Used for
Legacy (disabled)	`eyJ`	~208-690 chars	Old anon + service_role
New anon	`sb_publishable_`	46 chars	`NEXT_PUBLIC_SUPABASE_ANON_KEY`
New service_role	`sb_secret_`	~100+ chars	`SUPABASE_SERVICE_ROLE_KEY`

Emergency fix (30 min):

Get new-format keys from Management API — the supabase_anon_key entry (type: publishable) is pre-provisioned by Supabase; do NOT create a new one:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Developer workstation only — administration.env.sops is excluded from secrets-sync.yml (#4585).
# On a bms-4 worker use: brandpilot.env.sops / SUPABASE_MANAGEMENT_TOKEN (worker-reachable — #5319; see § bms-4 worker — token source)
$env:SB_MGMT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops |
    Select-String "^SUPABASE_ACCESS_TOKEN=").ToString().Split("=",2)[1]
$keys = Invoke-RestMethod "https://api.supabase.com/v1/projects/mwkqmgadqnkkihjdeqsi/api-keys?reveal=true" `
    -Headers @{ Authorization = "Bearer $env:SB_MGMT" }
$env:NEW_ANON_KEY = ($keys | Where-Object { $_.name -eq "supabase_anon_key" }).api_key
# For service_role: use the named key already in SOPS (sb_secret_ prefix) — verify with:
# ($keys | Where-Object { $_.type -eq "service_role" }) | Select name, prefix, type

Update et-operational-platform.env.sops — see § Step 6 (et-op SOPS block).
Update Vercel env vars AND trigger rebuild — see § Step 6 (both are required, order matters).
Verify rebuild is complete before reporting fixed — check deployment state:

(Invoke-RestMethod "https://api.vercel.com/v6/deployments?projectId=prj_ziLl911FOYLAeukQujL4NjxR4eWy&teamId=team_RuXLTWx5DXK6n0HUKoOUTGY3&limit=5" `
    -Headers @{ Authorization = "Bearer $env:VT" }).deployments |
    Select-Object @{n="branch";e={$_.meta.githubCommitRef}}, state, url

Hard-refresh browser (Ctrl+Shift+R) — old JS bundle may be cached locally.

Do NOT: Only update the env var without triggering a Vercel rebuild. NEXT_PUBLIC_* vars are baked into the JS bundle at build time — the old bundle keeps using the old key regardless of env var changes.

Key hierarchy

SUPABASE_ACCESS_TOKEN  (sbp_... — personal access token)
  └── Supabase Management API  https://api.supabase.com/v1/projects/{ref}/api-keys
        ├── creates / deletes / lists project API keys
        └── project API keys:
              ├── sb_secret_...   (service_role — bypasses RLS, full data access)
              └── sb_publishable_... (anon — subject to RLS policies)

SUPABASE_ACCESS_TOKEN is the root credential for all key operations. It lives in secrets/administration.env.sops (developer-only — excluded from secrets-sync.yml) and GH Secret SUPABASE_ACCESS_TOKEN. It is not in monitoring.env.sops and must not be added there (#2620, #4585, PR #4700). Its own rotation procedure: supabase-access-token-rotation.md.

bms-4 worker — token source

The PowerShell snippets in this playbook are developer-workstation procedures that read SUPABASE_ACCESS_TOKEN from secrets/administration.env.sops (or the role-scoped ROLE_SECRET_MANAGER_SUPABASE_ACCESS_TOKEN in secrets/role-secret-manager.env.sops). A standard bms-4 worker cannot decrypt either file — its age key (~/.age/p24-infra-keys.txt) is not a recipient of role-secret-manager.env.sops or administration.env.sops (confirmed by the #4400 Master-key deliverability re-check in secret-rotation-access-matrix.md).

On bms-4, read the worker-reachable copy — SUPABASE_MANAGEMENT_TOKEN in secrets/brandpilot.env.sops — and substitute it for $env:SB_MGMT in every step below (it is the same class of Supabase management token):

# bms-4 worker — read the management token silently, never echo it
export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
SB_MGMT=$(sops -d --input-type dotenv --output-type dotenv secrets/brandpilot.env.sops \
  | grep '^SUPABASE_MANAGEMENT_TOKEN=' | cut -d= -f2-)
REF="mwkqmgadqnkkihjdeqsi"
# Use "$SB_MGMT" only as a request header argument, e.g.:
#   curl -sf "https://api.supabase.com/v1/projects/$REF/api-keys" -H "Authorization: Bearer $SB_MGMT"
# When done: unset SB_MGMT

If secrets/brandpilot.env.sops is not decryptable on the worker either, the rotation is blocked on token reachability — comment on the issue and hand off to a role-secret-manager session (or the developer) that holds role-secret-manager decrypt access. Do not attempt to read role-secret-manager.env.sops / administration.env.sops from a standard worker.

Consumer inventory

All consumers of Supabase project API keys. Update this table after every key change.

Consumer	Key type	SOPS var	Where injected
bms-4 n8n (6 credentials)	service_role	`SUPABASE_SERVICE_ROLE_KEY`	n8n encrypted credential vault via `sync-n8n-supabase-creds.py`
et-operational-platform (Next.js)	service_role	`SUPABASE_SERVICE_KEY`	Vercel env var `SUPABASE_SERVICE_KEY`
et-operational-platform (frontend)	anon	`SUPABASE_ANON_KEY`	Vercel env var `NEXT_PUBLIC_SUPABASE_ANON_KEY`
brandpilot (Next.js)	service_role	`SUPABASE_SERVICE_KEY`	Vercel env var `SUPABASE_SERVICE_KEY` in brandpilot project
vps-i1 monitoring scripts	service_role	`SUPABASE_SERVICE_KEY`	`/opt/p24-infra/monitoring/.env` via secrets-sync.yml
CI/CD GH Actions (16 workflows — see below)	service_role	`SUPABASE_SERVICE_KEY`	GH Secret `SUPABASE_SERVICE_KEY` in radieu/p24-infra
p24-meta-dispatcher CF Worker	service_role	`SUPABASE_SERVICE_KEY`	Cloudflare Worker secret bound by `deploy-meta-dispatcher.yml` — sourced from `secrets/monitoring.env.sops` (not the GH Secret). See note below.
Grafana readonly datasource	postgres role	`SUPABASE_DB_PASSWORD`	Not a project API key — separate procedure

Target state (Gap 1 migration — planned): per-component named keys. See § Per-component key isolation migration below.

CI/CD GH Actions consumers (radieu/p24-infra)

All workflows that read the SUPABASE_SERVICE_KEY GH Secret. Re-generate this list with grep -rl 'SUPABASE_SERVICE_KEY' .github/workflows/ | sort and update it after every key change. All 16 must be re-verified after deleting any key they depend on (issue #2005, step 5).

Workflow	Notes
`atrax-data-freshness.yml`	GPS/atrax freshness check
`backup-n8n-cloud.yml`	n8n cloud backup
`credential-rotation.yml`	90-day rotation scheduler (`dev_r_services.next_due`)
`db-maintenance.yml`	Supabase DB maintenance
`dispatch-to-queue.yml`	worker-queue dispatch
`grafana-backup.yml`	Grafana config backup
`health-check.yml`	infra health check — run manually post-deletion as the smoke test
`infra-task-request.yml`	infra-task-request executor
`n8n-maintenance.yml`	n8n maintenance jobs
`nc-alert-instant-dispatch.yml`	NC-alert instant dispatch
`nightly-devops-triage.yml`	nightly triage (also uses `SUPABASE_SERVICE_ROLE_KEY`)
`pwd-rotation-check.yml`	password rotation watchdog
`resource-incident-triage.yml`	resource incident triage (also uses `SUPABASE_SERVICE_ROLE_KEY`)
`secrets-sync.yml`	secret deployment + n8n credential sync (also uses `SUPABASE_SERVICE_ROLE_KEY`)
`waha-shadow-observe.yml`	WAHA shadow observation
`waha-stats-report.yml`	WAHA stats report

CI/CD pre-deletion gate (human-gated — do NOT automate): before deleting the old shared service_role key, the operator must confirm the SUPABASE_SERVICE_KEY GH Secret holds the new per-component key, not the old shared one. GitHub Actions secret values are write-only and cannot be read back, so the prefix cannot be verified by reading the secret. Use one of:

Overwrite from the source of truth (preferred): re-set the secret from the SOPS value so its contents are known with certainty — gh secret set SUPABASE_SERVICE_KEY --body "$KEY" --repo radieu/p24-infra where $KEY is read silently from secrets/monitoring.env.sops (never echoed). This makes step 1 a no-op.

Indirect liveness check: after deletion, manually run health-check.yml and confirm all 16 workflows above still pass (issue #2005, step 5).

Deleting the old key is irreversible (Management API DELETE …/api-keys/{id}). It is a human operator action — see § Step 8. Old key stays live until the operator runs the deletion, so there is no production impact until then.

Non-GH-Secret consumer: p24-meta-dispatcher CF Worker

deploy-meta-dispatcher.yml is not in the table above because it does not read the SUPABASE_SERVICE_KEY GH Secret. It is a separate deletion-impact path that the 16-workflow inventory misses:

It decrypts secrets/monitoring.env.sops via the CI age key (AGE_KEY_GHA) and binds the SUPABASE_SERVICE_KEY value as a Cloudflare Worker secret (wrangler secret put) on the p24-meta-dispatcher Worker. The value never echoes (piped to wrangler via stdin).
Source-of-truth status (verified 2026-06-29): the SUPABASE_SERVICE_KEY entry in secrets/monitoring.env.sops already holds the new p24_monitoring key (prefix sb_secret_Iwpck), not the old shared key. So the Worker’s source is safe — the next run of deploy-meta-dispatcher.yml binds the new key. (Verified by prefix classification only; no key value was printed.)
Deletion-impact caveat (human-gated): the currently-deployed CF Worker secret may still hold the old shared key if it was last bound before the 2026-06-28 rotation. After the operator deletes the old key (Step 8), they should re-run deploy-meta-dispatcher.yml (gh workflow run deploy-meta-dispatcher.yml --repo radieu/p24-infra) so the live Worker is re-bound with the new key, then confirm the Worker no longer 500s.

Management API — key operations

All operations use SUPABASE_ACCESS_TOKEN (sbp_… prefix). Never print the token value.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Developer workstation only — administration.env.sops is excluded from secrets-sync.yml (#4585).
# On a bms-4 worker use: brandpilot.env.sops / SUPABASE_MANAGEMENT_TOKEN (worker-reachable — #5319; see § bms-4 worker — token source)
$env:SB_MGMT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops |
    Select-String "^SUPABASE_ACCESS_TOKEN=").ToString().Split("=",2)[1]
$REF = "mwkqmgadqnkkihjdeqsi"

List existing keys (names + IDs, values masked)

$keys = Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys" `
    -Headers @{ Authorization = "Bearer $env:SB_MGMT" }
$keys | Select-Object name, id, prefix, type | Format-Table

Create a new service_role key (full value returned once)

# reveal=true is in use per Supabase support's suggestion (SU-439714) — does not by itself resolve #4626.
$body = @{ name = "p24_monitoring_$(Get-Date -Format 'yyyyMMdd')"; type = "secret" } | ConvertTo-Json
$new = Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys?reveal=true" `
    -Method POST -Headers @{ Authorization = "Bearer $env:SB_MGMT"; "Content-Type" = "application/json" } `
    -Body $body
# Full key value is in $new.api_key — capture immediately into env var, never print
$env:NEW_SERVICE_KEY = $new.api_key
Write-Host "Key ID: $($new.id)   Name: $($new.name)   Prefix: $($new.prefix)"

Create a new anon (publishable) key

# reveal=true is in use per Supabase support's suggestion (SU-439714) — does not by itself resolve #4626.
$body = @{ name = "p24_anon_$(Get-Date -Format 'yyyyMMdd')"; type = "anon" } | ConvertTo-Json
$new = Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys?reveal=true" `
    -Method POST -Headers @{ Authorization = "Bearer $env:SB_MGMT"; "Content-Type" = "application/json" } `
    -Body $body
$env:NEW_ANON_KEY = $new.api_key
Write-Host "Key ID: $($new.id)   Name: $($new.name)   Prefix: $($new.prefix)"

Delete an old key by ID (irreversible)

# Only run after ALL consumers verified with new key
Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys/$OLD_KEY_ID" `
    -Method DELETE `
    -Headers @{ Authorization = "Bearer $env:SB_MGMT" } `
    -Body (@{ reason = "rotating_key" } | ConvertTo-Json) `
    -ContentType "application/json"
Write-Host "Key $OLD_KEY_ID deleted"

Clear management token when done

$env:SB_MGMT = ""; $env:SOPS_AGE_KEY_FILE = ""

Scheduled rotation — step by step

Before starting: read static-api-key-incident-rotation.md §Step 0 — open a pending log entry in docs/secrets-rotation-log.md first.

Step 1 — Note the current key IDs (to delete later)

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
# Developer workstation only — administration.env.sops is excluded from secrets-sync.yml (#4585).
# On a bms-4 worker use: brandpilot.env.sops / SUPABASE_MANAGEMENT_TOKEN (worker-reachable — #5319; see § bms-4 worker — token source)
$env:SB_MGMT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops |
    Select-String "^SUPABASE_ACCESS_TOKEN=").ToString().Split("=",2)[1]
$REF = "mwkqmgadqnkkihjdeqsi"
$keys = Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys" `
    -Headers @{ Authorization = "Bearer $env:SB_MGMT" }
# Note the IDs of existing service_role + anon keys — needed for Step 7
$keys | Select-Object name, id, prefix, type | Format-Table

Step 2 — Create new keys via API

# reveal=true is in use per Supabase support's suggestion (SU-439714) — does not by itself resolve #4626.
# Service role key
$body = @{ name = "p24_service_$(Get-Date -Format 'yyyyMMdd')"; type = "secret" } | ConvertTo-Json
$newSvc = Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys?reveal=true" `
    -Method POST -Headers @{ Authorization = "Bearer $env:SB_MGMT"; "Content-Type" = "application/json" } `
    -Body $body
$env:NEW_SERVICE_KEY = $newSvc.api_key   # capture immediately — never print
$env:NEW_SERVICE_KEY_ID = $newSvc.id
Write-Host "New service key created. ID: $env:NEW_SERVICE_KEY_ID"
 
# Anon key
$body = @{ name = "p24_anon_$(Get-Date -Format 'yyyyMMdd')"; type = "anon" } | ConvertTo-Json
$newAnon = Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys?reveal=true" `
    -Method POST -Headers @{ Authorization = "Bearer $env:SB_MGMT"; "Content-Type" = "application/json" } `
    -Body $body
$env:NEW_ANON_KEY = $newAnon.api_key
$env:NEW_ANON_KEY_ID = $newAnon.id
Write-Host "New anon key created. ID: $env:NEW_ANON_KEY_ID"

Step 2.5 — Newly-minted key liveness gate (mandatory — do NOT skip, added #4626)

Why this step exists (#4626 incident, 2026-07-30): the Management API can return HTTP 200 with a plausible, correctly-formatted sb_secret_... key that does not authenticate against the project’s own data API (GET .../rest/v1/... → 401 {"message":"Invalid API key"}) — a control-plane/data-plane desync, or a management token whose owning org lacks write-linkage to this project. Before this gate, the only liveness check was the pre-deletion gate ([§ Step 7]), which runs after Steps 3–6 have already pushed the key to SOPS, GH Secrets, Vercel, n8n and the CF Worker. In #4626 a broken key was distributed to every consumer and lived ~40 min before the Step 7 gate caught it, forcing a full rollback. Validate at mint time — reject a dead key in seconds, before it touches anything.

Rule: a freshly-minted key is UNTRUSTED until it returns 200 from the project data API. Never proceed to Step 3 (SOPS write) or any distribution until the check below passes.

# Uses $env:NEW_SERVICE_KEY / $env:NEW_ANON_KEY captured in Step 2. Never print the values.
$REST = "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1"
 
# 1. service_role key — MUST return 200 against real data (bypasses RLS)
$svcOK = $false
try {
    $r = Invoke-WebRequest "$REST/dev_r_services?limit=1" -UseBasicParsing `
        -Headers @{ apikey = $env:NEW_SERVICE_KEY; Authorization = "Bearer $env:NEW_SERVICE_KEY" }
    $svcOK = ($r.StatusCode -eq 200)
    Write-Host "service_role liveness: HTTP $($r.StatusCode)"
} catch {
    Write-Host "service_role liveness FAILED: $($_.Exception.Response.StatusCode.value__) — key is DEAD"
}
 
# 2. anon (publishable) key — MUST NOT return 401 Invalid API key (200 on the REST root spec)
$anonOK = $false
try {
    $r = Invoke-WebRequest "$REST/" -UseBasicParsing `
        -Headers @{ apikey = $env:NEW_ANON_KEY; Authorization = "Bearer $env:NEW_ANON_KEY" }
    $anonOK = ($r.StatusCode -eq 200)
    Write-Host "anon liveness: HTTP $($r.StatusCode)"
} catch {
    Write-Host "anon liveness FAILED: $($_.Exception.Response.StatusCode.value__) — key is DEAD"
}
 
if (-not ($svcOK -and $anonOK)) {
    Write-Host "ABORT — a minted key does not authenticate. Deleting the dead key(s); NOT distributing."
    # Roll back the useless keys immediately (safe: they were never distributed).
    foreach ($id in @($env:NEW_SERVICE_KEY_ID, $env:NEW_ANON_KEY_ID)) {
        if ($id) {
            Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys/$id" `
                -Method DELETE -Headers @{ Authorization = "Bearer $env:SB_MGMT" } `
                -Body (@{ reason = "minted_key_failed_liveness" } | ConvertTo-Json) -ContentType "application/json"
            Write-Host "Deleted dead key $id"
        }
    }
    $env:NEW_SERVICE_KEY = ""; $env:NEW_ANON_KEY = ""
    throw "Minted key failed liveness gate — investigate SUPABASE_ACCESS_TOKEN org/project linkage (#4626) before retrying. Do NOT distribute."
}
Write-Host "Liveness gate PASSED — both keys authenticate. Safe to distribute (Step 3)."

Linux / bms-4 equivalent (when rotating from a server; read the key silently into a var, never echo it):

code=$(curl -s -o /dev/null -w '%{http_code}' \
  "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/dev_r_services?limit=1" \
  -H "apikey: $NEW_SERVICE_KEY" -H "Authorization: Bearer $NEW_SERVICE_KEY")
[ "$code" = "200" ] || { echo "DEAD key (HTTP $code) — abort, do not distribute"; exit 1; }

Step 3 — Update SOPS (source of truth)

Gate check first: do not enter this step until § Step 2.5 passed. Distributing an unverified key is exactly the #4626 failure mode — it costs a full rollback across SOPS + GH Secrets + Vercel + n8n + the CF Worker.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$plain = sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops
$temp = "$PWD\secrets\monitoring-edit.env.sops"
$updated = $plain `
    -replace "(?m)^SUPABASE_SERVICE_KEY=.*$", "SUPABASE_SERVICE_KEY=$env:NEW_SERVICE_KEY" `
    -replace "(?m)^SUPABASE_SERVICE_ROLE_KEY=.*$", "SUPABASE_SERVICE_ROLE_KEY=$env:NEW_SERVICE_KEY" `
    -replace "(?m)^SUPABASE_ANON_KEY=.*$", "SUPABASE_ANON_KEY=$env:NEW_ANON_KEY"
[System.IO.File]::WriteAllText($temp, ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv $temp
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }
Remove-Item $temp
# Also update n8n-bms4.env.sops (n8n reads from this file)
$plain = sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops
$temp = "$PWD\secrets\n8n-bms4-edit.env.sops"
$updated = $plain `
    -replace "(?m)^SUPABASE_SERVICE_ROLE_KEY=.*$", "SUPABASE_SERVICE_ROLE_KEY=$env:NEW_SERVICE_KEY" `
    -replace "(?m)^SUPABASE_ANON_KEY=.*$", "SUPABASE_ANON_KEY=$env:NEW_ANON_KEY"
[System.IO.File]::WriteAllText($temp, ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv $temp
[System.IO.File]::WriteAllText("$PWD\secrets\n8n-bms4.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt — do NOT commit" }
Remove-Item $temp

Step 4 — Update GH Secrets

gh secret set SUPABASE_SERVICE_KEY --body $env:NEW_SERVICE_KEY --repo radieu/p24-infra
gh secret set SUPABASE_SERVICE_KEY --body $env:NEW_SERVICE_KEY --repo radieu/et-operational-platform
gh secret set SUPABASE_ANON_KEY    --body $env:NEW_ANON_KEY    --repo radieu/p24-infra
gh secret set SUPABASE_ANON_KEY    --body $env:NEW_ANON_KEY    --repo radieu/et-operational-platform

Step 5 — Commit SOPS + trigger secrets-sync

git checkout -b fix/rotate-supabase-keys-$(Get-Date -Format 'yyyy-MM-dd') origin/main
git add secrets/monitoring.env.sops secrets/n8n-bms4.env.sops
git commit -m "chore: rotate SUPABASE_SERVICE_ROLE_KEY + SUPABASE_ANON_KEY (scheduled 90d)"
git push -u origin HEAD
gh pr create --base main --title "chore: rotate Supabase project API keys (90d)"
gh pr merge --merge --delete-branch
# secrets-sync.yml fires automatically on merge to dev.
# It deploys .env to vps-i1, bms-4, AND runs sync-n8n-supabase-creds.py on bms-4.
# Wait for the GH Actions run to complete before proceeding.
gh run watch --repo radieu/p24-infra

Step 6 — Update Vercel env vars + trigger rebuild

CRITICAL — NEXT_PUBLIC_* vars are baked into the JS bundle at build time. Updating the env var in Vercel does NOT fix running deployments — you must trigger a full rebuild. Vercel CLI is NOT installed on this workstation — use the REST API pattern below.

# Extract VERCEL_TOKEN (from monitoring.env.sops, not et-op SOPS)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:VT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
    Select-String "^VERCEL_TOKEN=").ToString().Split("=",2)[1]
$env:SOPS_AGE_KEY_FILE = ""
 
$ET_OP_PID = "prj_ziLl911FOYLAeukQujL4NjxR4eWy"
$TEAM_ID   = "team_RuXLTWx5DXK6n0HUKoOUTGY3"
$hdrs = @{ Authorization = "Bearer $env:VT"; "Content-Type" = "application/json" }
 
# 1. Find env var IDs
$envs = (Invoke-RestMethod -Uri "https://api.vercel.com/v10/projects/$ET_OP_PID/env?teamId=$TEAM_ID" -Headers $hdrs).envs
$svcId  = ($envs | Where-Object { $_.key -eq "SUPABASE_SERVICE_ROLE_KEY" }).id
$anonId = ($envs | Where-Object { $_.key -eq "NEXT_PUBLIC_SUPABASE_ANON_KEY" }).id
 
# 2. PATCH service role key (type="encrypted" for server-only vars)
$b = @{ value = $env:NEW_SERVICE_KEY; type = "encrypted" } | ConvertTo-Json -Compress
Invoke-RestMethod "https://api.vercel.com/v10/projects/$ET_OP_PID/env/$svcId?teamId=$TEAM_ID" -Headers $hdrs -Method PATCH -Body ([System.Text.Encoding]::UTF8.GetBytes($b)) | Out-Null
 
# 3. PATCH anon key (type="plain" — NEXT_PUBLIC_ vars are public, plain type required)
$b = @{ value = $env:NEW_ANON_KEY; type = "plain" } | ConvertTo-Json -Compress
Invoke-RestMethod "https://api.vercel.com/v10/projects/$ET_OP_PID/env/$anonId?teamId=$TEAM_ID" -Headers $hdrs -Method PATCH -Body ([System.Text.Encoding]::UTF8.GetBytes($b)) | Out-Null
 
# 4. Verify — use individual decrypt endpoint (list endpoint masks values)
$check = Invoke-RestMethod "https://api.vercel.com/v10/projects/$ET_OP_PID/env/$anonId?teamId=$TEAM_ID&decrypt=true" -Headers $hdrs
Write-Host "Anon key verified sb_publishable_: $($check.value -like 'sb_publishable_*')"
 
# 5. Trigger production redeploy (find latest prod deployment, rebuild with new env vars)
$deploys = (Invoke-RestMethod "https://api.vercel.com/v6/deployments?projectId=$ET_OP_PID&teamId=$TEAM_ID&limit=20" -Headers $hdrs).deployments
$prodDeploy = ($deploys | Where-Object { $_.meta.githubCommitRef -eq "main" -and $_.state -eq "READY" } | Select-Object -First 1).uid
$rb = @{ deploymentId = $prodDeploy; name = "et-operational-platform"; target = "production" } | ConvertTo-Json -Compress
$redeploy = Invoke-RestMethod "https://api.vercel.com/v13/deployments?teamId=$TEAM_ID&forceNew=1" -Headers $hdrs -Method POST -Body ([System.Text.Encoding]::UTF8.GetBytes($rb))
Write-Host "Production redeploy triggered: $($redeploy.id)"
 
$env:VT = ""

Also update secrets/et-operational-platform.env.sops (separate from monitoring.env.sops):

# Same SOPS write pattern as Step 3 but for et-operational-platform.env.sops
$tempPath = "$PWD\secrets\.tmp-et-op-update.env.sops"  # named *.env.sops so creation rule matches
$plain = sops --decrypt --input-type dotenv --output-type dotenv secrets\et-operational-platform.env.sops
$updated = $plain -replace "(?m)^NEXT_PUBLIC_SUPABASE_ANON_KEY=.*$", "NEXT_PUBLIC_SUPABASE_ANON_KEY=$env:NEW_ANON_KEY" `
                  -replace "(?m)^SUPABASE_SERVICE_ROLE_KEY=.*$", "SUPABASE_SERVICE_ROLE_KEY=$env:NEW_SERVICE_KEY"
[System.IO.File]::WriteAllText($tempPath, ($updated -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --config "$PWD\.sops.yaml" --encrypt --input-type dotenv --output-type dotenv $tempPath
[System.IO.File]::WriteAllText("$PWD\secrets\et-operational-platform.env.sops", ($enc -join "`n"), [System.Text.UTF8Encoding]::new($false))
sops --decrypt --input-type dotenv --output-type dotenv secrets\et-operational-platform.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS corrupt" }
Remove-Item $tempPath

Step 7 — Pre-deletion verification gate (mandatory — do not skip)

Before deleting old keys, confirm every consumer returns 200 with the new key:

# 1. Supabase REST API directly
$r = Invoke-RestMethod "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/dev_r_services?limit=1" `
    -Headers @{ apikey = $env:NEW_SERVICE_KEY; Authorization = "Bearer $env:NEW_SERVICE_KEY" }
Write-Host "Supabase REST: $($r.Count) rows — OK"
 
# 2. et-operational-platform healthcheck
$r = Invoke-RestMethod "https://et-operational-platform.vercel.app/api/health" -ErrorAction SilentlyContinue
Write-Host "et-op healthcheck: $($r.status)"
 
# 3. vps-i1 queue-exporter
$r = Invoke-WebRequest "http://217.154.82.162:9200/metrics" -UseBasicParsing
Write-Host "queue-exporter: HTTP $($r.StatusCode)"
 
# 4. n8n test execution — trigger atrax GPS sync workflow
# Check last execution status in n8n dashboard or via Supabase GPS staleness:
$r = Invoke-RestMethod "https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/p24_gps_current_state?limit=1&order=n8n_synced_at.desc" `
    -Headers @{ apikey = $env:NEW_SERVICE_KEY; Authorization = "Bearer $env:NEW_SERVICE_KEY" }
Write-Host "GPS last sync: $($r[0].n8n_synced_at)"

All checks green? Proceed. Any red? Fix before deleting.

Step 8 — Delete old keys (irreversible)

# Use the OLD key IDs noted in Step 1
$OLD_SVC_ID = "<id-from-step-1>"
$OLD_ANON_ID = "<id-from-step-1>"
Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys/$OLD_SVC_ID" `
    -Method DELETE -Headers @{ Authorization = "Bearer $env:SB_MGMT" } `
    -Body (@{ reason = "rotating_key" } | ConvertTo-Json) -ContentType "application/json"
Write-Host "Old service key deleted"
Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys/$OLD_ANON_ID" `
    -Method DELETE -Headers @{ Authorization = "Bearer $env:SB_MGMT" } `
    -Body (@{ reason = "rotating_key" } | ConvertTo-Json) -ContentType "application/json"
Write-Host "Old anon key deleted"

Step 9 — Update tracking + close log entry

-- Run via psql on bms-4: psql "$SUPABASE_DB_URL" -c "..."
UPDATE dev_r_services
SET last_rotated = current_date,
    next_due     = current_date + interval '90 days'
WHERE service_name IN ('SUPABASE_SERVICE_ROLE_KEY', 'SUPABASE_ANON_KEY');

Append to docs/secrets-rotation-log.md:

| <YYYY-MM-DD> | SUPABASE_SERVICE_ROLE_KEY + SUPABASE_ANON_KEY | scheduled 90d | claude | SOPS monitoring+n8n-bms4 + GH Secrets + vps-i1 secrets-sync + n8n sync + Vercel redeploy |

Step 10 — Clear env vars

$env:NEW_SERVICE_KEY = ""; $env:NEW_ANON_KEY = ""; $env:SB_MGMT = ""
$env:SOPS_AGE_KEY_FILE = ""; $env:NEW_SERVICE_KEY_ID = ""; $env:NEW_ANON_KEY_ID = ""

Per-component key isolation migration

Goal: Replace one shared key with 3 named service_role keys (Gap 1 from gap analysis). Prerequisite: secrets-sync.yml wired to run sync-n8n-supabase-creds.py (Gap 2). Risk: Medium — requires coordinated Vercel redeploy + n8n sync, but old key stays live until last consumer confirmed.

Phase 1 — Create named keys (no consumer impact yet)

# Developer workstation only — administration.env.sops is excluded from secrets-sync.yml (#4585).
# On a bms-4 worker use: brandpilot.env.sops / SUPABASE_MANAGEMENT_TOKEN (worker-reachable — #5319; see § bms-4 worker — token source)
$env:SB_MGMT = (sops --decrypt --input-type dotenv --output-type dotenv secrets\administration.env.sops |
    Select-String "^SUPABASE_ACCESS_TOKEN=").ToString().Split("=",2)[1]
$REF = "mwkqmgadqnkkihjdeqsi"
 
# reveal=true is in use per Supabase support's suggestion (SU-439714) — does not by itself resolve #4626.
foreach ($name in @("p24_n8n", "p24_vercel", "p24_monitoring")) {
    $body = @{ name = $name; type = "secret" } | ConvertTo-Json
    $k = Invoke-RestMethod "https://api.supabase.com/v1/projects/$REF/api-keys?reveal=true" `
        -Method POST -Headers @{ Authorization = "Bearer $env:SB_MGMT"; "Content-Type" = "application/json" } `
        -Body $body
    Write-Host "$name created — ID: $($k.id)"
    # Store in env vars keyed by name — never print
    Set-Item "env:KEY_$($name -replace '-','_')" $k.api_key
}

Phase 2 — Add new SOPS vars (old vars remain active)

Add to secrets/monitoring.env.sops:

SUPABASE_SERVICE_KEY_MONITORING — for vps-i1 scripts and CI/CD
SUPABASE_SERVICE_KEY_VERCEL — for et-operational-platform + brandpilot

Add to secrets/n8n-bms4.env.sops:

SUPABASE_SERVICE_KEY_N8N — for n8n credentials

Use the SOPS write pattern from § Step 3 above.

Phase 3 — Migrate consumers one by one

3a. Monitoring scripts (vps-i1) — lowest risk, internal only Update monitoring.env.sops to replace SUPABASE_SERVICE_KEY references in monitoring stack scripts with SUPABASE_SERVICE_KEY_MONITORING. Merge + secrets-sync. Verify vps-i1 queue-exporter still returns metrics.

3b. CI/CD GH Secrets

gh secret set SUPABASE_SERVICE_KEY_MONITORING --body $env:KEY_p24_monitoring --repo radieu/p24-infra

3c. Vercel — et-operational-platform + brandpilot Update Vercel env vars to use SUPABASE_SERVICE_KEY_VERCEL. Trigger redeploy. Verify healthcheck returns 200.

3d. n8n credentials (bms-4) Update n8n-bms4.env.sops: set SUPABASE_SERVICE_ROLE_KEY = value of p24_n8n key. Merge + secrets-sync (automatically runs sync-n8n-supabase-creds.py). Verify GPS data freshness < 10 min.

Phase 4 — Delete old shared key

Run pre-deletion gate from § Step 7. All green → delete old shared service_role key via management API. Update dev_r_services + rotation log.

Phase 5 — n8n credential consolidation (Gap 6)

After per-component keys are live:

Audit the 3 httpHeaderAuth n8n credentials — test if they can be converted to supabaseApi type
Where possible, replace with supabaseApi type credential using SUPABASE_SERVICE_KEY_N8N
Goal: reduce from 6 to 3–4 credentials, all using consistent types

n8n credential sync

For n8n-specific credential sync procedure, diagnosis steps, and the credential ID table, see: n8n/n8n-supabase-credential-rotation.md

The sync script (scripts/sync-n8n-supabase-creds.py) is idempotent and safe to run at any time. It only patches credentials where the embedded key differs from SOPS.

After secrets-sync.yml is updated (Gap 2), this runs automatically on every merge that touches secrets/n8n-bms4.env.sops.

Verification

# 1. SOPS canary (no output = OK)
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops | Out-Null
 
# 2. Confirm key length (sb_secret_ prefix + body = ~100+ chars)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$len = ((sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
    Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1]).Length
Write-Host "SUPABASE_SERVICE_KEY length: $len"   # expect > 80
 
# 3. Rotation log entry exists
Select-String -Path "docs\secrets-rotation-log.md" -Pattern "SUPABASE" | Select-Object -Last 3

Escalation

Symptom	Action
Management API 401 on key creation	Rotate `SUPABASE_ACCESS_TOKEN` first — see `supabase-access-token-rotation.md`
Minted key returns 200 from Management API but `401 {"message":"Invalid API key"}` from the data API (`.../rest/v1/...`)	Control-plane/data-plane desync, or the management token’s owning org lacks write-linkage to project `mwkqmgadqnkkihjdeqsi` (#4626 — still open, root cause unknown). Supabase support’s `?reveal=true` theory was tested 2026-08-06 and disproven (a throwaway key minted + re-fetched with `reveal=true` still 401’d) — this playbook uses `reveal=true` on create/read-back calls per their suggestion, but do not expect it to fix the 401. The § Step 2.5 gate catches this before distribution — delete the dead key, do NOT distribute. Fix: verify org membership at `app.supabase.com/account/tokens` (Tier-3 human), or mint via a known-good token / dashboard, before retrying rotation.
n8n workflows return 401 after rotation	Run `sync-n8n-supabase-creds.py` manually — see `n8n/n8n-supabase-credential-rotation.md`
Vercel returns 500 after key update	Check Vercel function logs; confirm `SUPABASE_SERVICE_KEY` env var was updated and redeploy triggered
Old key still works 5 min after delete	Supabase CDN caching — wait 2 min and retest; if persists, contact Supabase support
GPS data stale > 10 min after rotation	See `n8n-supabase-credential-rotation.md §Confirm the diagnosis`

Prevention notes

Supabase does not send expiry warnings for project API keys.
credential-rotation.yml (Monday 06:00 UTC) checks dev_r_services.next_due and opens a human-action issue when rotation is due. This only works if last_rotated is updated after every rotation (Step 9).
After secrets-sync.yml is updated (Gap 2), n8n sync is automatic on merge — no manual step.
The management API (SUPABASE_ACCESS_TOKEN) enables full rotation automation without dashboard access.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="SUPABASE_SERVICE_KEY",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — Supabase service role key rotated and all consumers updated",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'SUPABASE_SERVICE_KEY', 'success', 'Scheduled rotation — Supabase service role key rotated and all consumers updated', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

supabase-service-key-rotation