p24-infra Docs

n8n-supabase-credential-rotation

8 min read

Playbook — n8n Supabase credentials revoked after a Supabase key rotation

Trigger: One or more n8n workflows that read/write Supabase suddenly fail with 401 - "Unregistered API key" / "Double check the provided API key as it is not registered for this project." The most visible symptom is the Atrax data-freshness alert (#1371) going from false-positive (empty ages) to genuinely stale — GPS age climbing steadily past the 10-min threshold while p24_gps_current_state / p24_l_cars_atrax stop advancing entirely.

This is distinct from the transient false-positive case in atrax-data-freshness-false-positive.md. Here the data is really stale because the n8n GPS-sync workflow cannot authenticate to Supabase at all.

Root cause (incident 2026-06-26 → 06-27)

Supabase migrated to the new key system (sb_secret_… / sb_publishable_…) and the legacy JWT keys were disabled (2026-06-21). When a Supabase key is rotated, the new value must land in two independent places:

  1. SOPS (secrets/n8n-bms4.env.sops → SUPABASE_SERVICE_ROLE_KEY, SUPABASE_ANON_KEY) — deployed to .env files by secrets-sync.yml.
  2. n8n credentials — the key is embedded inside each n8n credential object, encrypted with the n8n instance key. secrets-sync.yml does NOT touch n8n credentials.

In this incident SOPS was updated and exactly one of six n8n credentials was fixed manually; the other five kept the old, now-revoked key. Every workflow using one of those five started returning 401 Unregistered API key. The atrax GPS-sync workflow (atrax-kravag-scheduled-fleet-updates, CCx9UMdphmGficDX) writes via an HTTP Request node using a generic httpHeaderAuth credential (supabaseKapbaraAuthBearer), so fixing only the supabaseApi credentials would not have restored it. Fleet data stopped at 2026-06-26 17:45 UTC and stayed frozen ~12 h.

The six Supabase credentials in bms-4 n8n

Credential idNameTypeKey fieldKey kind
8bVcpRNbyLOHJDrUSupabase accountsupabaseApiserviceRolesb_secret_
1pvpQp7GfpAxNFIhSupabase - kapibara - service keysupabaseApiserviceRolesb_secret_
rFL6zCStG02XlFGKsupabase-dev-rsupabaseApiserviceRolesb_publishable_
cMpMbkDFAJzC3O4fsupabaseKapbaraAuthBearerhttpHeaderAuthvaluesb_secret_
aMxMREZhtabQV3Z7supabase-anon-apikeyhttpHeaderAuthvaluesb_publishable_
ZNuDXmNs3nT58L6psupabase-service-role-keyhttpHeaderAuthvaluesb_secret_

The atrax write node (update-atrax-pojazdy-w car-atrax) uses cMpMbkDFAJzC3O4f. The read node (Get many rows) uses 8bVcpRNbyLOHJDrU. Both must be valid for the workflow to complete.

Confirm the diagnosis

Run on bms-4 (has docker access + the age key). Never print key values — fingerprints only.

# Is the data really stale? (psql or n8n SQL node)
#   SELECT max(n8n_synced_at) FROM p24_gps_current_state;   -- > 10 min old == genuinely stale
 
# Which n8n executions failed and why?
PGUSER=$(docker exec bms-4-n8n-postgres-1 printenv POSTGRES_USER)
PGDB=$(docker exec bms-4-n8n-postgres-1 printenv POSTGRES_DB)
docker exec bms-4-n8n-postgres-1 psql -U "$PGUSER" -d "$PGDB" -At -c \
 "SELECT id,status,\"stoppedAt\" FROM execution_entity \
  WHERE \"workflowId\"='CCx9UMdphmGficDX' ORDER BY \"stoppedAt\" DESC LIMIT 5;"
# 'error' rows whose data contains "Unregistered API key" == this playbook.

Fix — one command

Run the sync script on bms-4. It auto-discovers every Supabase credential, tests each embedded key, and replaces only the revoked ones with the current SOPS key (by prefix). Idempotent and secret-safe.

cd /opt/p24-infra   # or the working checkout
export SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt
python3 scripts/sync-n8n-supabase-creds.py --dry-run   # preview
python3 scripts/sync-n8n-supabase-creds.py             # apply

No n8n restart is needed — credentials are decrypted from the DB per execution. The next scheduled run of CCx9UMdphmGficDX (every ~3 min) will succeed; data freshness returns within ~5 min.

import failed for <id>: with an empty reason (#2070): earlier the script wrote its docker cp import file as mode 0600 owned by the host runner uid, so the container’s node user (uid 1000) could not read it and n8n import:credentials silently no-op’d — diagnosis was correct (fixed=N in --dry-run) but the apply reported failed=N. Fixed: the script now chmod 0644s the temp file before docker cp (works for any runner uid) and surfaces the CLI exit code when stderr is empty. If you still see this on an old checkout, git pull first, or use the container-side manual fallback below (export/edit/import entirely inside bms-4-n8n-1).

Manual fallback (if the script is unavailable)

For each revoked credential id, replacing the key field without ever printing it:

export SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt
NEWKEY=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops \
  | grep '^SUPABASE_SERVICE_ROLE_KEY=' | cut -d= -f2-)        # or SUPABASE_ANON_KEY for publishable
docker exec bms-4-n8n-1 n8n export:credentials --id=<CID> --decrypted --output=/tmp/c.json
docker cp bms-4-n8n-1:/tmp/c.json /tmp/c.json
NEWKEY="$NEWKEY" python3 -c "import json,os; d=json.load(open('/tmp/c.json')); c=d[0]; \
  f='value' if c['type']=='httpHeaderAuth' else 'serviceRole'; c['data'][f]=os.environ['NEWKEY']; \
  json.dump(d,open('/tmp/c.json','w'))"
docker cp /tmp/c.json bms-4-n8n-1:/tmp/c.json
docker exec bms-4-n8n-1 n8n import:credentials --input=/tmp/c.json
rm -f /tmp/c.json; docker exec bms-4-n8n-1 rm -f /tmp/c.json; unset NEWKEY

Verify

# next atrax execution must be 'success', data fresh < 10 min
# then nudge the freshness check so it auto-closes the atrax-stale issue:
gh workflow run atrax-data-freshness.yml --repo radieu/p24-infra

Variant — inline hardcoded apikey/Authorization in node params (#2341)

A nastier variant: the workflow does not reference an n8n credential at all — the Supabase key is inlined as a literal header value in the HTTP Request node’s parameters.headerParameters (apikey: <key> and/or Authorization: Bearer <key>), with authentication: none. When the key is rotated, these nodes keep the stale literal and start returning 401 "Invalid API key" / "Authorization failed".

sync-n8n-supabase-creds.py does NOT fix this case — it only rewrites the embedded key inside credential objects; it never touches workflow node parameters. So a rotation can leave every credential green while an inline-key workflow stays broken (this is how the #2269 rotation broke fleet-update-v2-batch / AJ1px9uHIfbsriof — discovered as #2341). It also violates CLAUDE.md (never hardcode secret values in n8n node params).

Detect inline-key nodes (fingerprints only — never print the value)

PGUSER=$(docker exec bms-4-n8n-postgres-1 printenv POSTGRES_USER)
PGDB=$(docker exec bms-4-n8n-postgres-1 printenv POSTGRES_DB)
# List httpRequest nodes whose inline headers include apikey/Authorization (NAMES only, no values)
docker exec bms-4-n8n-postgres-1 psql -U "$PGUSER" -d "$PGDB" -At -c \
 "SELECT w.id, w.name, n->>'name'
  FROM workflow_entity w,
       jsonb_array_elements(w.nodes::jsonb) n,
       jsonb_array_elements(COALESCE(n#>'{parameters,headerParameters,parameters}','[]'::jsonb)) hp
  WHERE n->>'type'='n8n-nodes-base.httpRequest'
    AND lower(hp->>'name') IN ('apikey','authorization')
    AND (n#>>'{parameters,authentication}') IS DISTINCT FROM 'genericCredentialType';"

Fix — repoint to the shared credential, delete the inline headers

Use the existing httpHeaderAuth credential supabase-service-role-key (ZNuDXmNs3nT58L6p, header name apikey, holds the current sb_secret_… service-role key). The new sb_secret_ key in the apikey header alone authenticates as service_role for PostgREST/RPC — the separate Authorization: Bearer header is no longer needed. For each affected node:

  1. Set parameters.authentication = "genericCredentialType", parameters.genericAuthType = "httpHeaderAuth".
  2. Add credentials.httpHeaderAuth = {"id":"ZNuDXmNs3nT58L6p","name":"supabase-service-role-key"}.
  3. Remove the inline apikey and Authorization entries from parameters.headerParameters.parameters (keep Content-Type / Prefer).

Edit via the n8n UI, or export → patch the node JSON → re-import (mirror the export:workflow / import:workflow flow used for credentials above). No restart needed; the next scheduled run picks up the change.

Verify the credential is current before trusting it (fingerprint match, never echo)

docker exec bms-4-n8n-1 n8n export:credentials --id=ZNuDXmNs3nT58L6p --decrypted --output=/tmp/c.json
CRED_FP=$(docker exec bms-4-n8n-1 node -e 'const f=require("fs"),c=require("crypto");const d=JSON.parse(f.readFileSync("/tmp/c.json"));let v=(Array.isArray(d)?d[0]:d).data.value.replace(/^Bearer\s+/i,"").trim();process.stdout.write(c.createHash("sha256").update(v).digest("hex").slice(0,12))')
docker exec bms-4-n8n-1 rm -f /tmp/c.json
ENV_FP=$(grep -m1 '^SUPABASE_SERVICE_ROLE_KEY=' /opt/p24-infra/bms-4/.env | cut -d= -f2- | tr -d '"\r\n' | sha256sum | cut -c1-12)
[ "$CRED_FP" = "$ENV_FP" ] && echo "credential is current ✅" || echo "credential STALE — run sync-n8n-supabase-creds.py ❌"

Then confirm: latest execution_entity row for the workflow is success, and the target table (p24_l_cars_atrax) updated_at is advancing (< 10 min old).

Prevention

This playbook covers incident recovery only — restoring n8n credentials after they have broken due to a Supabase key rotation.

For the full scheduled rotation procedure (creating new Supabase keys, distributing to all consumers, per-component key isolation plan, and pre-deletion verification): → supabase-service-key-rotation.md

Key points from that playbook relevant to n8n:

  • Supabase project API keys can be created/deleted via the management API using SUPABASE_ACCESS_TOKEN — no dashboard required.
  • After secrets-sync.yml is updated (Gap 2 from the gap analysis), sync-n8n-supabase-creds.py will run automatically on every merge that touches secrets/n8n-bms4.env.sops.
  • Until then: always run the script manually immediately after any Supabase key rotation in SOPS.

Gap analysis: supabase-secrets-management-gaps.md §Gap 2

Related / follow-ups

  • Sibling: atrax-data-freshness-false-positive.md (transient blip, not a real outage) and atrax-fleet-sync-409-replate.md.
  • n8n execution backlog: during this incident the DB held ~64k zombie new executions from 2026-06-17 plus thousands of crashed rows. They did not cause the outage but bloat the n8n DB — consider tightening EXECUTIONS_DATA_PRUNE / EXECUTIONS_DATA_MAX_AGE and pruning the orphaned new rows in a separate maintenance task.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="N8N_SUPABASE_CREDENTIAL",
    result="success",  # "success" | "failed" | "skipped"
    detail="Rotation — n8n Supabase credential (API key or service key) updated in n8n and SOPS",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'N8N_SUPABASE_CREDENTIAL', 'success', 'Rotation — n8n Supabase credential (API key or service key) updated in n8n and SOPS', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks