p24-infra Docs

atrax-daily-workflows-paired-item-and-stale-key

13 min read

Playbook — atrax daily workflows error on every scheduled run (paired-item + stale Supabase key)

Workflows (bms-4 n8n):

  • atrax-daily-stats-collector (id Wi3ZJcPw0YbxLXwa, daily 00:00 UTC)
  • atrax-drivers-daily-snapshot (id akgnvueDkDPbLJ49, daily 04:00 UTC)
  • atrax-ecodriving-daily (id zdzyIQ8UBsDAF0wI, daily 04:30 UTC)

Issue: #1990 (spun off from the #1986 crash storm). After the worker crash loop was resolved, the scheduler fires these three daily workflows but each errors in <1 s.

Key correction to the #1990 premise. The fast-fail pattern is not a single shared credential problem. There are two distinct root causes — one credential, two paired-item expression bugs. Fix them separately.

Confirmation commands

Pull the failing node + error for the latest run of each workflow from the n8n execution store:

docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -At -F'|' -c "
SELECT w.name, we.id, we.status, we.\"startedAt\"
FROM execution_entity we JOIN workflow_entity w ON w.id = we.\"workflowId\"
WHERE w.name ~* 'daily-stats-collector|drivers-daily-snapshot|ecodriving-daily'
  AND we.\"startedAt\" > NOW() - INTERVAL '4 days'
ORDER BY w.name, we.\"startedAt\" DESC;"
 
# Then read the error payload for a given execution id:
docker exec bms-4-n8n-postgres-1 psql -U n8n -d n8n -At -c \
  "SELECT LEFT(data, 4000) FROM execution_data WHERE \"executionId\" = <EID>;"

Root cause 1 — atrax-daily-stats-collector: stale hardcoded Supabase key (401)

  • Failing node: fetch_vehicles (HTTP Request) — GET https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/p24_l_cars.
  • Error: NodeApiError: Authorization failed / 401 - {"message":"Invalid API key","hint":"Double check your API key."}.
  • Cause: the node carries a hardcoded Supabase key in its apikey + Authorization: Bearer header parameters (not an n8n credential). That key is the legacy service_role JWT that was scrubbed from n8n-workflows/ git history (commit b7b9fe0) and deactivated during the 2026-06-21 scoped-key migration (see docs/secrets-rotation-log.md). The live node was never updated, so once the legacy key was retired the node started returning 401.

Fix (operational — requires the n8n UI or CLI; touches a secret value):

  1. Prefer converting fetch_vehicles to use the same predefined Supabase / Header-Auth credential that the working atrax-ecodriving-daily → fetch_drivers_list node uses, instead of a hardcoded header. This removes the secret from the node entirely.
  2. If a hardcoded header must stay, replace the dead key with the current valid key read from SOPS — never echo the value: 
    KEY=$(sops -d --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/n8n-bms4.env.sops | grep "^SUPABASE_SERVICE_KEY=" | cut -d= -f2-)
# paste $KEY into the node's apikey + Authorization headers via the UI; then: unset KEY
  3. Follow docs/playbooks/n8n/n8n-supabase-credential-rotation.md for distribution discipline. Do not commit the key to the exported JSON (n8n redacts it on export — keep it that way).

Root cause 2 — drivers-daily-snapshot + ecodriving-daily: paired-item expression failure

  • Failing nodes: map_drivers (Code) in drivers-daily-snapshot; fetch_drivers_list (Supabase filter) and build_request (Code) in ecodriving-daily.
  • Error: ExpressionError: Invalid expression (drivers-snapshot surfaces it as the n8n-2.26.x TypeError: Cannot assign to read only property 'name' wrapper) raised from createNoConnectionError / pairedItem resolution.
  • Cause: these nodes reference $('compute_yesterday').item.json.stat_date (and $('read_token').item). .item relies on paired-item linking back to the source node. The upstream chain runs through a Code node (compute_yesterday, which returns items without pairedItem metadata) and the newer n8n-nodes-base.dataTable token nodes (save_token/read_token) that do not propagate pairedItem. Under n8n 2.26.3 the paired-item resolver is strict, so .item can no longer be resolved → “Invalid expression”.

Fix (in version control — see the PR for #1990): replace .item with .first() on every reference to a single-item source node. compute_yesterday always emits exactly one item and read_token returns one token row, so .first() is semantically identical and does not depend on paired-item linking:

WorkflowNodeBeforeAfter
drivers-daily-snapshotmap_drivers$('compute_yesterday').item.json.stat_date$('compute_yesterday').first().json.stat_date
ecodriving-dailyfetch_drivers_list filter$('compute_yesterday').item...$('compute_yesterday').first()...
ecodriving-dailybuild_request$('compute_yesterday').item...$('compute_yesterday').first()...
ecodriving-dailyflatten_and_map$('compute_yesterday').item...$('compute_yesterday').first()...
ecodriving-dailypost_ecodriving$('read_token').item...$('read_token').first()...

Applying the fix to LIVE n8n (mandatory — repo is export-only)

n8n-workflows/ is a one-way export snapshot (scripts/export-n8n-workflows.sh). Editing the JSON in git does not change the running workflows. To restore data collection, import the corrected definitions into the live bms-4 n8n DB:

# copy the corrected file in, then import (writes to the production n8n DB):
docker cp n8n-workflows/atrax-drivers-daily-snapshot_akgnvueDkDPbLJ49.json bms-4-n8n-1:/tmp/wf.json
docker exec bms-4-n8n-1 n8n import:workflow --input=/tmp/wf.json
# repeat for atrax-ecodriving-daily_zdzyIQ8UBsDAF0wI.json

⚠️ LANDMINE — import:workflow CLOBBERS hardcoded secrets with the export redaction placeholder (#5722)

scripts/export-n8n-workflows.sh redacts hardcoded secret-like body params before committing to git: get_token’s client_secret is stored in the git JSON as the literal string REDACTED_CREDENTIAL, not the real value. n8n import:workflow writes the JSON verbatim, so importing the git file overwrites the real live client_secret with REDACTED_CREDENTIAL → Atrax OAuth then fails 401 {"error":"invalid_client","error_description":"Bad client credentials"} at get_token on the next run (looks like a credential incident but is import damage). Confirmed on 2026-08-06 (#5722).

After ANY import:workflow of these workflows, restore the real client_secret — it survives in workflow_history (n8n snapshots the pre-import version). Recover it with a pure-SQL copy from the newest pre-import snapshot (value never leaves the DB, never printed):

-- find the pre-import snapshot: newest workflow_history row whose createdAt precedes the import;
-- its get_token client_secret is real (length ~31), the post-import one is 'REDACTED_CREDENTIAL' (19).
WITH rs AS (
  SELECT (SELECT p->>'value' FROM jsonb_array_elements(n->'parameters'->'bodyParameters'->'parameters') p
          WHERE p->>'name'='client_secret') AS secret
  FROM workflow_history wh, jsonb_array_elements(wh.nodes::jsonb) n
  WHERE wh."versionId"='<PRE_IMPORT_VERSION_ID>' AND n->>'name'='get_token')
UPDATE workflow_entity we SET nodes = (
  SELECT jsonb_agg(CASE WHEN elem->>'name'='get_token'
    THEN jsonb_set(elem,'{parameters,bodyParameters,parameters}',
      (SELECT jsonb_agg(CASE WHEN p->>'name'='client_secret'
         THEN jsonb_set(p,'{value}',to_jsonb((SELECT secret FROM rs))) ELSE p END)
       FROM jsonb_array_elements(elem->'parameters'->'bodyParameters'->'parameters') p))
    ELSE elem END) FROM jsonb_array_elements(we.nodes::jsonb) elem)::jsonb::json
WHERE we.id='<WORKFLOW_ID>';

Then deactivate→reactivate the workflow (n8n public API POST /api/v1/workflows/{id}/deactivate then /activate) so the running process reloads the restored definition — n8n update:workflow --active from the CLI updates the DB but the running instance keeps the stale in-memory copy until reloaded.

Permanent fix (recommended, #5722 follow-up): convert get_token’s client_secret to an env reference ={{ $env.ATRAX_CLIENT_SECRET }} (like username/password already are) so no secret lives in the DB/git and the export→import round trip has nothing to clobber. Requires secret-manager to add ATRAX_CLIENT_SECRET to secrets/n8n-bms4.env.sops + secrets-sync.yml distribution to the n8n containers (main + workers, since queue-mode workers execute the nodes).

Then trigger one manual run per workflow and confirm status=success in the execution store. These are schedule-trigger-only workflows: n8n execute --id cannot run them (it starts only from start/manualTrigger nodes, and inside the queue-mode container it also collides with the running task-broker on port 5679). Trigger through the running engine instead — log in to the internal REST API as owner (POST /rest/login), then POST /rest/workflows/{id}/run with body {"workflowData":<def>,"runData":{},"triggerToStartFrom":{"name":"Schedule Trigger"}}.

Root cause 3 — drivers-daily-snapshot: compute_yesterday runs AFTER map_drivers (never worked) (#5722)

Even with .first(), map_drivers fails Cannot assign to read only property 'name' of object 'Error: Node 'compute_yesterday' hasn't been executed'. read_token fans out to two parallel branches — [fetch_rtdrivers, compute_yesterday] — and n8n runs the fetch_rtdrivers → map_drivers branch first, so compute_yesterday has not executed when map_drivers evaluates $('compute_yesterday').first(). .item merely surfaced the same “not executed” as the paired-item Invalid expression wrapper. Robust fix (dev-coder): linearize compute_yesterday to the front — Schedule Trigger → compute_yesterday → get_token → save_token → read_token → fetch_rtdrivers → map_drivers and drop the read_token → compute_yesterday edge (get_token uses $env, so it ignores compute_yesterday’s pass-through item; fetch_rtdrivers keeps read_token as its direct input for Bearer {{ $json.value }}). Do not rely on reordering the parallel branch — branch execution order is not a documented guarantee.

Root cause 4 — Supabase upsert nodes are misconfigured; tables were always empty (#5722)

upsert_drivers fails Could not get parameter "tableId", and both p24_atrax_drivers_daily and p24_driver_ecodriving_daily are empty — these workflows have never written a row. operation: "upsert" is not a valid operation for n8n-nodes-base.supabase typeVersion 1 (the node only supports create/get/getAll/update/delete; no other workflow on the bms-4 instance uses upsert), and fieldsUi.fieldValues is empty (no column mapping). Both upsert nodes (upsert_drivers, upsert_ecodriving) need a real rebuild — dev-coder: switch to create with a full fieldsUi column mapping, or an HTTP Request node hitting Supabase REST with Prefer: resolution=merge-duplicates for true upsert. Note atrax-ecodriving-daily is blocked by drivers: its fetch_drivers_list reads p24_atrax_drivers_daily (filtered to yesterday), so it returns 0 drivers and skips to a hollow success until drivers actually populates that table.

Backfill is not possible. drivers-daily-snapshot captures point-in-time current tachograph state (dayDrivingTime, thisWeekRemainingTime, …), not historical daily data — past days cannot be reconstructed. Ecodriving backfill is moot until drivers works (and would additionally require Atrax historical date-range support).

Resolution (#5729, 2026-08-06)

Root causes 3–4 + the import landmine were fixed in the #5729 PR (dev-coder rebuild of the two workflow JSONs) — mirroring the already-working atrax-daily-stats-collector:

  • Bug 1 (upsert): each upsert_* Supabase node (invalid operation: upsert, empty mapping) was replaced by an aggregate node (aggregateAllItemData → one { data: [...] } item) feeding an HTTP Request node that POSTs to /rest/v1/<table>?on_conflict=<cols> with Prefer: resolution=merge-duplicates,return=minimal, authed by the existing httpHeaderAuth credential supabase-service-role-key (id ZNuDXmNs3nT58L6p) — no hardcoded key. A migration (20260806…_atrax_daily_unique_indexes_5729.sql) adds the required unique indexes p24_atrax_drivers_daily(driver_id,stat_date) and p24_driver_ecodriving_daily(driver_id,atrax_car_id,stat_date).
  • Bug 2 (drivers topology): linearized to Schedule → compute_yesterday → get_token → save_token → read_token → fetch_rtdrivers → map_drivers → aggregate_drivers → upsert_drivers (dropped read_token→compute_yesterday), so compute_yesterday always runs before map_drivers.
  • Bug 3 (import landmine): both get_token client_secret params converted to ={{ $env.ATRAX_CLIENT_SECRET }}. Distribution of ATRAX_CLIENT_SECRET to secrets/n8n-bms4.env.sops + the n8n containers (main + 3 queue workers) is a secret-manager step (#5730) — the live import must not run until that value is live, or get_token → 401.

The repo JSON is the source; a follow-up infra-task import into live bms-4 n8n is still required after merge + secret distribution (see “Applying the fix to LIVE n8n” above).

Resolution (#5918, 2026-08-09) — live import done for drivers; the other two are Atrax-signature-blocked

The #5729 rebuild had merged but never been imported to live, and a later re-export (#5913/#5914, n8n-export-transient-retry) clobbered the corrected repo JSONs with the still- broken live versions — so at the start of #5918 both repo HEAD and live were the broken versions (the good #5729 defs survived only in merge commit d00d77f). Empirical investigation on bms-4 then showed the documented root causes are not the live blockers for two of the three workflows:

  • atrax-drivers-daily-snapshot — FIXED & VERIFIED. Its only external dependency, GET /api/rtdrivers, is an open Atrax endpoint (works with a plain Bearer token). Recovered the #5729 def from d00d77f, confirmed it uses $env for all secrets (no hardcoded/REDACTED values) and the valid supabase-service-role-key httpHeaderAuth credential (ZNuDXmNs3nT58L6p), imported it to live, reactivated, and triggered one manual run → status=success, 256 rows landed in p24_atrax_drivers_daily for stat_date 2026-08-08 (table was previously empty). Note: the n8n supabase-service-role-key credential is valid (post-#5760 rotation) — the earlier “stale key” hypothesis was wrong; its 401s came from elsewhere (see below).

  • atrax-daily-stats-collector — BLOCKED (Atrax request signature). The live 401 is NOT the Supabase key. fetch_vehicles/upsert_stats (Supabase) succeed with the valid credential; the failure is at fetch_exploitation → GET https://tronik.atrax4.com/api/exploitation → 401 {"message":"Invalid request hash"}, reproduced with a fresh valid token directly. This endpoint requires an Atrax request signature/hash no workflow implements, so p24_gps_daily_stats has always been empty and the 2026-08-08 backfill is not possible via this API. (The atrax_kravag-scheduled-fleet-updates workflow “succeeds” only because its scheduled path does not execute its orphan /api/exploitation node.)

  • atrax-ecodriving-daily — BLOCKED (Atrax request signature) + downstream of drivers. post_ecodriving → POST /api/statistics/daily/drivers → 400 {"message":"Błędna sygnatura czasu"} (invalid time signature) for every date format tried — same signature-gating class as exploitation. Left the old (hollow-success) live def in place deliberately: importing #5729 now that drivers populates would make post_ecodriving fire and turn the hollow success into a hard error with no data gain. Not imported pending the Atrax signature fix.

The Atrax signature-gating on the statistics/exploitation endpoints needs vendor docs or reverse-engineering (a dev/vendor task) — tracked as a separate escalation issue (#5921); it is out of scope for the infra-task import.

Confirmation (#5722 re-queue, 2026-08-11) — #5921 is the terminal ecodriving blocker; two more downstream bugs found first

A later re-queue of #5722 re-diagnosed atrax-ecodriving-daily as “just the native-Supabase upsert_ecodriving node” and specced an aggregate + HTTP-Request-upsert rebuild (mirroring the working upsert_drivers), unaware of the 5921 signature block above. An infra-task worker on bms-4 then drove that rebuild all the way to a live end-to-end run (temporarily pointing the Schedule Trigger at an every-minute cron so the real queue-mode scheduler — which has the DataTable module the n8n execute CLI lacks — ran it; live def restored to the original afterward, git == live preserved). That live run walked the fix past the upsert node and proved the true failure order:

  1. skip_if_no_drivers boolean-IF misroutes every valid item to the skip branch. The IF condition is leftValue = {{ $json.skip }} (a boolean from build_request) equal false, typeValidation: strict. On live runs where build_request correctly emitted skip:false with a request body (drivers present for the date), the item still exited on the false/skip output — post_ecodriving never ran. This is why every past atrax-ecodriving-daily run was a hollow status=success with 0 rows even after p24_atrax_drivers_daily was populated. n8n 2.26.x renders a boolean-false expression unreliably in a strict boolean IF. Fix: drop the IF entirely (the working atrax-drivers-daily-snapshot sibling has no such node) and have build_request return [] when driverIds.length === 0, so post_ecodriving simply receives 0 items and the run ends cleanly.
  2. post_ecodriving is missing rawContentType: "application/json". With contentType: "raw" but no rawContentType, n8n does not send a JSON media type and Atrax replies HTTP 415 Unsupported Media Type. The working upsert_drivers node sets rawContentType: "application/json". Fix: add it (a manual Content-Type: application/json header alone is not sufficient in raw mode).
  3. Only after (1) and (2) are fixed does the real blocker surface: post_ecodriving → POST https://tronik.atrax4.com/api/statistics/daily/drivers → HTTP 400 {"message":"Błędna sygnatura czasu"} — the #5921 Atrax request-signature gate (same class as /api/exploitation’s Invalid request hash). A plain date and every other timestamp shape are rejected; the endpoint needs the vendor signature scheme. This is the terminal blocker and it is out of infra-task scope — it belongs to #5921.

Net: the aggregate + HTTP-upsert rebuild (#5729 spec) and fixes (1) and (2) above are all necessary for ecodriving, but all three are moot until #5921 lands — deploying them without the signature turns the current hollow-success into a daily hard error at post_ecodriving with no data gain (exactly why #5918 left the old def in place). Whoever implements #5921 must apply (1)+(2)+the #5729 upsert rebuild in the same change so ecodriving works in one shot. Live was left on the original (hollow-success) def; the ecodriving table stays empty until #5921.

Prevention

  • New n8n nodes that read a value produced earlier in the flow should use .first() (or .all()), not .item, whenever the source is a single-item node or the chain passes through a Code/dataTable node — .item paired-item linking is fragile across those node types and n8n upgrades.
  • Supabase-calling nodes should use a predefined credential, never a hardcoded apikey/Bearer header, so key rotations propagate automatically.
  • Never hardcode a secret in an httpRequest body param — the git export redacts it to REDACTED_CREDENTIAL and re-importing the file clobbers the live value (Root causes above). Use an $env.* reference so the DB/git never hold the secret.
  • Before declaring an n8n data pipeline “fixed”, verify the destination table actually received rows — a green status=success can be hollow (0 items upserted, or a skip branch taken), as both atrax tables were until #5722.

First documented: 2026-06-29 (issue #1990). Root causes 3–4 + import landmine: 2026-08-06 (#5722).

Graph View