p24-infra Docs

atrax-data-freshness-false-positive

3 min read

Playbook — Atrax data-freshness false-positive “stale” alert

Trigger: A GitHub issue titled Atrax data stale: ... (label atrax-stale) is opened by the Atrax Data Freshness workflow, and/or a Still stale: comment shows empty ages (GPS age: s | cars age: s / GPS age: n/a).

Related: #1371. Sibling pattern: pinbox24 no-logs false positive (a check failing on transient fetch error rather than the thing it claims to measure).

What the alert actually means

.github/workflows/atrax-data-freshness.yml runs every 10 min on the self-hosted bms4 runner and checks two Supabase tables via the REST API:

CheckTableFieldThreshold
gps-stalep24_gps_current_staten8n_synced_at< 600 s (10 min)
cars-atrax-stalep24_l_cars_atraxupdated_at< 90000 s (25 h)

The data is produced by the n8n GPS-sync workflow atrax-kravag-scheduled-fleet-updates (CCx9UMdphmGficDX on bms-4). When that workflow is healthy, both tables update every ~30–60 s.

Confirm: is the data actually stale? (do this FIRST)

Run against Supabase (mcp__claude_ai_Supabase__execute_sql, project mwkqmgadqnkkihjdeqsi):

SELECT 'gps_current_state' AS check, MAX(n8n_synced_at) AS latest, NOW() - MAX(n8n_synced_at) AS age
FROM p24_gps_current_state
UNION ALL
SELECT 'cars_atrax', MAX(updated_at), NOW() - MAX(updated_at)
FROM p24_l_cars_atrax;
  • Age is small (seconds/minutes) → the data is fine; this was a false positive (transient Supabase/network blip during the check). The alerting issue auto-closes on the next successful run.
  • Age exceeds the threshold → genuine staleness; go to “If genuinely stale” below.

Why the false positives happen

The check fired the stale label whenever curl returned a non-200 / empty body — i.e. a transient network or Supabase 5xx blip was indistinguishable from real staleness, and there was no retry. A single hiccup = a false “stale” alert with empty ages. Compounded by the bms4 self-hosted runner backlog (scheduled runs queueing for hours, then running a stale workflow snapshot).

The fix (shipped in #1371)

atrax-data-freshness.yml now:

  1. Retries the Supabase call: curl --retry 4 --retry-delay 5 --retry-all-errors --max-time 25.
  2. Captures the HTTP status (-w '%{http_code}') and distinguishes three states per check: ok / stale (age measured > threshold) / fetch-error (could not reach Supabase).
  3. Only a measured stale state opens/keeps the atrax-stale issue. A pure fetch-error is logged as a run ::warning:: and does not page — if data is truly going stale, the next run that reaches Supabase reports it correctly.
  4. Real ages (or n/a) and a human-readable reason are included in the issue/comment body.

If genuinely stale (age > threshold)

  1. Open the n8n GPS-sync workflow: https://n8n.bms-4.infra.zintegrowana.online/workflow/CCx9UMdphmGficDX (atrax-kravag-scheduled-fleet-updates). Check the last execution + error.
  2. Common causes: expired ATRAX_AUTH_STRING credential, Atrax API 409/replate errors (see atrax-fleet-sync-409-replate.md), a revoked Supabase key in the n8n credential after a Supabase key rotation (401 Unregistered API key in the execution — see n8n/n8n-supabase-credential-rotation.md), or a bms-4 n8n worker stall.
  3. Trigger a manual run (idempotent). Re-run the freshness check: gh workflow run atrax-data-freshness.yml --repo radieu/p24-infra.
  4. Once data is fresh, the workflow auto-closes the atrax-stale issue.

Prevention / escalation

  • A flood of Still stale: comments with empty/n/a ages after this fix means the Supabase REST API or the SUPABASE_SERVICE_KEY is genuinely failing — check Supabase status and the key validity (curl .../rest/v1/ should return 200). Escalate via Discord + a human-action issue.
  • Persistent cancelled/queued freshness runs → the bms4 self-hosted runner is backlogged; check runner health on bms-4.

Graph View