Playbook: Atrax Credential Rotation

Covers: ATRAX_AUTH_STRING · ATRAX_USERNAME · ATRAX_PASSWORD · ATRAX_CLIENT_ID · ATRAX_CLIENT_SECRET

Overview

Atrax is the external fleet management system used by Ecotrans. All credentials are managed via a UI-only web portal — no public API for credential rotation.

As of #1713 (2026-08-05), Atrax is accessed via a dedicated “p24-infra” service account — not a shared/personal login. ATRAX_USERNAME / ATRAX_PASSWORD were rotated to this dedicated account’s credentials on that date.

All five keys appear in two SOPS files and must be updated atomically:

SOPS file	Keys present
`secrets/monitoring.env.sops`	All 5 keys
`secrets/n8n-bms4.env.sops`	All 5 keys

Corrected 2026-08-05 (#1713): this table previously listed secrets/vps-h1.env.sops as a third file holding all 5 keys. That stopped being true on 2026-07-05 (commit 7fb1f7be, post-WAHA-decommission cleanup) — ATRAX_USERNAME, ATRAX_PASSWORD, ATRAX_CLIENT_ID, ATRAX_CLIENT_SECRET were deliberately removed from vps-h1.env.sops because no vps-h1 service consumes them. The playbook was never updated to match. The live consumer is bms-4/docker-compose.yml (n8n main + 3 workers), which pulls these from n8n-bms4.env.sops via secrets-sync.yml. Do not re-add these keys to vps-h1.env.sops in a future rotation — verify current consumers with grep -rl ATRAX_ --include=docker-compose*.yml . before trusting this table if it drifts again.

Tier: ❌ Tier 3 Manual — human must obtain new credentials from the Atrax portal.

Detection — how an expired Atrax credential surfaces

Symptom: the Prometheus alert N8nWorkflowPersistentlyFailing fires for an Atrax-connected n8n workflow (e.g. atrax, kravag-scheduled-fleet-updates, id CCx9UMdphmGficDX). Every scheduled run fails at the OAuth token step.

The failing node is an HTTP Request node (usually named get_token) doing a password grant to https://tronik.atrax4.com/oauth/token. When the credentials are stale, Atrax returns:

HTTP 400  {"error":"invalid_grant","error_description":"Bad credentials"}

Confirm the root cause (read-only, run on bms-4)

Pull the error from the n8n Postgres directly (no API key needed — n8n + its DB run on bms-4):

# execution status history for the workflow
docker exec bms-4-n8n-postgres-1 sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U n8n -d n8n -tAF"|" -c \
  "SELECT id,status,\"startedAt\" FROM execution_entity \
   WHERE \"workflowId\"='"'"'CCx9UMdphmGficDX'"'"' ORDER BY \"startedAt\" DESC LIMIT 10;"'
 
# error JSON of the latest failed execution (look for invalid_grant / Bad credentials)
docker exec bms-4-n8n-postgres-1 sh -c 'PGPASSWORD=$POSTGRES_PASSWORD psql -U n8n -d n8n -tAc \
  "SELECT substring(data,1,4000) FROM execution_data WHERE \"executionId\"=<LATEST_ERROR_ID>;"'

status='crashed' rows with clustered stoppedAt values are restart-reaped artifacts (see n8n/n8n-bms4-worker-crash-storm.md) — the genuine functional failure is status='error'.

Rule out a deploy-sync / wrong-file issue before declaring a rotation. Hash-compare the deployed value against every SOPS source — never print the value:

c=$(docker exec bms-4-n8n-1 sh -c 'printf %s "$ATRAX_PASSWORD" | sha256sum' | cut -c1-12)
echo "container: $c"
export SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt
for f in secrets/n8n-bms4.env.sops secrets/monitoring.env.sops; do
  v=$(sops -d --input-type dotenv --output-type dotenv "$f" 2>/dev/null | grep -m1 '^ATRAX_PASSWORD=' | cut -d= -f2- | tr -d '"')
  printf '%s %s\n' "$f" "$(printf %s "$v" | sha256sum | cut -c1-12)"; unset v
done

Container hash ≠ its SOPS source → deploy-sync issue, re-run secrets-sync.yml (no rotation needed).
Both SOPS hashes must match each other (the atomicity rule below — see Overview for why vps-h1.env.sops is no longer part of this check). If they diverge, a prior rotation was applied non-atomically — the live consumer may be using whichever file feeds it.
Container matches source but Atrax still rejects it → the stored credential is genuinely expired/revoked → proceed with the Tier 3 rotation below.

⚠️ Do NOT brute-test multiple stored passwords against the live endpoint — Atrax may lock the account. One diagnostic token request to reproduce invalid_grant is enough; the workflow itself is already retrying on schedule.

When to rotate

On exposure: immediately (< 1 hour)
Scheduled: when Atrax sends an expiry notice, or annually

Pre-rotation checklist

Coordinate timing — GPS sync n8n workflow (bms-4) will fail while old creds are invalid
Verify current credentials still authenticate against Atrax (log into portal or test GPS sync)

Rotation steps

1. Obtain new credentials (human required)

Log in to the Atrax admin portal (URL stored near ATRAX_* keys in SOPS)
Navigate to credential management / OAuth application settings
Rotate ATRAX_CLIENT_ID / ATRAX_CLIENT_SECRET and/or ATRAX_AUTH_STRING
Change ATRAX_USERNAME / ATRAX_PASSWORD if password rotation is required
Store new values temporarily in .env.local or pass via $env: variables — never in chat

2. Update both SOPS files atomically (Claude handles)

Follow docs/playbooks/sops-edit-operations.md for each file. Prefer scripts/sops-set.ps1 -Pairs/-PairsFile (atomic write + built-in canary + recipient-count verification) over the hand-rolled pattern below.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
 
# Repeat the edit pattern for each of the 2 files:
# secrets\monitoring.env.sops, secrets\n8n-bms4.env.sops
# (NOT secrets\vps-h1.env.sops — these keys were removed there 2026-07-05, #1713 correction, see Overview)
# Keys to update in each: ATRAX_AUTH_STRING, ATRAX_USERNAME, ATRAX_PASSWORD,
#                          ATRAX_CLIENT_ID, ATRAX_CLIENT_SECRET
 
# Canary both before any git add:
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
sops --decrypt --input-type dotenv --output-type dotenv secrets\n8n-bms4.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS canary failed — do NOT commit" }

Both files must pass the canary before any git add. Also worth a quick hash-compare of the rotated key(s) across both files (never print the value — sha256[:12] only) to confirm atomicity, same pattern as atrax-credential-rotation.md §Confirm the root cause step 2 above.

3. Commit and deploy

git add secrets/monitoring.env.sops secrets/n8n-bms4.env.sops
git commit -m "chore: rotate ATRAX credentials $(Get-Date -Format 'yyyy-MM-dd')
 
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>"
git push
# secrets-sync.yml fires on merge → deploys to vps-i1 and bms-4

4. Restart consumers

secrets-sync.yml restarts affected containers automatically on merge; these are the manual fallback. Headscale was removed 2026-07-08 — connect over the public IPs (SSH by key only).

# bms-4 (54.36.123.110) — n8n compose lives at /opt/p24-infra/bms-4 — the only live ATRAX consumer
ssh root@54.36.123.110 "cd /opt/p24-infra/bms-4 && docker compose restart n8n"

5. Verify

Trigger a test GPS sync or run the Atrax-connected n8n workflow manually — confirm no auth errors.

6. Log rotation

| YYYY-MM-DD HH:MM UTC | #ISSUE | ATRAX_AUTH_STRING | reason | radieu | monitoring + n8n-bms4.env.sops |

Recovery

If one SOPS file was updated but the other was not: the old credentials remain valid at the Atrax provider until explicitly revoked there — finish updating both files before removing old credentials from the portal.

References

secret-rotation-access-matrix.md — Tier 3 classification, cross-file duplication table
docs/playbooks/sops-edit-operations.md — SOPS write pattern (Windows)

p24-infra Docs

Explorer

atrax-credential-rotation