traccar-admin-key-rotation

Playbook: TRACCAR_PASSWORD / TRACCAR_ADMIN_KEY / TRACCAR_FORWARD_TOKEN / TRACCAR_GW_ADMIN_KEY Rotation

Service: Traccar 6.x GPS tracking server (vps-i1) Secrets: TRACCAR_PASSWORD, TRACCAR_ADMIN_KEY, TRACCAR_FORWARD_TOKEN, TRACCAR_GW_ADMIN_KEY Rotation frequency: 180 days or on suspected exposure Last rotated: 2026-07-05 (TRACCAR_PASSWORD), 2026-06-27 (TRACCAR_ADMIN_KEY), 2026-06-26 (FORWARD_TOKEN + GW_ADMIN_KEY) Next due: 2026-12-27 (2027-01-01 for TRACCAR_PASSWORD)

Accuracy notice (2026-08-02, #5090). Only the TRACCAR_PASSWORD section below has been re-verified against the live Traccar 6.14.4 deployment. The TRACCAR_ADMIN_KEY sections (Methods A and B) describe a tc_users.token column that does not exist on 6.14.4 — the 2026-07-05 rotation-log entry records that live TRACCAR_ADMIN_KEY rotation was blocked for exactly this reason and flags those sections as outdated. Treat Method A/B as unverified until someone reworks them; they are left in place unchanged because #5090 scoped only the password procedure. See docs/secrets-rotation-log.md (2026-07-05 entries).

---

What these secrets do

Key	Length	Purpose	Who uses it
TRACCAR_PASSWORD	24–32 chars	Traccar app-admin login password. Was overloaded until 2026-08-02 — read the dual-consumer warning below before touching it	rotate_traccar_password(), services/traccar/scripts/*.py (no longer backup-ionos.sh — that uses MYSQL_PASSWORD)
TRACCAR_ADMIN_KEY	64 hex chars	Admin API session token for Traccar REST API	cost-exporter, ad-hoc API calls
TRACCAR_FORWARD_TOKEN	64 hex chars	Bearer token Traccar sends to traccar-gw CF Worker on each GPS position	traccar-gw CF Worker — validates incoming GPS positions
TRACCAR_GW_ADMIN_KEY	96 hex chars	X-Admin-Key header for admin endpoints on traccar-gw CF Worker	n8n workflows calling traccar-gw /admin/* routes

Note: All three keys are different values even though they are related. TRACCAR_ADMIN_KEY and TRACCAR_FORWARD_TOKEN are both 64-char hex strings but are NOT the same value. TRACCAR_GW_ADMIN_KEY is 96 chars and controls the CF Worker admin endpoints, not Traccar itself.

---

Where the secrets are stored

Location	Keys stored	How to update
secrets/monitoring.env.sops	All four	SOPS write pattern (see §Updating SOPS)
traccar-gw CF Worker secrets	FORWARD_TOKEN, ADMIN_KEY	wrangler secret put FORWARD_TOKEN + wrangler secret put ADMIN_KEY
Traccar runtime DB	TRACCAR_PASSWORD (as PBKDF2 hash)	tc_users.hashedPassword + tc_users.salt — never plaintext
Traccar runtime DB	TRACCAR_ADMIN_KEY value	Documented as the token column in tc_users — column absent on 6.14.4, see accuracy notice
Traccar traccar.xml	MySQL traccar DB user password (database.password)	Managed by generate-config.sh; XML-entity-encoded — see mysql-root-password-reset.md
GH Secrets (radieu/p24-infra)	TRACCAR_PASSWORD	gh secret set — written by rotate_traccar_password()

---

Critical context — Traccar 6.x API token behaviour

• POST /api/session/token returns HTTP 400 in Traccar 6.x when called with a Bearer token (confirmed 2026-06-27, issue #1636). This endpoint only accepts a valid session cookie from a prior POST /api/session login — it does NOT work with an existing API token directly.
• API tokens are stored in the token column of the tc_users MySQL table.
• Traccar caches tokens in memory — after updating the DB column, restart the traccar container for the new token to take effect.
• TRACCAR_FORWARD_TOKEN is configured in traccar.xml as server.forwardUrl’s Bearer token. Changing this token requires updating traccar.xml AND restarting Traccar.
• TRACCAR_GW_ADMIN_KEY is a CF Worker secret — it is independent of Traccar and can be rotated without restarting anything on vps-i1.

---

Rotation — Method A: Web UI (preferred for TRACCAR_ADMIN_KEY)

Use this when you have working web UI access to Traccar. Tokens generated in the UI are immediately active without a restart.

Step 1 — Log in to Traccar web UI
  URL: https://traccar.vps-i1.infra.zintegrowana.online
  User: admin
  Password: from TRACCAR_PASSWORD in secrets/monitoring.env.sops

Step 2 — Generate new API token
  Settings (gear icon, top-right) → Settings → API tokens → + Add
  Name: radieu-admin-YYYY-MM-DD
  No expiry (leave expiration blank)
  → Click Save → Copy the token value (shown only once, 64 hex chars)

Step 3 — Update SOPS (see §Updating SOPS below)
  Keys to update: TRACCAR_ADMIN_KEY

Step 4 — Append to docs/secrets-rotation-log.md (newest first, after header)
  | YYYY-MM-DD | TRACCAR_ADMIN_KEY | reason | rotator | SOPS monitoring.env.sops |

Step 5 — Log the operation to infra_operations audit log
  source /opt/p24-infra/scripts/lib/log_op.sh
  log_op "radieu" "credential_rotation" "TRACCAR_ADMIN_KEY" "success" \
    "Scheduled 180d rotation — new token via Traccar web UI" "traccar"

---

Rotation — Method B: DB-direct (fallback for TRACCAR_ADMIN_KEY)

Use this when the web UI is inaccessible, or when rotating from a script without human interaction. The DB-direct method writes the token directly into the MySQL tc_users table.

# Step 1 — SSH to vps-i1
ssh root@217.154.82.162
 
# Step 2 — Read MySQL root password safely (never echo)
MYSQL_ROOT_PASSWORD=$(grep MYSQL_ROOT_PASSWORD /root/traccar/.env | cut -d= -f2-)
 
# Step 3 — Generate a cryptographically random 64-char hex token
NEW_TOKEN=$(openssl rand -hex 32)
echo "Token length: ${#NEW_TOKEN}"  # Expected: 64
# DO NOT echo the token value itself
 
# Step 4 — Write the new token to the DB (updates first admin user found)
docker exec traccar-db mysql -u root -p"$MYSQL_ROOT_PASSWORD" traccar \
  -e "UPDATE tc_users SET token='$NEW_TOKEN' WHERE administrator=1 ORDER BY id LIMIT 1;"
 
# Step 5 — Verify the row was updated (check count only, not value)
docker exec traccar-db mysql -u root -p"$MYSQL_ROOT_PASSWORD" traccar \
  -e "SELECT COUNT(*) AS updated FROM tc_users WHERE token IS NOT NULL AND administrator=1;"
# Expected: 1
 
# Step 6 — Restart Traccar to flush the in-memory token cache
docker restart traccar
# Wait ~20s for startup
sleep 25
 
# Step 7 — Verify the new token works
HTTP_CODE=$(curl -s -o /dev/null -w "%{http_code}" \
  -H "Authorization: Bearer $NEW_TOKEN" \
  http://localhost:8082/api/devices)
echo "HTTP response: $HTTP_CODE"
# Expected: 200 (returns JSON array of devices)
 
# Step 8 — Store the token value safely for SOPS update
# IMPORTANT: write to a temp file on the server, never paste in chat
echo "$NEW_TOKEN" > /tmp/traccar_new_token.txt
chmod 600 /tmp/traccar_new_token.txt
 
# Step 9 — Clear the variable
unset NEW_TOKEN MYSQL_ROOT_PASSWORD
 
# Step 10 — On LOCAL (Windows PowerShell): update SOPS
# Read the token from the server:
#   ssh root@217.154.82.162 "cat /tmp/traccar_new_token.txt"
# Then follow §Updating SOPS below, substituting the token into TRACCAR_ADMIN_KEY
# After SOPS update, delete the temp file:
#   ssh root@217.154.82.162 "rm -f /tmp/traccar_new_token.txt"

---

Rotation — TRACCAR_FORWARD_TOKEN

TRACCAR_FORWARD_TOKEN is both in SOPS and in traccar.xml on the server. It must be updated in three places: SOPS, traccar.xml, and the traccar-gw CF Worker secret.

# Step 1 — SSH to vps-i1
ssh root@217.154.82.162
 
# Step 2 — Generate new forward token
NEW_FORWARD=$(openssl rand -hex 32)
echo "Token length: ${#NEW_FORWARD}"  # Expected: 64
 
# Step 3 — Update traccar.xml (replace the forwardUrl Bearer token)
# View current forwardUrl line (safe — shows URL structure, not token in isolation)
grep "forwardUrl" /root/traccar/traccar.xml
 
# Perform substitution:
OLD_FORWARD=$(grep TRACCAR_FORWARD_TOKEN /root/traccar/.env | cut -d= -f2-)
sed -i "s/$OLD_FORWARD/$NEW_FORWARD/" /root/traccar/traccar.xml
# Verify the line changed (show only the structure, NOT the value):
grep -c "forwardUrl" /root/traccar/traccar.xml  # Must still be 1
 
# Step 4 — Update /root/traccar/.env on the server
sed -i "s/TRACCAR_FORWARD_TOKEN=.*/TRACCAR_FORWARD_TOKEN=$NEW_FORWARD/" /root/traccar/.env
 
# Step 5 — Restart Traccar to pick up new traccar.xml
docker restart traccar
sleep 25
 
# Step 6 — Update traccar-gw CF Worker secret (on local Windows PowerShell)
# From C:\code_2026\p24-infra\infra-src\traccar-gw:
#   $env:NEW_FORWARD = "<read from server>"
#   Write-Output $env:NEW_FORWARD | wrangler secret put FORWARD_TOKEN
#   $env:NEW_FORWARD = ""
 
# Step 7 — Store token for SOPS update, then follow §Updating SOPS below
echo "$NEW_FORWARD" > /tmp/traccar_new_forward.txt
chmod 600 /tmp/traccar_new_forward.txt
unset NEW_FORWARD OLD_FORWARD

---

Rotation — TRACCAR_GW_ADMIN_KEY

TRACCAR_GW_ADMIN_KEY is the X-Admin-Key for the traccar-gw CF Worker admin endpoints. No Traccar restart needed — the CF Worker reads secrets from Cloudflare at runtime.

# Step 1 — Generate new admin key (96 hex chars = 48 random bytes)
NEW_GW_ADMIN=$(openssl rand -hex 48)
echo "Key length: ${#NEW_GW_ADMIN}"  # Expected: 96
 
# Step 2 — Update CF Worker secret
# Run from C:\code_2026\p24-infra\infra-src\traccar-gw (local Windows PowerShell):
#   $env:NEW_GW_ADMIN = "<value>"
#   Write-Output $env:NEW_GW_ADMIN | wrangler secret put ADMIN_KEY
#   $env:NEW_GW_ADMIN = ""
 
# Step 3 — Update SOPS (see §Updating SOPS)
# Key to update: TRACCAR_GW_ADMIN_KEY
 
# Step 4 — Verify the new key works (from vps-i1 or local)
# Check HTTP 200 from a CF Worker admin endpoint:
#   curl -s -o /dev/null -w "%{http_code}" \
#     -H "X-Admin-Key: $NEW_GW_ADMIN" \
#     https://traccar-gw.infra.zintegrowana.online/admin/status
# Expected: 200 or 404 (not 401/403)
 
unset NEW_GW_ADMIN

---

Rotation — TRACCAR_PASSWORD

TRACCAR_PASSWORD is the Traccar admin user’s login password — the credential used for POST /api/session (web UI + REST API), not an API token. It is the only Traccar secret that is input to authentication rather than a bearer credential, which is why it needs its own procedure.

Canonical automation: rotate_traccar_password() in scripts/rotate-credentials.py (dispatch table key TRACCAR_PASSWORD). Access-matrix row: secret-rotation-access-matrix.md.

✅ Prerequisite — TRACCAR_ADMIN_EMAIL must exist in SOPS (satisfied since #5099)

rotate_traccar_password() authenticates with TRACCAR_ADMIN_EMAIL + TRACCAR_PASSWORD before it can change anything. If either is absent the handler hits this branch and returns False → human-action issue, by design:

if not email or not old_pass:
    print("  TRACCAR_ADMIN_EMAIL / TRACCAR_PASSWORD missing — cannot authenticate to Traccar API")
    return False

TRACCAR_ADMIN_EMAIL landed in secrets/monitoring.env.sops in 9f3676f (#5099/#5111), so this prerequisite is met and Method A is available. The value is the Traccar admin user (id=1). It was historically absent — it existed only in the developer’s local .env.local under the different name TRACCAR_USER — which is why every rotation attempt before 2026-08-02 fell through to human-action (#5090). Confirm it is still present before rotating:

sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops \
  | grep -c '^TRACCAR_ADMIN_EMAIL='   # expect 1 — never print the value

⚠️ Dual-consumer warning — TRACCAR_PASSWORD was overloaded (split since #5099)

Status 2026-08-02 — the overload is fully resolved, structurally and by value.

1. Split (#5099/#5111, 9f3676f): MYSQL_PASSWORD is its own key in monitoring.env.sops.
2. Divergence (#3352, 2026-08-02): the app-side TRACCAR_PASSWORD was rotated to a fresh 48-char value, so the two keys now carry different secrets. The post-rotation Probe 2 stayed OK, confirming the DB side was untouched.
3. De-aliasing (#5098): backup-common.sh no longer reads TRACCAR_PASSWORD at all. The old ${MYSQL_PASSWORD:-${TRACCAR_PASSWORD}} fallback was removed rather than merely warned about — now that the values differ it could only ever supply a wrong password. A missing MYSQL_PASSWORD logs a loud ERROR and backup-ionos.sh’s : "${MYSQL_PASSWORD:?…}" guard aborts the run (pushing backup_last_failure_timestamp), instead of failing silently 24h later. scripts/tests/test_backup_common_mysql_password.py fails if the alias is ever reinstated.

The section is kept because the shape of the hazard is worth remembering: it is still possible to reintroduce the overload by hand — e.g. the mysql-root-password-reset.md drift repair writing a MySQL password into TRACCAR_PASSWORD. The pre-flight below stays mandatory, and the post-rotation Probe 2 re-check remains the check that catches a regression.

This key name historically served two unrelated credentials, and rotating one does not rotate the other. Read this before touching the value.

Consumer	What it actually needs	Path
Traccar app login (POST /api/session, PUT /api/users/{id})	Traccar admin user password, hashed into tc_users	rotate_traccar_password(), services/traccar/scripts/*.py, docs/traccar-api-testing.md
Nightly vps-i1 backup (mysqldump -u traccar)	MySQL traccar DB user password, stored in traccar.xml database.password	backup-common.sh (reads MYSQL_PASSWORD only — no TRACCAR_PASSWORD fallback since #5098) → backup-ionos.sh

MYSQL_PASSWORD is now a key in monitoring.env.sops (alongside MYSQL_ROOT_PASSWORD), so the backup path resolves it directly and there is no fallback to fall through to. Note that backup-common.sh reads $SOPS_SECRETS_FILE (default /opt/p24-infra/secrets/monitoring.env.sops) directly, not a deployed .env — so the split took effect as soon as the SOPS change merged, without waiting on secrets-sync.yml. Before #5099 this key was absent and the fallback applied unconditionally, which is the break described below.

Consequence (historical — this is what the split and #5098 de-aliasing removed): a successful REST-API rotation changed only the Traccar app-user password, wrote that new value to SOPS, and thereby silently broke the nightly mysqldump — surfacing later as a BackupStale alert, not as a rotation failure. The reverse still holds and is not automatically prevented: the mysql-root-password-reset.md §TRACCAR_PASSWORD drift repair writes the MySQL password into TRACCAR_PASSWORD, which then makes POST /api/session return 401. That is the most likely explanation for the 401 recorded on 2026-07-05 and again on 2026-08-02 (#5090).

Do not rotate until the pre-flight below tells you which credential the current value is.

Step 0 — Pre-flight: identify what the current value is

Run both probes against the same SOPS value. Exactly one should pass; the result decides your path.

# On vps-i1. Reads the value into a variable — never echoed. (secret-manager or sys-admin role.)
ssh root@217.154.82.162 '
  PW=$(sops -d --input-type dotenv --output-type dotenv \
    /opt/p24-infra/secrets/monitoring.env.sops | grep "^TRACCAR_PASSWORD=" | cut -d= -f2-)
  EMAIL=$(sops -d --input-type dotenv --output-type dotenv \
    /opt/p24-infra/secrets/monitoring.env.sops | grep "^TRACCAR_ADMIN_EMAIL=" | cut -d= -f2-)
 
  # Probe 1 — Traccar app login (expect 200 if this is the app-admin password)
  echo -n "app login: "
  curl -s -o /dev/null -w "%{http_code}\n" -X POST http://localhost:8082/api/session \
    --data-urlencode "email=${EMAIL}" --data-urlencode "password=${PW}"
 
  # Probe 2 — MySQL traccar user (expect OK if this is the DB password)
  echo -n "mysql user: "
  docker exec -e MYSQL_PWD="$PW" traccar-db mysqldump --single-transaction \
    --no-tablespaces -u traccar traccar --where="1 LIMIT 1" >/dev/null 2>&1 \
    && echo OK || echo REJECTED
 
  unset PW EMAIL
'

Probe 1	Probe 2	Meaning	Do this
200	REJECTED	Value is the app-admin password (the intended meaning)	Proceed to Method A
401	OK	Value is the MySQL password — app login is desynced	Split the keys first (see below), then Method C
401	REJECTED	Value matches neither — fully desynced	Method C (web UI reset), then re-run pre-flight
200	OK	Both credentials genuinely share one value	Rotate both sides together, or split first

Splitting the keys was the durable fix — **done in 5099 (9f3676f): monitoring.env.sops carries a distinct MYSQL_PASSWORD holding the traccar DB-user password. The consumer side followed in #5098: backup-common.sh reads MYSQL_PASSWORD and nothing else — there is no TRACCAR_PASSWORD fallback left to fall back to. If you ever need to re-derive that DB password, extract it from traccar.xml with Python’s XML parser — never grep, which returns & undecoded and produces a wrong value (mysql-root-password-reset.md §Prevention).

Expected pre-flight result today: row 1 (200 / REJECTED). Both keys held the same value up to and including the #3352 rotation, which made row 4 the expected result at that time; that rotation gave TRACCAR_PASSWORD a fresh distinct app-admin value, so Probe 2 — which tests the Traccar value against the MySQL user — now correctly returns REJECTED. A row-4 (200/OK) result is no longer benign: it means the two keys have been re-coupled to one value and the overload is back. Rotating TRACCAR_PASSWORD alone remains correct — just re-run Probe 2 afterwards, against MYSQL_PASSWORD, to confirm the DB side is untouched.

Method A — Autonomous, via the Traccar REST API (preferred)

Available only once the prerequisite is met and pre-flight Probe 1 returns 200. This is what rotate_traccar_password() does; run the handler rather than reproducing it by hand.

rotate-credentials.py takes no CLI flags — it is configured entirely through environment variables (DRY_RUN, FORCE_ALL, ONLY_SERVICES). There is no --only option; passing one is silently ignored and rotates whatever else happens to be due.

FORCE_ALL=true is required alongside ONLY_SERVICES whenever the credential is not currently past next_due — and also to override an auto_rotate=false DB flag for a service that does have a registered rotator (see the force_auto branch in main()).

# On the rotation host, with monitoring SOPS env loaded (secret-manager role).
# Dry run first — prints the plan without touching Traccar, SOPS, or GH Secrets.
DRY_RUN=true ONLY_SERVICES=TRACCAR_PASSWORD FORCE_ALL=true python3 scripts/rotate-credentials.py
 
# Real rotation
DRY_RUN=false ONLY_SERVICES=TRACCAR_PASSWORD FORCE_ALL=true python3 scripts/rotate-credentials.py

Do not source the decrypted SOPS output to load that env (#5058) — a value containing an unquoted & is backgrounded by bash and its job-control line prints KEY=value to the transcript. sops exec-env also does not work here: it infers the input type from the file extension and rejects .env.sops (flag provided but not defined: -input-type). Inject the decrypted keys into the child process’s environment instead, without ever printing them.

Run this from a dedicated worktree/branch, not the shared /opt/p24-infra checkout. sops_update_key() does git add/commit/push in the repo root that contains the script, and _ensure_rotation_branch() only creates a branch when it finds itself on main — so running it from the shared checkout leaves that checkout on a rotation branch and trips the next worker’s stale-checkout gate.

Sequence the handler performs, in order:

1. POST /api/session with TRACCAR_ADMIN_EMAIL + current TRACCAR_PASSWORD → session cookie
   • the admin user object (id, historically 1).
2. Generates a new 48-char password, sets user['password'], and PUT /api/users/{id} with the full user object. Traccar re-hashes it (PBKDF2) server-side.
3. gh_secret_set('TRACCAR_PASSWORD', …) → GH Secret.
4. sops_update_key('secrets/monitoring.env.sops', 'TRACCAR_PASSWORD', …).
5. supabase_update_rotation(...) → advances next_due in dev_r_services.

Any non-2xx at step 1 or 2 returns False before SOPS is touched — fail-closed, so the live server and SOPS never diverge. Do not “fix” a failure by writing a new password into SOPS first; that is precisely the desync this playbook exists to prevent.

Method B — DB-direct PBKDF2 (fallback when the API is reachable but auth fails)

Used successfully on 2026-07-05. Traccar 6.14.4 hashes passwords with PBKDF2WithHmacSHA1, 1000 iterations, 24-byte output, stored as tc_users.hashedPassword with a per-user tc_users.salt. Write both columns together or login breaks.

ssh root@217.154.82.162   # PLAYBOOK: traccar-admin-key-rotation.md
 
MYSQL_ROOT_PASSWORD=$(grep MYSQL_ROOT_PASSWORD /root/traccar/.env | cut -d= -f2-)
 
# Confirm the admin user id and the column names actually present on this version
docker exec traccar-db mysql -u root -p"$MYSQL_ROOT_PASSWORD" traccar \
  -e "SELECT id, name, email, administrator FROM tc_users WHERE administrator=1;"
docker exec traccar-db mysql -u root -p"$MYSQL_ROOT_PASSWORD" traccar \
  -e "SHOW COLUMNS FROM tc_users LIKE '%assword%';"

Generate the hash off-box and write it in (the plaintext never reaches argv or the shell history — it is passed through the environment):

NEW_PASS=$(openssl rand -base64 24 | tr -d '/+=' | cut -c1-24)
 
# salt + hash, computed the way Traccar 6.14.4 does it
read -r SALT_HEX HASH_HEX <<<"$(NEW_PASS="$NEW_PASS" python3 - <<'PY'
import hashlib, os
salt = os.urandom(16)
pw = os.environ['NEW_PASS'].encode()
print(salt.hex(), hashlib.pbkdf2_hmac('sha1', pw, salt, 1000, 24).hex())
PY
)"
 
docker exec traccar-db mysql -u root -p"$MYSQL_ROOT_PASSWORD" traccar \
  -e "UPDATE tc_users SET hashedPassword='$HASH_HEX', salt='$SALT_HEX' WHERE id=1;"
 
# Traccar caches user records — restart before verifying
docker restart traccar && sleep 25
 
# Verify (status code only)
curl -s -o /dev/null -w "app login: %{http_code}\n" -X POST http://localhost:8082/api/session \
  --data-urlencode "email=<admin email>" --data-urlencode "password=$NEW_PASS"
# Expected: 200
 
# Hand off to SOPS via a mode-600 temp file — never paste the value into chat
umask 077; printf '%s' "$NEW_PASS" > /tmp/traccar_new_password.txt
unset NEW_PASS SALT_HEX HASH_HEX MYSQL_ROOT_PASSWORD

Then follow §Updating SOPS with key TRACCAR_PASSWORD, and delete /tmp/traccar_new_password.txt afterwards.

Confirm hashedPassword/salt against the SHOW COLUMNS output before running the UPDATE — the schema has changed across Traccar 6.x releases, and this playbook already carries one stale column reference (tc_users.token, see the accuracy notice at the top).

Method C — Manual web UI reset (last resort, human-action)

Use when both pre-flight probes fail, i.e. no known credential can authenticate.

1. Log in to https://traccar.vps-i1.infra.zintegrowana.online
   If the admin password is unknown, reset it via Method B (root MySQL access does not
   require knowing the current password).
2. Settings (gear, top-right) → Users → select the admin user → Password → set a new value
3. Save, then log out and log back in to confirm it took effect
4. Record the value into SOPS via §Updating SOPS (key: TRACCAR_PASSWORD)

Because this path needs interactive UI access it cannot be executed by a worker — it is genuinely human-action. The access matrix classifies TRACCAR_PASSWORD as Tier 1 (API-rotatable), which is correct only once the prerequisite above is satisfied; before that, Method C is the only route.

Post-rotation distribution — all of these, or the rotation is incomplete

Target	How	Verify
secrets/monitoring.env.sops	§Updating SOPS	canary decrypt exits 0
GH Secret TRACCAR_PASSWORD	gh secret set (handler does this)	gh secret list --repo radieu/p24-infra | grep TRACCAR_PASSWORD
vps-i1 /opt/p24-infra/monitoring/.env	secrets-sync.yml on merge to main	grep -q "^TRACCAR_PASSWORD=" …/.env && echo exists
vps-i1 /root/traccar/.env	secrets-sync.yml	as above
dev_r_services.next_due	supabase_update_rotation()	row shows the new last_rotated
docs/secrets-rotation-log.md	append newest-first, after the header	entry present in the merged PR
Nightly backup path	re-run the Probe 2 mysqldump test	OK — catches the dual-consumer break immediately

The last row is not optional. It is the only check that catches the backup regression described in the dual-consumer warning, and it fails silently for up to 24h otherwise.

Known failure modes

Symptom	Cause	Action
Handler logs TRACCAR_ADMIN_EMAIL / TRACCAR_PASSWORD missing	Prerequisite not met — key absent from SOPS	secret-manager adds TRACCAR_ADMIN_EMAIL; #5090
POST /api/session → 401 for every candidate identity	SOPS holds the MySQL password, not the app password	Pre-flight table row 2 — split the keys, then Method C
Rotation reports success, BackupStale fires within 24h	Dual-consumer break — DB user password unchanged (should be impossible since 5111; means the split was undone)	Restore the MYSQL_PASSWORD key and the no-fallback guard; repair per mysql-root-password-reset.md
mysqldump denied but GPS data still flowing	Same as above — app path healthy, DB path stale	mysql-root-password-reset.md §TRACCAR_PASSWORD drift
Login still fails right after a Method B UPDATE	Traccar user cache not flushed	docker restart traccar, wait 25s, retry
UPDATE reports 0 rows changed	Wrong id, or column named differently on this version	Re-run the SELECT + SHOW COLUMNS probes above

---

Updating SOPS (monitoring.env.sops)

Run on local Windows PowerShell after collecting the new token value(s):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
Set-Location C:\code_2026\p24-infra
 
# 1. Decrypt to LF-only plaintext temp file (MUST be inside secrets/ to match path_regex)
$plain = sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops
$tempPath = "$PWD\secrets\monitoring-edit.env.sops"
[System.IO.File]::WriteAllText($tempPath, ($plain -join "`n") + "`n",
  [System.Text.UTF8Encoding]::new($false))
 
# 2. Update the key(s) — repeat for each rotated key
#    Replace TRACCAR_ADMIN_KEY, TRACCAR_FORWARD_TOKEN, or TRACCAR_GW_ADMIN_KEY as needed
#    Example for TRACCAR_ADMIN_KEY — store new token in env var first, never print it:
$env:NEW_TOKEN = (ssh root@217.154.82.162 "cat /tmp/traccar_new_token.txt")
$content = [System.IO.File]::ReadAllText($tempPath)
$content = $content -replace "(?m)^TRACCAR_ADMIN_KEY=.*", "TRACCAR_ADMIN_KEY=$env:NEW_TOKEN"
[System.IO.File]::WriteAllText($tempPath, $content, [System.Text.UTF8Encoding]::new($false))
$env:NEW_TOKEN = ""
 
# 3. Re-encrypt
$enc = sops --encrypt --input-type dotenv --output-type dotenv $tempPath
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring.env.sops", ($enc -join "`n") + "`n",
  [System.Text.UTF8Encoding]::new($false))
 
# 4. Canary — MANDATORY before git add
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "SOPS decrypt failed — do NOT commit. See docs/playbooks/sops-windows-crlf.md" }
Write-Host "Canary OK — safe to git add"
 
# 5. Clean up temp file and server temp files
[System.IO.File]::Delete($tempPath)
# ssh root@217.154.82.162 "rm -f /tmp/traccar_new_token.txt /tmp/traccar_new_forward.txt"

---

Triggering secrets-sync to deploy to vps-i1

After merging the SOPS update to dev or main, the secrets-sync.yml workflow auto-deploys /root/traccar/.env on vps-i1. Check it ran:

# From local (Windows PowerShell):
gh run list --workflow secrets-sync.yml --repo radieu/p24-infra --limit 3

If secrets-sync hasn’t run yet, trigger manually:

gh workflow run secrets-sync.yml --repo radieu/p24-infra

After secrets-sync completes, restart affected containers on vps-i1:

ssh root@217.154.82.162 "cd /root/traccar && docker compose restart traccar"

---

Verification after rotation

# 1. Verify SOPS contains the new key (presence check only — no value display)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
(sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\monitoring.env.sops `
  | Select-String "^TRACCAR_ADMIN_KEY=") -ne $null
# Expected: True
 
# 2. Verify TRACCAR_ADMIN_KEY works (HTTP status only — value stays in server env):
ssh root@217.154.82.162 '
  TOKEN=$(grep TRACCAR_ADMIN_KEY /root/traccar/.env 2>/dev/null | cut -d= -f2-)
  if [ -z "$TOKEN" ]; then
    TOKEN=$(grep TRACCAR_ADMIN_KEY /opt/p24-infra/monitoring/.env 2>/dev/null | cut -d= -f2-)
  fi
  HTTP=$(curl -s -o /dev/null -w "%{http_code}" \
    -H "Authorization: Bearer $TOKEN" http://localhost:8082/api/devices)
  echo "TRACCAR_ADMIN_KEY status: $HTTP"
  unset TOKEN
'
# Expected: TRACCAR_ADMIN_KEY status: 200
 
# 3. Verify Traccar is running and healthy after any restart:
ssh root@217.154.82.162 "docker inspect --format='{{.State.Status}}' traccar"
# Expected: running
ssh root@217.154.82.162 "curl -s -o /dev/null -w '%{http_code}' http://localhost:8082/api/server"
# Expected: 200

---

Escalation path

Symptom	Likely cause	Action
POST /api/session returns 401 for TRACCAR_PASSWORD	Login identity unknown, or SOPS value is the MySQL password	§Rotation — TRACCAR_PASSWORD → Step 0 pre-flight
Autonomous TRACCAR_PASSWORD rotation returns human-action	TRACCAR_ADMIN_EMAIL absent from SOPS	§Rotation — TRACCAR_PASSWORD → Prerequisite (#5090)
BackupStale alert after a Traccar password rotation	Dual-consumer break — the MYSQL_PASSWORD → TRACCAR_PASSWORD fallback was reintroduced, or the two keys were re-coupled to one value	§Rotation — TRACCAR_PASSWORD → Dual-consumer warning
ERROR: MYSQL_PASSWORD is unset after loading secrets (#5098) in /var/log/p24-backup.log	The key is missing from this host’s monitoring.env.sops — the backup aborts rather than using TRACCAR_PASSWORD	Re-sync/re-decrypt monitoring.env.sops; never substitute TRACCAR_PASSWORD
POST /api/session/token returns 400	Traccar 6.x does not accept Bearer token for this endpoint	Use DB-direct method (Method B) or web UI (Method A)
DB update returns “0 rows affected”	No admin user with administrator=1 found	Check: SELECT id, name, administrator FROM tc_users; — use the correct id in WHERE clause
HTTP 401 after token update	Token cache not flushed	docker restart traccar on vps-i1; wait 25s
Traccar fails to start after traccar.xml edit	Malformed XML or wrong token substitution	Restore traccar.xml from repo: cp /opt/p24-infra/services/traccar/traccar.xml /root/traccar/traccar.xml then re-run generate-config.sh
traccar-gw CF Worker returns 403 on GPS positions	FORWARD_TOKEN mismatch between Traccar and CF Worker	Ensure both Traccar (traccar.xml) and CF Worker secret have the same value
traccar-gw admin endpoints return 401	TRACCAR_GW_ADMIN_KEY not updated in CF Worker	Re-run wrangler secret put ADMIN_KEY from infra-src/traccar-gw/
secrets-sync failed to deploy	Workflow failed or not triggered	Check gh run list --workflow secrets-sync.yml; trigger manually if needed
GPS positions stop arriving in Supabase	FORWARD_TOKEN mismatch	Verify traccar.xml forwardUrl token matches CF Worker FORWARD_TOKEN secret

---

Prevention

• Rotation schedule: 180-day recurring reminder from last-rotated date.
• MYSQL_PASSWORD is split out of TRACCAR_PASSWORD — keep it that way (done). While backup-common.sh fell back to TRACCAR_PASSWORD for the MySQL traccar user, every app-password rotation was also a silent backup outage. The dedicated MYSQL_PASSWORD key in monitoring.env.sops (#5111) plus the removal of the fallback in backup-common.sh (#5098) removed the whole failure class; scripts/tests/test_backup_common_mysql_password.py guards the consumer side against a revert. Do not reintroduce a TRACCAR_PASSWORD fallback for MySQL — see §Rotation — TRACCAR_PASSWORD → Dual-consumer warning.
• Keep TRACCAR_ADMIN_EMAIL in SOPS. Without the login identity, TRACCAR_PASSWORD cannot be rotated by any automation and permanently degrades to human-action. Note the local .env.local name is TRACCAR_USER; the SOPS/automation name is TRACCAR_ADMIN_EMAIL.
• Always run the Step 0 pre-flight before rotating TRACCAR_PASSWORD — it is the only cheap way to detect an existing desync before a rotation compounds it.
• The credential-rotation.yml GH Actions workflow (Monday 06:00 UTC) opens a human-action issue when next_due passes. Confirm all three key names are in the workflow’s key list.
• Never use TRACCAR_ADMIN_KEY directly in n8n node parameters — always reference as a credential; see feedback_n8n_no_hardcoded_credentials.md.
• TRACCAR_FORWARD_TOKEN is embedded in the live traccar.xml on vps-i1 (/root/traccar/traccar.xml), which is NOT tracked in the repo with the real value. The repo copy at services/traccar/traccar.xml does not contain a forwardUrl entry — the live file is managed separately via generate-config.sh. After rotating, confirm the repo copy does NOT contain the real token value before committing.
• TRACCAR_GW_ADMIN_KEY has no TTL on the CF Worker side — rotate proactively and whenever any n8n workflow credential referencing it is modified.