Playbook: Mezmo Key Rotation

Related: docs/mezmo-operations.md, docs/secrets-rotation-log.md, issues #1897 / #1904 / #1911

Key types

Three separate key categories, different purposes:

Key	SOPS var	SOPS file	Used by
Service key (`sts_…`)	`P24_INFRA_MEZMO_SERVICE_KEY`	`monitoring.env.sops`	Mezmo management API (cost-exporter, scripts) — not per-server
Ingestion key (32 hex) — infra	per-server — see below	per-server file	logdna-agent on each log-shipping host
Ingestion key (32 hex) — app-level	per-app — see below	pinbox24-w3/w4 files	Pinbox24 backend’s own `logdna` SDK call, not the host-level `logdna-agent`

App-level ingestion keys — Pinbox24 W3/W4 (#3603)

Distinct from the per-server infra keys above: this is a 32-hex ingestion key consumed directly by the Pinbox24 app code, not by a host’s logdna-agent systemd/Docker process. app-backend/helperFunction/crudService.js calls require('logdna') / Logger.createLogger(process.env.LOGDNA_KEY, ...) — the SOPS var is the value fed to that SDK call.

App	SOPS var	SOPS file	Mezmo key name	Confirmed wired into running app?
W3 (v32-prod)	`V32_LOGDNA_KEY`	`pinbox24-w3.env.sops`	(legacy, pre-#3603 naming)	Confirmed — see #3603 investigation
W4 (v42-prod)	`V42_LOGDNA_KEY`	`pinbox24-w4.env.sops`	`pinbox24-w4-app`	Not confirmed — v42-prod app source wasn’t verified per #3603. Key is minted and stored in SOPS only; wiring `process.env.LOGDNA_KEY` into the running v42-prod container is a separate, unconfirmed follow-up.

Deliberately separate keys per app — never share one value across pinbox24-w3.env.sops and pinbox24-w4.env.sops. A shared value across multiple SOPS files is exactly the multi-file-sync failure mode 1904 eliminated for the per-server infra keys; the same reasoning applies here. Create + name each via the standard §B2 API pattern below (rename to something distinguishing it from the per-server infra keys and from the other app, e.g. pinbox24-w4-app).

Per-server ingestion keys (#1904)

Each log-shipping host has its own ingestion key. This removes the multi-file sync requirement that caused #1897 (a single shared key duplicated across three SOPS files; rotating one and missing another → 403 on the missed host).

Server	SOPS var	SOPS file	Consumer	Mezmo key name
vps-i1	`MEZMO_INGESTION_KEY_VPS_I1`	`monitoring.env.sops`	`monitoring/docker-compose.yml` → `mezmo-agent`	`p24-vps-i1`
vps-h1	`MEZMO_INGESTION_KEY_VPS_H1`	`vps-h1.env.sops`	`secrets-sync.yml` `sync-vps-h1` job → `/root/.env` + `/root/docker-compose.yml` (host-managed Docker `mezmo-agent`, project `root`, container `root-mezmo-agent-1`; corrected #5454 — no `/opt/mezmo-agent` exists on this host, #5055’s assumption by analogy with bms-1 was wrong)	`p24-vps-h1`
bms-1	`MEZMO_INGESTION_KEY_BMS1`	`bms-servers.env.sops`	`secrets-sync.yml` `sync-bms-1` job → `/opt/mezmo-agent/.env` (Docker `mezmo-agent`; corrected #1915 — bms-1 moved off the systemd `/etc/logdna.env` agent per #1425, see the note in §B4/B5 below)	`p24-bms-1`
bms-2	`MEZMO_INGESTION_KEY_BMS2`	`bms-servers.env.sops`	`/etc/logdna.env` (only if `logdna-agent` installed)	`p24-bms-2`
bms-3	`MEZMO_INGESTION_KEY_BMS3`	`bms-servers.env.sops`	`/etc/logdna.env` (only if `logdna-agent` installed)	`p24-bms-3`
(logical — rotation scripts, not a physical server)	`MEZMO_INGESTION_KEY_ROTATION`	`monitoring.env.sops`	`scripts/lib/sops-common.psm1`’s `Send-MezmoLog` (issue #3451) — SOPS rotation-script stage logging from windows-dev	`p24-rotation-scripts`

vps-h1 was missing from the original #1904 plan. It also ships logs via a Docker mezmo-agent reading ${MEZMO_INGESTION_KEY} from vps-h1.env.sops, so it needs its own key too.

Benefit: rotating one host’s key edits exactly one SOPS file and restarts exactly one agent — a failed or stale rotation degrades only that host, never the whole fleet.

Until the §C migration runs: the fleet still shares one MEZMO_INGESTION_KEY (monitoring.env.sops + vps-h1.env.sops) / LOGDNA_API_KEY (bms-servers.env.sops). While the shared key is still live, the legacy “rotate all files together” rule below (§B-legacy) applies.

#1915 status: the code side of Part C steps 3-4 has already landed (monitoring/docker-compose.yml and secrets-sync.yml’s sync-bms-1 / sync-vps-h1 jobs already read the per-host names, falling back to the shared key when a per-host key is absent — safe to merge/run at any point). Starting Part C now only requires steps 1-2 (host check + mint keys) and populating SOPS (step 3) — the compose/workflow edits it used to require are already in place. Step 6 (retire the shared key) is unaffected and still requires full per-host verification first.

Mezmo keys API — verified reference (2026-06-28, #1904)

Authenticate with the service key using Authorization: Token <key> (the servicekey: header is deprecated). Base https://api.mezmo.com.

Operation	Request	Notes
List	`GET /v1/config/keys`	`[{id, key, name, type, created}]`. `key` is the secret value — never print it.
Create ingestion key	`POST /v1/config/keys?type=ingestion`	`type` is a query param, not a body field (body `{"type":…}` → 400). Empty body OK. Returns the new key; auto-names it `ingestion-key-NNNN` — rename right after.
Rename	`PUT /v1/config/keys/{id}` body `{"name":"p24-vps-i1"}`	`Content-Type: application/json` required (body present).
Delete	`DELETE /v1/config/keys/{id}`	Send NO `Content-Type` header / no body. A JSON content-type header makes the server validate an empty payload → 400 `"value" is not allowed`. Correct form → 200 `{"deleted":true}`. `curl -X DELETE -H "Authorization: Token $K"` works; `requests`/`Invoke-RestMethod` with a `json=`/body arg do not — pass only the auth header.

List keys by name/type without exposing values (any host with the age key):

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
K=$(sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops | grep '^P24_INFRA_MEZMO_SERVICE_KEY=' | cut -d= -f2-)
curl -s -H "Authorization: Token $K" https://api.mezmo.com/v1/config/keys \
  | python3 -c "import json,sys;print(json.dumps([{'id':x['id'],'name':x.get('name'),'type':x.get('type')} for x in json.load(sys.stdin)]))"
unset K

Part A: Service Key Rotation

Triggers: cost-exporter 401/403, scripts/mezmo-manage.py API failure, 180-day schedule, suspected exposure.

⚠️ Rule out a transient Mezmo auth-backend blip BEFORE rotating on a 401 (#4486). A 401 NotAuthorized whose body reads "Token Validation Error: Unable to validate your servicekey at the moment" and appears as a short burst (a few seconds) that self-recovers is Mezmo’s token validation backend being briefly unavailable — the key is fine, do NOT rotate. Distinguish it from a genuinely expired/revoked key by validating the live key first (A1 below — GET /v1/config/view): 200 = key valid → transient upstream, no rotation; a sustained 401 where the live key also returns 401 → key really is dead → rotate. Both known occurrences of MezmoExporterApiErrors (dedup-key 44aa2e722c469d09) were transient upstream, self-recovered, no rotation: #4195 (HTTP 500 ServerError) and #4486 (HTTP 401 “…at the moment” burst, live key still 200). The exporter fails soft — it counts each error into mezmo_exporter_api_errors_total and returns []; the alert (increase(...[15m]) > 5) keeps firing for up to 15 min after recovery purely from the lookback window. Confirm clearance with increase(mezmo_exporter_api_errors_total[15m]) back below 5 rather than re-triggering on the lag.

A1 — Confirm problem

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$key = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
  Select-String "^P24_INFRA_MEZMO_SERVICE_KEY=").ToString().Split("=",2)[1]
Invoke-RestMethod -Uri "https://api.mezmo.com/v1/config/view" -Headers @{"Authorization"="Token $key"}
$key = ""

A2 — Generate new service key (dashboard only)

https://app.mezmo.com > Settings > Organization > API Keys > Create Service Key. (The service key — unlike ingestion keys — can only be created in the dashboard; the API ?type=ingestion form creates ingestion keys only.)

A3 — Update monitoring.env.sops

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$dec = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops) `
  -replace "^P24_INFRA_MEZMO_SERVICE_KEY=.*", "P24_INFRA_MEZMO_SERVICE_KEY=$env:NEW_KEY"
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring-edit.env.sops",
  ($dec -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\monitoring-edit.env.sops
[System.IO.File]::WriteAllText("$PWD\secrets\monitoring.env.sops",
  ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item secrets\monitoring-edit.env.sops -Force
$env:NEW_KEY = ""
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "Canary FAILED" }

SOPS temp file MUST be named *.env.sops inside secrets/. Any other name or location destroys the file.

A4 — Commit, push, restart cost-exporter

git add secrets\monitoring.env.sops
git commit -m "patch(#NNN): rotate P24_INFRA_MEZMO_SERVICE_KEY"
git push

After secrets-sync.yml completes on vps-i1:

ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose up -d --no-deps cost-exporter"

A5 — Revoke old key and log

Dashboard: API Keys, delete old sts_ key. Append to docs/secrets-rotation-log.md.

Part B: Ingestion Key Rotation — ONE server (per-server model)

Triggers: a single host’s logdna-agent exits with 403, that host’s key exposed, 180-day schedule.

Because keys are per-server, this edits exactly one SOPS file and restarts one agent. Example below rotates bms-1; substitute the host’s row from the §1904 table for any other server.

B1 — Confirm problem (host-specific)

ssh root@94.23.26.113 "journalctl -u logdna-agent --since='1 hour ago' | grep -i '403\|forbidden'"

B2 — Create + name the replacement ingestion key via API

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$svc = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops |
  Select-String "^P24_INFRA_MEZMO_SERVICE_KEY=").ToString().Split("=",2)[1]
$resp = Invoke-RestMethod -Method POST -Uri "https://api.mezmo.com/v1/config/keys?type=ingestion" `
  -Headers @{"Authorization"="Token $svc"}
$env:NEW_INGEST  = $resp.key
$env:NEW_KEY_ID  = $resp.id
# Rename to the canonical per-server name:
Invoke-RestMethod -Method PUT -Uri "https://api.mezmo.com/v1/config/keys/$($resp.id)" `
  -Headers @{"Authorization"="Token $svc"; "Content-Type"="application/json"} -Body '{"name":"p24-bms-1"}' | Out-Null
$svc = ""

(Linux equivalent: see the §1904 API reference — curl -s -X POST -H "Authorization: Token $SVC" ".../keys?type=ingestion", capture .id/.key with python, then PUT to rename.)

B3 — Update the host’s SOPS file (ONE file only)

bms-1 → bms-servers.env.sops, var MEZMO_INGESTION_KEY_BMS1. No other SOPS file is touched.

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$dec = (sops --decrypt --input-type dotenv --output-type dotenv secrets\bms-servers.env.sops) `
  -replace "^MEZMO_INGESTION_KEY_BMS1=.*", "MEZMO_INGESTION_KEY_BMS1=$env:NEW_INGEST"
[System.IO.File]::WriteAllText("$PWD\secrets\bms-servers-edit.env.sops",
  ($dec -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\bms-servers-edit.env.sops
[System.IO.File]::WriteAllText("$PWD\secrets\bms-servers.env.sops",
  ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item secrets\bms-servers-edit.env.sops -Force
sops --decrypt --input-type dotenv --output-type dotenv secrets\bms-servers.env.sops | Out-Null
if ($LASTEXITCODE -ne 0) { throw "Canary FAILED" }

For vps-i1 rotation the file is monitoring.env.sops / var MEZMO_INGESTION_KEY_VPS_I1; for vps-h1 it is vps-h1.env.sops / var MEZMO_INGESTION_KEY_VPS_H1.

B4 — Commit and push

git add secrets\bms-servers.env.sops
git commit -m "patch(#NNN): rotate MEZMO_INGESTION_KEY_BMS1 (bms-1)"
git push

secrets-sync.yml deploys monitoring.env.sops (vps-i1), vps-h1.env.sops (vps-h1), and — since #1425 — bms-servers.env.sops’s MEZMO_INGESTION_KEY_BMS1 (bms-1, via the dedicated sync-bms-1 job) to those hosts; Docker mezmo-agent restarts automatically on all three. Only bms-2/bms-3 are not covered by secrets-sync (no CI SSH reachability — #1335) — deploy those manually (B5).

B5 — Deploy to bms-2 / bms-3 (systemd agent)

bms-1 no longer uses this procedure — since #1425 it runs a Docker mezmo-agent at /opt/mezmo-agent/ kept in sync automatically by secrets-sync.yml’s sync-bms-1 job (§B4 above); /etc/logdna.env on bms-1 is stale/unused. This systemd procedure now applies to bms-2 and bms-3 only, which use /etc/logdna.env with MZ_INGESTION_KEY (NOT /etc/logdna.conf / key=). SSH user: ubuntu for both.

WARNING — PowerShell BOM bug (#1897): Write-Output $key | ssh … 'read NEW_KEY' adds a UTF-8 BOM to stdin, making the key 35 chars instead of 32. The agent starts but gets 403. Use the base64 method:

$env:_K = $env:NEW_INGEST   # 32-char value from B2
$py = 'import re,sys; k=sys.argv[1]; c=open("/etc/logdna.env").read(); open("/etc/logdna.env","w").write(re.sub(r"MZ_INGESTION_KEY=.*",f"MZ_INGESTION_KEY={k}",c)); print("len:",len(k),"ok:",len(k)==32)'
$b64 = [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($py))
# bms-2 / bms-3 (ubuntu + sudo):
ssh -o BatchMode=yes -i C:\Users\konar\.ssh\id_ed25519 ubuntu@145.239.133.104 `
  "echo $b64 | base64 -d | sudo python3 - $env:_K && sudo systemctl restart logdna-agent && sleep 8 && sudo systemctl is-active logdna-agent"
$env:_K = ""; $env:NEW_INGEST = ""

Verify: wait ~60 s, check app.mezmo.com for incoming logs from that host’s tag.

B6 — Revoke the OLD ingestion key for that host

GET /v1/config/keys, find the previous p24-bms-1 key id, then delete it. No Content-Type header:

Invoke-RestMethod -Method DELETE -Uri "https://api.mezmo.com/v1/config/keys/$OLD_KEY_ID" `
  -Headers @{"Authorization"="Token $svc"}
# (curl: curl -s -X DELETE -H "Authorization: Token $SVC" https://api.mezmo.com/v1/config/keys/$OLD_KEY_ID )

B7 — Log rotation

Append to docs/secrets-rotation-log.md (name the per-server var, e.g. MEZMO_INGESTION_KEY_BMS1).

Part B-legacy: shared ingestion key (pre-migration only)

Until §C runs, the fleet shares one ingestion key. While that is the case, the OLD rule still holds: rotate MEZMO_INGESTION_KEY in monitoring.env.sops AND vps-h1.env.sops AND LOGDNA_API_KEY in bms-servers.env.sops together, then redeploy all hosts (Docker agents via secrets-sync; BMS via B5). Missing any file → 403 on that host (root cause of #1897). Run §C to retire this rule.

Part C: One-time migration — shared key → per-server keys (#1904)

Execute once. It is additive and safe: the shared key stays valid until every host is verified on its own key, so there is no ingestion outage window.

Confirm which BMS hosts run the agent: ssh <user>@<ip> "systemctl is-active logdna-agent". Skip bms-2/bms-3 if inactive.
Create + name 5 ingestion keys via the API (§B2): p24-vps-i1, p24-vps-h1, p24-bms-1, and p24-bms-2 / p24-bms-3 where present.
Add the per-server vars to the right SOPS files (§B3 safe-edit), keeping the old MEZMO_INGESTION_KEY / LOGDNA_API_KEY entries for now:
- MEZMO_INGESTION_KEY_VPS_I1 → monitoring.env.sops
- MEZMO_INGESTION_KEY_VPS_H1 → vps-h1.env.sops
- MEZMO_INGESTION_KEY_BMS1 (+ _BMS2 / _BMS3) → bms-servers.env.sops
~~Repoint the Docker agents in the SAME commit as the SOPS change~~ — already done (#1915, code-only PR merged ahead of the live values above): monitoring/docker-compose.yml mezmo-agent reads LOGDNA_AGENT_KEY=${MEZMO_INGESTION_KEY_VPS_I1:-${MEZMO_INGESTION_KEY}}, and secrets-sync.yml’s sync-bms-1 / sync-vps-h1 jobs read MEZMO_INGESTION_KEY_BMS1 / MEZMO_INGESTION_KEY_VPS_H1 with an automatic fallback to the old shared key when a per-host key is still empty. No code change is needed at this step any more — just populate the SOPS values in step 3 and move to step 5.
Deploy + verify each host (§B4/§B5): merge for vps-i1/vps-h1 + agent restart; SSH-update /etc/logdna.env on each BMS host. Confirm each host’s logs in the Mezmo UI.
Retire the shared key: once all hosts are verified, remove MEZMO_INGESTION_KEY from monitoring.env.sops + vps-h1.env.sops and LOGDNA_API_KEY from bms-servers.env.sops, then delete the old shared Mezmo key via the API (§B6, no Content-Type). Delete this Part B-legacy section in the same PR.
Log the migration in docs/secrets-rotation-log.md and update docs/mezmo-operations.md §2.

Escalation

SOPS decrypt fails: check age key at C:\Users\konar\.age\p24-infra-keys.txt ($HOME/.age/p24-infra-keys.txt on Linux/VPS/workers)
SOPS corrupt (“no matching creation rules found”): git checkout -- secrets\<file>.env.sops
bms-1 still 403: check key length ssh root@94.23.26.113 "grep '^MZ_INGESTION_KEY=' /etc/logdna.env | cut -d= -f2 | wc -c" (expect 33 incl newline)
bms-2 SSH unavailable: jump via ssh -J root@94.23.26.113 root@145.239.133.104
API delete returns 400 "value" is not allowed: you sent a Content-Type header — resend with only Authorization.

Prevention

After the §C migration, rotate per-server (§B) — one SOPS file, one host, no cross-file sync.
Before §C, rotate ALL shared-key locations together (§B-legacy) — monitoring.env.sops, vps-h1.env.sops, bms-servers.env.sops.
Never use Write-Output $secret | ssh — PowerShell adds BOM; use base64 encoding.
After any rotation, run the API list probe to confirm one key per host and no orphans.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="credential_rotation",
    resource="MEZMO_INGESTION_KEY",
    result="success",  # "success" | "failed" | "skipped"
    detail="Scheduled rotation — Mezmo ingestion key regenerated and deployed to all servers",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'credential_rotation', 'MEZMO_INGESTION_KEY', 'success', 'Scheduled rotation — Mezmo ingestion key regenerated and deployed to all servers', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

mezmo-key-rotation