mezmo-operations

Mezmo (LogDNA) — Operations Workbook

Service: mezmo (SaaS — mezmo.com, formerly LogDNA) Compliance: dev_r_services.service_name = 'mezmo' Credential key: P24_INFRA_MEZMO_SERVICE_KEY (in secrets/monitoring.env.sops)

---

1. What Mezmo does in this ecosystem

Mezmo (formerly LogDNA) provides centralized log ingestion and search for Pinbox24 production containers running on bms-1.

The logdna-agent systemd service on bms-1 ships logs from these containers to the Mezmo cloud:

Container / log dir	Description
v42-prod	Pinbox24 v42 production
v42-notify-prod	Pinbox24 v42 notification service
s3-v2-v42-prod	Pinbox24 v42 S3/storage service

Logs are available for short-term search and debugging at https://app.mezmo.com.

---

2. Credentials

Two key categories — a single management/service key, plus one per-server ingestion key.

Service (management) key — used by cost-exporter and scripts/mezmo-manage.py, not per-server:

Credential	Type	Purpose	SOPS file
P24_INFRA_MEZMO_SERVICE_KEY	Service key (sts_...)	Mezmo management API	monitoring.env.sops

Per-server ingestion keys (#1904) — each log-shipping host has its own key, so rotating one host never touches another host’s SOPS file (the shared-key cross-file sync was the root cause of #1897):

Server	SOPS var	SOPS file	Consumer
vps-i1	MEZMO_INGESTION_KEY_VPS_I1	monitoring.env.sops	monitoring/docker-compose.yml mezmo-agent
vps-h1	MEZMO_INGESTION_KEY_VPS_H1	vps-h1.env.sops	.github/workflows/secrets-sync.yml (sync-vps-h1 job) → /root/.env + /root/docker-compose.yml (host-managed Docker mezmo-agent, project root, container root-mezmo-agent-1; corrected #5454 — there is no /opt/mezmo-agent on this host, no hostinger/docker-compose.yml in this repo)
bms-1	MEZMO_INGESTION_KEY_BMS1	bms-servers.env.sops	.github/workflows/secrets-sync.yml (sync-bms-1 job) → /opt/mezmo-agent/.env (Docker mezmo-agent; corrected #1915 — bms-1 moved off the systemd /etc/logdna.env agent per #1425)
bms-2	MEZMO_INGESTION_KEY_BMS2	bms-servers.env.sops	/etc/logdna.env (if logdna-agent installed — unconfirmed, verify before minting)
bms-3	MEZMO_INGESTION_KEY_BMS3	bms-servers.env.sops	/etc/logdna.env (if logdna-agent installed — unconfirmed, verify before minting)

Migration status (updated 2026-08-11, #5454 verified live): 4 per-host keys minted and stored in SOPS (PR #5448). vps-i1, bms-1, and vps-h1 are migrated — secrets-sync.yml’s sync-vps-h1 job now targets the real live path (/root/docker-compose.yml + /root/.env, project root, container root-mezmo-agent-1) instead of the nonexistent /opt/mezmo-agent PR #5055 assumed, idempotently repoints LOGDNA_AGENT_KEY to ${MEZMO_INGESTION_KEY_VPS_H1:-${MEZMO_INGESTION_KEY}}, and force-recreates the container. vps-h1 cutover confirmed live 2026-08-11 (sync-vps-h1 run 31537467669): root-mezmo-agent-1 was recreated + started on the per-host key. A follow-up bug was fixed in the same pass — the recreate step called the v1 docker-compose binary, which is not installed on vps-h1 (Compose v2 plugin only), so it had silently failed every prior run (masked by continue-on-error) and the container had never actually been recreated; the step now prefers docker compose v2 with a v1 fallback. bms-2 is NOT migrated — its logdna-agent has an unrelated, pre-existing HTTP2 transport failure (0 successful requests) that must be fixed first, see issue #5453. bms-3 is inactive, out of scope. The old shared MEZMO_INGESTION_KEY (monitoring.env.sops + vps-h1.env.sops) / LOGDNA_API_KEY (bms-servers.env.sops) remains authoritative and must NOT be retired until #5453 is resolved and bms-2 is verified on its own key — see docs/secrets-rotation-log.md for full per-host detail.

Rotation procedure (per-server + one-time migration): docs/playbooks/mezmo-key-rotation.md

Reading the service key

# Windows dev workstation
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "P24_INFRA_MEZMO"

# Linux / VPS
export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops | grep P24_INFRA_MEZMO

---

3. BMS logdna-agent configuration

The systemd logdna-agent config lives at /etc/logdna.env on each BMS host (verified 1911 — NOT /etc/logdna.conf). The ingestion key is the MZ_INGESTION_KEY field. SSH user: root for bms-1 (94.23.26.113), ubuntu for bms-2 (145.239.133.104) and bms-3 (51.68.155.224).

# View current config (redacts key value from display)
ssh root@94.23.26.113 "grep -v 'KEY\|TOKEN' /etc/logdna.env"
 
# Check agent service status
ssh root@94.23.26.113 "systemctl status logdna-agent"
 
# View agent logs
ssh root@94.23.26.113 "journalctl -u logdna-agent --since='1 hour ago'"
 
# Restart after config change
ssh root@94.23.26.113 "systemctl restart logdna-agent"

Key config fields in /etc/logdna.env:

Field	Value
MZ_INGESTION_KEY	Per-server ingestion key — MEZMO_INGESTION_KEY_BMS1 (bms-1) etc. (#1904)
LOGDNA_LOGDIR	Docker log dirs (typically /var/lib/docker/containers)
LOGDNA_TAGS	e.g. bms-1,pinbox24 (used for log filtering in Mezmo UI)
LOGDNA_HOSTNAME	OVH hostname (e.g. ns367522 for bms-1)

---

4. Log retention

Plan	Retention
Free	0 days (no historical search)
Lite / Pay-as-you-go	Typically 3–7 days

To check the current retention via API:

curl -u "${P24_INFRA_MEZMO_SERVICE_KEY}:" \
  https://api.mezmo.com/v1/config/ingestion

---

5. Key rotation procedure

The full rotation procedure is the single source of truth in docs/playbooks/mezmo-key-rotation.md — do not duplicate it here. That playbook covers:

• Part A — service key rotation (P24_INFRA_MEZMO_SERVICE_KEY, dashboard-created)
• Part B — per-server ingestion key rotation (one SOPS file, one agent) — the common case (#1904)
• Part B-legacy — shared-key rotation across all three files (pre-migration, #1897)
• Part C — one-time migration from the shared key to per-server keys

IMPORTANT — Auth header: Mezmo’s servicekey: request header is deprecated. All API calls (Config API, Export API, usage API) must use Authorization: Token <key>. The cost-exporter (collect_mezmo) and scripts/mezmo-manage.py already use this header — verify after any update.

Ingestion keys are API-creatable (POST /v1/config/keys?type=ingestion, type as a query param) — see the verified API reference in the rotation playbook. Only the service key must be created in the dashboard.

---

6. Querying logs via Mezmo API

# Search logs in the last 1 hour
curl -u "${P24_INFRA_MEZMO_SERVICE_KEY}:" \
  "https://api.mezmo.com/v1/export?from=$(date -d '1 hour ago' +%s000)&to=$(date +%s000)&query=pinbox24" \
  | jq '.lines[] | .timestamp,.line'
 
# Get today's usage (lines ingested)
curl -u "${P24_INFRA_MEZMO_SERVICE_KEY}:" \
  "https://api.mezmo.com/v1/usage/account?from=$(date -d 'today 00:00' +%s000)&to=$(date +%s000)"

Full API reference: https://docs.mezmo.com/log-analysis-api

---

7. Cost monitoring

The cost-exporter service on vps-i1 queries the Mezmo usage API daily and exposes:

Metric	Description
mezmo_active_lines_today	Log lines ingested today
mezmo_retention_days	Configured log retention (days)
cost_collector_last_success_timestamp_seconds{collector="mezmo"}	Last successful collection
cost_collector_errors_total{collector="mezmo"}	Collection errors

The env var P24_INFRA_MEZMO_SERVICE_KEY must be present in monitoring/.env on vps-i1 (deployed from SOPS by secrets-sync.yml).

---

8. Troubleshooting

Symptom	Cause	Fix
logdna-agent stopped on bms-1	Service crashed or key invalid	systemctl restart logdna-agent; check MZ_INGESTION_KEY in /etc/logdna.env
No logs in Mezmo UI	Agent not running, wrong log dirs, or ingestion quota	Check journalctl -u logdna-agent; verify LOGDNA_LOGDIR in /etc/logdna.env
cost_collector_errors_total{collector="mezmo"} rising	Invalid service key or plan without usage API	Rotate key per §5; check plan includes usage endpoint
403 from cost-exporter	Service key expired or revoked	Rotate key per §5

---

9. Related issues and references

Reference	Description
Issue #728	LOGDNA_API_KEY rotation procedure (original)
Issue #1115	Register Mezmo as p24-infra resource + monitor cost/usage
Issue #1112	Phase 5: Mezmo webhook → n8n mezmo-alert-router (MongoTimeoutError, No-logs alerts)
Issue #1158	P24_INFRA_MEZMO_SERVICE_KEY rotation + auth header fix (servicekey: → Authorization: Token)
secrets/monitoring.env.sops	Primary credential store (P24_INFRA_MEZMO_SERVICE_KEY)
secrets/bms-servers.env.sops	Legacy alias (LOGDNA_API_KEY) for bms-1 agent
monitoring/exporters/cost-exporter/main.py	Mezmo usage collector (collect_mezmo)
docs/13-external-services/README.md	External services inventory
n8n-workflows/mezmo-alert-router.json	Sanitized n8n workflow export (secrets as placeholders)
docs/playbooks/mezmo-key-rotation.md	Step-by-step key rotation playbook

---

10. Mezmo alert views → n8n webhook routing

Eight Mezmo alert views are configured to fire a webhook to n8n on bms-4:

View name	View ID	Condition	Threshold	n8n action
MongoTimeoutError	1ce14a4838	MongoTimeoutError -host:bms-4 -host:vps-i1	10/5min GH; 1/30s Discord	Discord warning + GH issue (label: bug)
No logs from Pinbox24 w3 >24h	—	Absence: app:pm2_v42-prod absent >24h on host w3	—	Discord critical + GH issue
No logs from Pinbox24 w4 >24h	—	Absence: app:pm2_v42-prod absent >24h on host w4	—	Discord critical + GH issue
Pinbox24 Login Errors (w4)	506d9df6a4 alert:9c6b9f6b7f	app:pm2_v42-prod ("unauthorized" OR "Invalid credentials")	5/5min	Discord warning (n8n)
Pinbox24 File Upload Errors (w4)	5b77b86858 alert:950975627f	app:pm2_v42-prod (upload OR putObject) (error OR failed)	3/5min	Discord warning (n8n)
Pinbox24 File Download Errors (w4)	9299c405ba alert:1f03b5153c	app:pm2_v42-prod (download OR getObject OR presign) (error OR 404 OR 500)	3/5min	Discord warning (n8n)
Pinbox24 Excel Import Errors (w4)	db96953bfc alert:539a45644f	app:pm2_v42-prod (import OR xlsx) error	3/5min	Discord warning (n8n)
Pinbox24 User Creation Errors (w4)	64c44cdb78 alert:713e5e725e	app:pm2_v42-prod (createUser OR register) error	3/5min	Discord warning (n8n)

MongoTimeoutError view query — self-ref host filter (#1433). The view originally matched the bare token MongoTimeoutError across all hosts, so agent transcripts on the AI-runner hosts (bms-4 hourly-devops-triage.log and dev-issue-<N>.log, vps-i1) that quote [MongoTimeoutError] re-triggered the alert and spawned false-positive GitHub issues (#1429/#1431/#1433). Worker-log filenames are unbounded and Mezmo app: queries do not support wildcards, so ingestion exclusions cannot catch them — the fix is the host filter -host:bms-4 -host:vps-i1 in the view query. Genuine client-side mongojs timeouts originate only from the Pinbox24 app server bms-1, which the filter preserves.

The view’s n8n (GH-issue) channel also carries triggerlimit=10 / triggerinterval=5m (was 1 / 30s): a single transient self-healing timeout no longer opens an issue — only a sustained burst (≥10 in 5 min) does, matching the MongoDB Slow Queries (rs0) view and the Prometheus Pinbox24MongoTimeouts (> 5 for 10m) intent. The Discord channel keeps 1 / 30s for immediate low-cost visibility. Apply view edits via PUT /v1/config/view/{viewid} (note: GET returns bodyTemplate as a string but PUT requires it as an object, and alertid must be omitted from the PUT body).

n8n router self-ref host guard (#1434) — defense-in-depth. The MongoTimeoutError view filter above lives only in Mezmo config (not in git), so it can be silently reverted by a UI edit or a view re-import. The mezmo-alert-router “Prepare Alert” node therefore carries an independent, code-reviewed guard: const SELF_REF_HOSTS = ['bms-4', 'vps-i1']; and if (a.includes('mongotimeout') && SELF_REF_HOSTS.includes(host)) { continue; }. A MongoTimeoutError alert reported for an agent-transcript host is dropped entirely (no Discord, no GH issue) before the Discord/GH-issue nodes run. Confirmed root cause: 30/30 host:bms-4 MongoTimeoutError lines over 72h came from hourly-devops-triage.log (18) and dev-issue-<N>.log worker transcripts (12) — zero genuine DB timeouts; bms-4 does not run the Pinbox24 app. Other alert types (triage, agent_timeout, no-logs) from these hosts are legitimate and still pass through. The guard is in the repo export and the live workflow (Sc2AKCdslinnSvWs); update both together — edit n8n-workflows/mezmo-alert-router.json, then PUT the live node via /api/v1/workflows/{id} preserving the live secret values.

Webhook endpoint: https://n8n.bms-4.infra.zintegrowana.online/webhook/mezmo-alert
n8n workflow ID: Sc2AKCdslinnSvWs (mezmo-alert-router)
Workflow export: n8n-workflows/mezmo-alert-router.json

Mezmo webhook body template

All three views use:

{"alert":"{{name}}","host":"{{host}}","lines":"{{lines}}"}

Maps to $json.body.alert, $json.body.host, $json.body.lines in n8n.

Updating views via API

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$key = (sops --decrypt --input-type dotenv --output-type dotenv secrets\monitoring.env.sops | Select-String "P24_INFRA_MEZMO_SERVICE_KEY").Line.Split("=",2)[1]
$headers = @{ "Authorization" = "Token $key"; "Content-Type" = "application/json" }
 
# List views
Invoke-RestMethod -Uri "https://api.mezmo.com/v1/config/view" -Headers $headers -Method GET

See docs/playbooks/mezmo-key-rotation.md for the full key rotation procedure.

11. MongoDB slow query logging + metric (issue #1196)

The two rs0 data-bearing members ship their MongoDB log to Mezmo so slow queries are searchable and counted. bms-4 (arbiter) and bms-1 (no MongoDB) are excluded.

Phase 1 — log shipping (DONE, via #1202 / #1203)

Host	mongod.log shipped	Mezmo tags
bms-2 (PRIMARY)	/var/log/mongodb/mongod.log ✅	none — see caveat below
bms-3 (SECONDARY)	/var/log/mongodb/mongod.log ✅	mongodb, bms-3, p24-infra, mongodb-staging

Caveat — bms-2 has no Mezmo tags. Its logdna-agent ships mongod.log but without the mongodb,bms-2,rs0 tags the issue’s Phase 1 called for. Anything keyed on the mongodb tag will miss bms-2. Both the exporter metric and the alert view below therefore key on log content + _host, not the tag. Adding the tags needs an edit to LOGDNA_TAGS in /etc/logdna.env on bms-2 (requires SSH; bms-4 agents cannot SSH there — #1335). Tracked as a follow-up.

MongoDB 7.0 writes slow operations (default slowOpThresholdMs=100) to mongod.log as JSON with "msg":"Slow query", attr.ns (db.collection) and attr.durationMillis.

Phase 2 — Mezmo view + alert

Field	Value
View name	MongoDB Slow Queries (rs0)
View ID	a81fb8283e
Query	"Slow query" (host:bms-2 OR host:bms-3)
Alert ID	459c367a88
Alert	presence, email radieu@gmail.com, >10 lines / 5 min (triggerlimit=10, triggerinterval=300)

Phase 3 — mongodb_slow_queries_total metric

mezmo-exporter (vps-i1, :9250, monitoring/exporters/mezmo-exporter/app.py) adds:

mongodb_slow_queries_total{server="bms-2|bms-3", database="<db>"}

A 5-minute-window Gauge (consistent with the other *_5m_total gauges in this exporter): each scrape queries the Mezmo Export API for "Slow query", keeps mongod.log lines, groups by _host + attr.ns database, and emits 0 for any data member with no slow queries that window. Parser: parse_slow_query_counts() (unit-tested in tests/test_app.py).

Phase 4 — Grafana

Dashboard pinbox24-operations.json, panel ID 115 (“MongoDB Slow Queries”) graphs sum by (server) (mongodb_slow_queries_total) alongside the existing proxy indicators (scanAndOrder, open cursors, writeConflicts).

11a. Mongoose disconnect counter (issue #2472)

mezmo-exporter also exposes a monotonic counter for MongoDB connection drops seen in the Pinbox24 PM2 app logs:

pinbox24_mongoose_disconnect_total{container="v42-prod|v32-prod"}

Each scrape queries the Mezmo Export API for "Mongoose disconnected" over a contiguous, non-overlapping window (starting where the previous cycle ended), then .inc()s the counter per container. Unlike the *_5m_total gauges this must not reuse the overlapping 5m lookback, which would 5× double-count. The container label is derived from the Mezmo _app/_file basename, so agent-transcript hosts that merely quote the phrase are excluded (same self-reference class as #1433). Parser: parse_mongoose_disconnects() (unit-tested in tests/test_app.py).

Alert MongooseDisconnect (rules/pinbox24.yml) fires immediately (for: 0m) on the first disconnect: increase(pinbox24_mongoose_disconnect_total[5m]) > 0 — catching a conn drop before the 10m Pinbox24MongoTimeouts alert would surface it.

12. Ingestion exclusion rules

Noisy / self-referential log lines are dropped at ingestion (before indexing and before the alert pipeline) via Mezmo ingestion exclusion rules. These are declared in scripts/mezmo-manage.py (EXCLUSION_RULES) and applied idempotently:

Rules must be active — created inactive by default (#1433). Mezmo creates an exclusion rule inactive, and an inactive rule filters nothing. An older exclusions apply omitted the active field and skipped any rule whose title already existed, so every rule was silently created inactive and never reconciled — the #1428 triage-log-self-ref guard had in fact never been filtering (latent cause of the recurring [MongoTimeoutError] FP issues). exclusions apply now reconciles: it PATCHes (PATCH /v1/config/ingestion/exclusions/{id} — POST/PUT return 405) any existing rule whose active/query/indexonly drifts from EXCLUSION_RULES, and every rule declares "active": True. After running apply, confirm with the API (the exclusions list table shows IndexOnly, not active): GET /v1/config/ingestion/exclusions → each p24 rule should report "active": true.

# List live rules
sops exec-env secrets/monitoring.env.sops 'python scripts/mezmo-manage.py exclusions list'
# Create any rules from EXCLUSION_RULES that don't exist yet (idempotent)
sops exec-env secrets/monitoring.env.sops 'python scripts/mezmo-manage.py exclusions apply'

On bms-4 the sops exec-env <file> '<cmd>' form mis-detects the .env.sops extension (“Error unmarshalling input json”). Workaround: capture the key into a shell var via MEZMO_KEY=$(sops --decrypt --input-type dotenv --output-type dotenv secrets/monitoring.env.sops | sed -n 's/^P24_INFRA_MEZMO_SERVICE_KEY=//p') then run P24_INFRA_MEZMO_SERVICE_KEY="$MEZMO_KEY" python3 scripts/mezmo-manage.py exclusions apply. (python3, not python, on bms-4.)

Rule title	Query	Why
healthcheck-platform-alert	/api/platform-alert	Health-check probe noise
redis-lifecycle	Redis is connected OR Redis is ready	Redis lifecycle chatter
websocket-alive-check	**Checking for Alive Sockets**	Socket keepalive noise
i18n-langs-fetch	/api/i18n/langs	i18n polling
drivers-broadcast	Total drivers in global list	Broadcast loop
favicon-fetch	GET /favicon.ico	Browser favicon hits
script-log-marker	==> SCRIPT LOG	Script log markers
triage-log-self-ref	app:hourly-devops-triage.log	Self-referential alert loop guard (#1428) — see below

Self-referential triage-log loop (#1428)

The hourly triage agent writes run summaries to /var/log/hourly-devops-triage.log on bms-4. Those summaries quote alert tokens verbatim (e.g. [MongoTimeoutError], [alert]). The file is shipped to Mezmo (app/source hourly-devops-triage.log), so the alert pipeline re-alerted on the agent’s own log lines, spawning false-positive GitHub issues (#1410 quoting #1407; 1430). The triage-log-self-ref rule drops the whole source at ingestion, breaking the loop at the source.

Mezmo’s _app is the file basename, so the exclusion keys on app:hourly-devops-triage.log (confirmed live: host bms-4, file /var/log/hourly-devops-triage.log).