p24-infra Docs

mezmo-exporter-api-errors-triage

4 min read

Playbook: MezmoExporterApiErrors — triage before touching the service key

Alert: MezmoExporterApiErrors (monitoring/prometheus/rules/pinbox24.yml) Service: mezmo-exporter (monitoring-mezmo-exporter-1 on vps-i1, 217.154.82.162, port 9250) Severity: warning Occurrences to date: #4195, #4486, #4513 — all three transient upstream Mezmo blips, all self-resolved, zero actions required Written after: #4513 (2026-07-28), which also recalibrated the alert so this class no longer pages

What the alert actually means

mezmo_exporter_api_errors_total increments on every failed Mezmo Export API query — 4xx, 5xx, timeout, or a missing/invalid P24_INFRA_MEZMO_SERVICE_KEY. When it rises, every pinbox24_*_5m metric collapses to 0, because _mezmo_query() fails soft and returns []. That collapse is why the alert exists: without it, exporter breakage masquerades as a Pinbox24 log-forwarding outage via Pinbox24NoLogs (this misdiagnosis cost ~20h on 2026-07-01/02, #2492).

The alert names P24_INFRA_MEZMO_SERVICE_KEY in its summary as a hint, not a diagnosis. Not one of the three firings so far was a key problem. Do not rotate anything before running the triage below.

The counter advances per QUERY, not per poll cycle

collect_all() issues 12 _mezmo_query() calls per cycle (app.py:571–770): 1 unlabeled mongo-timeout + 3 per backend container × 2 (v42-prod, v32-prod) + route-errors + log-volume + slow-queries + mongoose + pm2-restarts + mailgun. At POLL_INTERVAL_S=60:

SituationErrors produced
One bad poll cycle~12
Sustained breakage~12/min → ~180 per 15m

This 12× multiplier is what made the original increase[15m] > 5, for: 0m rule fire on every transient blip — one bad cycle already exceeded it. Since #4513 the rule is increase(mezmo_exporter_api_errors_total[5m]) > 24 with for: 10m: the short window drains within 5 minutes of the last error, so for: genuinely gates on persistence and only a sustained failure pages (at ~T+10 min). If you change POLL_INTERVAL_S or add/remove queries in collect_all(), recompute the threshold as ~2 cycles’ worth of queries.

Triage — read-only, ~2 minutes

Everything here is read-only; none of it modifies the server.

1. Is it still failing, or already recovered?

ssh root@217.154.82.162 'docker logs --since 30m monitoring-mezmo-exporter-1 2>&1 | grep "Mezmo Export API"'

Note the timestamp of the last error and the HTTP status. In all three occurrences so far the burst had already ended before the alert was even filed — the increase() lookback window simply still contained it.

2. Confirm data is flowing again

# On vps-i1 (or from any host that can reach the exporter):
curl -s http://localhost:9250/metrics | grep -E '^(mezmo_exporter_api_errors_total|pinbox24_log_lines_5m_total)'

pinbox24_log_lines_5m_total > 0 means the exporter is querying Mezmo successfully right now. Sample mezmo_exporter_api_errors_total twice ~90s apart — flat means fully recovered.

curl -s -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=increase(mezmo_exporter_api_errors_total[5m])'
curl -s 'http://localhost:9090/api/v1/alerts' | jq -r '.data.alerts[]? | select(.labels.alertname|test("Mezmo")) | "\(.labels.alertname) \(.state)"'

3. Classify by HTTP status — this is the decision point

Logged statusMeaningAction
5xx {"error":"Internal Server Error","code":"ServerError"}Mezmo-side outageNone. Verify recovery, close the issue. (#4195, #4513)
401 with "Unable to validate your servicekey at the moment"Mezmo’s own token-validation backend blipping — note the wording “at the moment”None. Transient; self-resolves in seconds. (#4486)
401/403 sustained, and a direct probe also returns 401Key genuinely expired/revokedRotate — see below
P4_INFRA_MEZMO_SERVICE_KEY not set in the logsKey missing from the container envRe-run secrets-sync.yml, then restart the exporter

4. The distinguishing test — probe the key directly

Status code only. Never print the key value (see CLAUDE.md §Secrets).

KEY=$(grep -m1 '^P24_INFRA_MEZMO_SERVICE_KEY=' /opt/p24-infra/monitoring/.env | cut -d= -f2-)
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Token $KEY" https://api.mezmo.com/v1/config/view
curl -s -o /dev/null -w '%{http_code}\n' -H "Authorization: Token $KEY" https://api.mezmo.com/v1/config/keys
unset KEY
  • 200 → the key is valid. Whatever you saw was transient. Do not rotate.
  • 401 → the key is dead. Proceed to rotation.

If the key really is dead

Rotation is a secret-manager operation — do not attempt SOPS writes from a sys-admin or dev session. Follow docs/playbooks/secret-manager-request.md: file a [SECRET-REQUEST] issue and dispatch it with job_type: secret-manager. The key lives in secrets/monitoring.env.sops as P24_INFRA_MEZMO_SERVICE_KEY; secrets-sync.yml deploys it to vps-i1 and restarts the exporter.

Precedent for a genuine key break: PR #2596 (#2492) — the exporter’s auth header was malformed, the metric sat at 0 for ~20h, and Pinbox24NoLogs misfired the whole time.

  • monitoring/prometheus/rules/tests/mezmo_exporter_api_errors_test.yml — promtool unit tests pinning the calibration (transient bursts must not fire; sustained failure must)
  • Pinbox24NoLogs — gated on increase(mezmo_exporter_api_errors_total[10m]) == 0 so exporter breakage can never masquerade as a log-forwarding outage
  • MailgunPipelineExporterErrors — same pattern for mailgun-pipeline-exporter, but correctly calibrated for its own 1-error-per-cycle, 300s-poll shape (> 2 in 15m)

Graph View