p24-infra Docs

mezmo-terraform-management

5 min read

Playbook — Manage Mezmo (LogDNA) alerts via Terraform

Status: scaffolding shipped (#4185); native bms-1 no-logs alert codified as DISABLED at source (#4197). terraform apply is a sys-admin follow-up. IaC location: terraform/mezmo/ Related ops docs: docs/mezmo-operations.md, docs/logdna-operations.md, docs/playbooks/mezmo-key-rotation.md

1. Why Terraform for Mezmo

Mezmo Views/Alerts are configured by clicking the dashboard, so their definitions live nowhere in git — they can be silently reverted by a UI edit or a re-import (this is exactly the failure mode called out for the MongoTimeoutError view in docs/mezmo-operations.md). The 14–16 Jul “bms-1 no logs” false positives were a mis-tuned absence alert that could only be fixed by hand.

Codifying Views/Alerts in terraform/mezmo/ makes them reviewable, diffable, and reproducible, and lets us disable/retune the absence window + remediation text through a PR. This is complementary to the runtime suppression guard already deployed in the mezmo-alert-router n8n workflow — one fixes the alert at the source (Mezmo config), the other is defense-in-depth at the router.

Decision (#4197) — the bms-1 native alert is DISABLED at source. The native View-alert fired direct to Discord, bypassing the n8n guard, so it kept producing false “no logs for 10+ min” pings even while pinbox24_log_lines_5m_total showed a healthy stream (~17 lines / 5 min). The guarded n8n path already suppresses nologs while logs flow, so it is now the single canonical no-logs gate. logdna_view.bms1_no_logs is codified with alert_enabled = false (default): the view is kept as a saved search but rendered with no alert channel, so it can never fire. Re-enabling is a one-line var flip (alert_enabled = true) that restores the retune config (30m window, mezmo-agent remediation, n8n webhook) preserved in main.tf.

2. TWO Mezmo Terraform providers — do not mix them up

ProviderProductManagesOurs?
logdna/logdnaLog Management (app.mezmo.com)logdna_view, logdna_alert, logdna_archive, logdna_category, logdna_index_rate_alert, logdna_ingestion_exclusion, logdna_key, logdna_member, logdna_stream_config, logdna_stream_exclusion✅ YES — where our bms-1 logs land via mezmo-agent/logdna-agent.
mezmo/mezmoPipeline (sources→processors→destinations)Pipeline objects❌ Not used — we do not route through Mezmo Pipeline today.

terraform/mezmo/ uses logdna/logdna exclusively.

3. Log flow being managed

logdna-agent (systemd, bms-1) → Mezmo log ingest → Views/Alerts → mezmo-alert-router n8n webhook (Sc2AKCdslinnSvWs, …/webhook/mezmo-alert) → Discord embed + optional GH issue.

The bms-1 no-logs alert is a logdna_view with an inline channel set to operator = "absence": it fires when no matching line arrives on hosts=["bms-1"] within the trigger window. Per #4197 that channel is now gated behind var.alert_enabled (default false) — the view exists but emits no channel, so the native alert is disabled and only the n8n-guarded path can page.

4. Credentials — reuse the existing service key

The logdna/logdna provider’s servicekey is the Mezmo Organization service key (Settings → Organization → API Keys). We already have one:

  • P24_INFRA_MEZMO_SERVICE_KEY (sts_…) in secrets/monitoring.env.sops — used today by scripts/mezmo-manage.py + cost-exporter against https://api.mezmo.com/v1/config/*, the same endpoints the Terraform provider drives.

Therefore issue #4185 deliverable (a) — “mint MEZMO_SERVICE_KEY” — is most likely a no-op: reuse P24_INFRA_MEZMO_SERVICE_KEY. A new key is only warranted if that key is found to lack management-write scope for Views/Alerts (Mezmo org service keys are org-wide, so it should suffice). Any actual SOPS write is a secret-manager action — a dev/sys-admin session must not write SOPS. Supply the key to Terraform at runtime, never in HCL:

export TF_VAR_mezmo_service_key="$(sops -d --input-type dotenv --output-type dotenv \
  secrets/monitoring.env.sops | grep '^P24_INFRA_MEZMO_SERVICE_KEY=' | cut -d= -f2-)"

5. Remote state (Wasabi S3)

s3://p24-infra/terraform/mezmo/terraform.tfstate, region eu-central-2, endpoint https://s3.eu-central-2.wasabisys.com. State is never committed (terraform/mezmo/.gitignore) — it can contain the service key. Backend credentials come from the environment (AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY = Wasabi p24-infra IAM).

6. Runbook — IMPORT before APPLY ⚠️

The bms-1 no-logs view already exists in Mezmo. apply without importing first creates a duplicate alert. Order of operations (sys-admin, after the PR merges to main):

cd terraform/mezmo
# 1. export TF_VAR_mezmo_service_key + AWS_* (see §4/§5, and README.md)
./import.sh                                        # init + discover the live view id
terraform import logdna_view.bms1_no_logs <VIEW_ID>
terraform plan                                     # alert_enabled=false ⇒ plan REMOVES the alert channel
terraform apply                                    # disables the native no-logs alert at source (#4197)

After apply, record the view id in docs/mezmo-operations.md §views (the bms1_no_logs_view_id output) and confirm the native no-logs Discord ping no longer fires on a healthy stream.

7. Extending — new views/alerts

  1. Add a logdna_view (inline channel) or logdna_alert + logdna_view (presetid) to main.tf.
  2. If a counterpart already exists in the Mezmo UI → import it first, then reconcile.
  3. Keep alert payloads compatible with the mezmo-alert-router “Prepare Alert” node (alert_type, host, severity, summary, remediation).
  4. PR to main; apply is sys-admin post-merge.

8. Gotchas

  • Absence window too tight → false positives (the 14–16 Jul incident). 30m is the tuned default.
  • Host label — Mezmo may key on the raw hostname, not the friendly bms-1. Confirm from the imported state / API before trusting var.bms1_host.
  • Remediation must point at mezmo-agent, not pm2 — root pm2 list is empty post-#3171.
  • Provider default URL is the legacy api.logdna.com — we override to api.mezmo.com.
  • logdna_view PUT/import quirks (bodyTemplate object vs string, alertid omission) are documented in docs/mezmo-operations.md; Terraform handles the round-trip, but expect the first post-import plan to show cosmetic diffs to reconcile.

Graph View