p24-infra Docs

netdata-health-alarm-tuning-bms1

4 min read

Playbook: Tune / Override Netdata Health Alarms on bms-1

Status: ACTIVE (first written 2026-07-17 for #4242) Host: bms-1 (94.23.26.113, ns367522) — Pinbox24 production Scope: silencing/tuning noisy Netdata health alarms on bms-1 via file-level overrides. Role: sys-admin / infra-task worker (root SSH on bms-1). Not a code-PR task — the live change is applied over SSH; this repo only carries the versioned copy + docs.

Why bms-1 Netdata is special

  • Netdata v1.19.0 (2020 build) — no netdatacli. The reload-health command on modern Netdata (netdatacli reload-health) does not exist here. Reload with the legacy signal: 
    killall -USR2 netdata
  • Netdata is installed directly on the host, outside the Ansible/SOPS/CI pipeline. There is no Netdata role in ansible/roles/ and nothing Netdata-related is deployed by secrets-sync.yml. → The change must be applied on the box, and captured in git only as a documented snapshot (see #2407 — bms-1 config outside the pipeline).
  • Netdata binds 127.0.0.1:19999 (local only, not scraped by Prometheus). Query the health API from the box itself, not remotely.

How Netdata health overrides work (v1.19.0)

Netdata reads health config from two directories:

  1. Stock: /usr/lib/netdata/conf.d/health.d/*.conf (ships with the package — do not edit, a package upgrade clobbers it).
  2. User: /etc/netdata/health.d/*.conf (your overrides — survives upgrades).

Override granularity is the FILE, not the alarm. If /etc/netdata/health.d/<name>.conf exists, Netdata loads it instead of the stock <name>.conf — the stock file is ignored entirely. So an override file must contain every alarm from the stock file it shadows, or the omitted alarms are lost. (E.g. tuning one of the three dbengine.conf alarms requires copying all three.)

Procedure

1. Inspect the stock alarm + live state

SSH="ssh -i ~/.ssh/vps_root_key -o BatchMode=yes -o LogLevel=ERROR root@94.23.26.113"
$SSH 'cat /usr/lib/netdata/conf.d/health.d/<file>.conf'
# Current alarm states (name / status / value / crit / warn / recipient):
$SSH 'curl -s "http://127.0.0.1:19999/api/v1/alarms?all" | python3 -c "import json,sys; a=json.load(sys.stdin)[\"alarms\"]; [print(v[\"name\"],v[\"status\"],v[\"value_string\"],\"crit=\",v[\"crit\"],\"warn=\",v[\"warn\"],\"to=\",v[\"recipient\"]) for k,v in a.items()]"'

2. Build the override locally

Copy the entire stock file, edit only the noisy alarm. Prefer, in order:

  1. Split by dimension / recipient — keep crit + paging only on the dimensions that mean real trouble; demote the noisy dimension to warn and/or to: silent. (This is the #4242 approach.)
  2. Raise the threshold + lengthen every/delay so brief per-cycle spikes don’t trip.

Never fully disable/delete a collector to silence an alarm — you lose the telemetry. Tune the alarm.

to: silent = alarm still evaluates and shows status in the dashboard/API but sends no notification (informational). to: sysadmin = routes to the configured notification channel (Discord).

3. Deploy + reload (mutating — # PLAYBOOK: required by pre-bash-safety hook)

scp -i ~/.ssh/vps_root_key <local-override>.conf root@94.23.26.113:/tmp/override.conf # PLAYBOOK: netdata-health-alarm-tuning-bms1.md
ssh -i ~/.ssh/vps_root_key root@94.23.26.113 \
  'cp /tmp/override.conf /etc/netdata/health.d/<file>.conf && chmod 644 /etc/netdata/health.d/<file>.conf && killall -USR2 netdata' # PLAYBOOK: netdata-health-alarm-tuning-bms1.md

4. Verify the reload

$SSH 'sleep 8; curl -s "http://127.0.0.1:19999/api/v1/alarms?all" | python3 -c "..."'  # confirm new crit/warn/to
$SSH 'pgrep -x netdata'                                                                 # netdata still up
$SSH 'grep -i error /var/log/netdata/error.log | tail'                                  # no NEW parse errors for your file

A health reload re-initialises alarms to UNINITIALIZED then re-evaluates — a one-off “not sending notification for CLEAR (last status was UNINITIALIZED)” line is normal, not an error.

5. Verify across a full cycle

If the noise is periodic (e.g. dbengine datafile rotation ~every 17 min), re-check the alarm status after at least one full cycle (~20 min) to confirm it never re-enters CRITICAL:

$SSH 'curl -s "http://127.0.0.1:19999/api/v1/alarms?all" | python3 -c "..."'  # status must not be CRITICAL

6. Record in git + audit

  • Commit the deployed file to docs/servers/config-snapshots/ and document it in the host ops doc (docs/servers/p4-ovh-bms-1-ns367522-operations.md).
  • Write an infra_operations audit row (op_type=config_change, env=bms-1).

Rollback

ssh -i ~/.ssh/vps_root_key root@94.23.26.113 \
  'rm -f /etc/netdata/health.d/<file>.conf && killall -USR2 netdata' # PLAYBOOK: netdata-health-alarm-tuning-bms1.md
# Removing the user override restores the stock alarm on the next reload.

Applied history

DateAlarmChangeIssue
2026-07-1710min_dbengine_global_flushing_errorscrit>0/sysadmin → warn>3000/silent (no crit); FS+IO alarms kept crit>0/sysadmin. Also symlinked missing /etc/netdata/apps_groups.conf → stock.#4242

Graph View

Backlinks