Server Monitoring Standard

Version: 1.0 — 2026-06-24 Owner: p24-infra admin (Claude DevOps role)

This document defines the minimum monitoring requirements for all p24-infra servers. Non-compliant servers must be brought into compliance within 30 days.

Requirements

R1 — Log shipping to Mezmo (mandatory)

Every server MUST run mezmo-agent (logdna/logdna-agent) shipping:

System logs from /var/log/messages (or /var/log/syslog on Ubuntu)
Docker container logs (via /var/run/docker.sock)
Kernel journal (OOM kills, hardware errors)

Verification: In Mezmo dashboard, query host:SERVER level:info returns results within 5 min.

Required environment variables:

LOGDNA_AGENT_KEY = value of MEZMO_INGESTION_KEY (from secrets/monitoring.env.sops)
LOGDNA_HOSTNAME = server short name (e.g. vps-i1, bms-4)
LOGDNA_TAGS = p24-infra,SERVER_NAME

R2 — node_exporter for Prometheus scraping (mandatory)

Every server MUST expose node_exporter on port 9100 providing CPU, RAM, disk, and vmstat metrics. Prometheus scrapes all servers at :9100 from monitoring/prometheus/prometheus.yml (job_name: node).

Verification:

curl -s http://SERVER_IP:9100/metrics | grep node_memory_MemAvailable_bytes
# Must return a non-zero value

R3 — CPU alerting (mandatory)

Alertmanager MUST fire when CPU (user+system+iowait) exceeds 80% for 5 minutes. Rule: HighCPU in monitoring/prometheus/rules/infrastructure.yml.

R4 — RAM alerting (mandatory)

Two RAM alerts MUST be configured:

HighMemory (percentage): fires when memory usage >85% for 5 minutes. Rule: present in monitoring/prometheus/rules/infrastructure.yml.
LowMemoryAbsolute (bytes): fires when node_memory_MemAvailable_bytes < 524288000 (500MB) for 2 minutes. Rule: monitoring/prometheus/rules/infrastructure.yml (added by issue #1200). This is the OOM early-warning signal — 500MB available leaves ~2-3 min before kernel OOM fires.

R5 — Disk alerting (mandatory)

Alertmanager MUST fire when root filesystem disk usage exceeds 85% for 5 minutes. Rule: LowDisk in monitoring/prometheus/rules/infrastructure.yml.

R6 — OOM detection (mandatory)

Alertmanager MUST fire when the kernel OOM killer has activated on any server. Rule: OOMKillDetected — increase(node_vmstat_oom_kill[5m]) > 0 File: monitoring/prometheus/rules/infrastructure.yml (added by issue #1200). OOM response: follow docs/playbooks/oom-auto-remediation.md.

R7 — Docker container resource limits (mandatory for Docker hosts)

All Docker containers on vps-i1 and bms-4 MUST have mem_limit set in docker-compose.yml.

Verification:

docker compose config | grep mem_limit
# Every service should have a mem_limit entry

Compliance status — current state (2026-06-24)

Server	IP	mezmo-agent	node_exporter :9100	mem_limits	Compliant
vps-i1	217.154.82.162	YES (Docker in monitoring stack)	YES	YES (all containers)	YES
bms-4	54.36.123.110	YES (Docker, added PR #1200)	YES	Partial (most containers)	PARTIAL
vps-h1	72.60.32.61	YES (root-mezmo-agent-1 Docker)	YES	N/A (systemd services)	YES
bms-1	94.23.26.113	YES (logdna-agent v3.11.2, systemd)	YES	N/A (Pinbox24 Docker)	YES
bms-2	145.239.133.104	YES (logdna-agent v3.11.2, systemd)	YES	N/A (MongoDB direct)	YES
bms-3	51.68.155.224	YES (logdna-agent v3.11.2, systemd)	YES	N/A (MongoDB + staging)	YES

bms-1 note: logdir includes /var/log (fixed 2026-06-24) — ships syslog, auth, and Pinbox24 app logs. bms-2 note: no Docker installed; the native systemd logdna-agent (v3, MZ_* config) is the active R1 mechanism and ships /var/log including mongod.log (slow queries, auth events). Live-verified shipping to Mezmo on 2026-06-25 (host:bms-2, ~190 lines/hr — see audit trail). The stale /opt/mezmo-agent/docker-compose.yml on disk is vestigial and non-functional (Docker is not installed on bms-2) — it does NOT run the agent and should be removed to avoid confusion; do not “fix” log shipping by installing Docker. See issue #1373. bms-3 note: native agent ships /var/log including mongod.log and staging app logs. vps-h1 note: observability-only exception to frozen-gateway policy — outbound HTTPS only, no new ports, no new service responsibilities.

mezmo-agent configuration reference

Docker Compose install (vps-i1, bms-4)

mezmo-agent:
  mem_limit: 256m
  image: logdna/logdna-agent:stable
  restart: unless-stopped
  environment:
    - LOGDNA_AGENT_KEY=${MEZMO_INGESTION_KEY}
    - LOGDNA_HOSTNAME=SERVER_NAME  # e.g. bms-4
    - LOGDNA_TAGS=p24-infra,SERVER_NAME
    - LOGDNA_LOG_DIRS=/var/log
    - LOGDNA_AUTOUPDATE=0
  volumes:
    - /var/log:/var/log:ro
    - /var/lib/docker/containers:/var/lib/docker/containers:ro
    - /var/run/docker.sock:/var/run/docker.sock:ro
  logging:
    driver: json-file
    options: { max-size: "10m", max-file: "3" }

Ingestion key: MEZMO_INGESTION_KEY from secrets/monitoring.env.sops.

Systemd or binary install (bms-1, bms-2, bms-3)

logdna-agent v3.x (Rust-based) is installed on these servers. v3 config differs completely from v2 (Node.js) — do NOT use logdna-agent -k or /etc/logdna.conf.

Install

# Ubuntu 20.04 / 22.04 / 24.04
echo "deb [arch=amd64] https://assets.logdna.com stable main" | sudo tee /etc/apt/sources.list.d/logdna.list
curl -fsSL https://assets.logdna.com/logdna.gpg | sudo apt-key add -
sudo apt-get update && sudo apt-get install -y logdna-agent

Configure via EnvironmentFile

v3 reads config from /etc/logdna.env loaded by the systemd unit (EnvironmentFile=-/etc/logdna.env). Use MZ_* prefix — not LOGDNA_*.

# Decrypt key inline from SOPS — never write to disk unencrypted
export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
KEY=$(sops -d --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/monitoring.env.sops | grep '^MEZMO_INGESTION_KEY=' | cut -d= -f2-)
 
# Write config — heredoc avoids BOM issues
sudo bash -c "cat > /etc/logdna.env" <<EOF
MZ_INGESTION_KEY=${KEY}
MZ_HOSTNAME=SERVER_NAME
MZ_TAGS=p24-infra,SERVER_NAME
MZ_LOG_DIRS=/var/log
MZ_AUTOUPDATE=0
EOF
 
# Strip BOM if file was written via SSH pipe from Windows (PowerShell adds \xef\xbb\xbf)
sudo sed -i 's/^\xef\xbb\xbf//' /etc/logdna.env
 
sudo chmod 600 /etc/logdna.env

Replace SERVER_NAME with the server’s short label (bms-1, bms-2, bms-3).

Enable and start

sudo systemctl enable logdna-agent
sudo systemctl restart logdna-agent
sudo systemctl status logdna-agent   # must show active (running)

Verify

sudo journalctl -u logdna-agent --no-pager -n 20
# Expect: "Sending 100 lines" or "connected" — not "http.ingestion_key is missing"

Then in Mezmo dashboard: query host:SERVER_NAME and confirm lines arrive within 5 minutes.

Troubleshooting v3

Symptom	Cause	Fix
Exit code 22, “http.ingestion_key is missing”	Wrong env var name (`LOGDNA_INGESTION_KEY` instead of `MZ_INGESTION_KEY`)	Rename to `MZ_INGESTION_KEY`
Same error even with correct name	BOM (`\xef\xbb\xbf`) prepended to file by Windows SSH pipe	`sudo sed -i 's/^\xef\xbb\xbf//' /etc/logdna.env`
Logs not appearing in Mezmo	Agent connected but wrong hostname	Check `MZ_HOSTNAME` in `/etc/logdna.env`

Official v3 install docs: https://docs.mezmo.com/log-management/ingestion/agent/agent-installation/linux-agent

Compliance check procedure

Run monthly or after any server change:

Mezmo: query host:SERVER — verify logs arrive for all 6 servers
Prometheus: query up{job="node"} — all 6 targets should show 1
Alert rules on vps-i1:

curl -s http://localhost:9090/api/v1/rules | python3 -m json.tool | grep LowMemoryAbsolute

Docker mem_limits on vps-i1 and bms-4:

docker compose config | grep mem_limit

Audit trail

Date	Change	Author
2026-06-24	Standard created (issue #1200, triggered by vps-i1 OOM crash 2026-06-24)	Claude DevOps
2026-06-24	mezmo-agent deployed to bms-1/bms-2/bms-3 via systemd (logdna-agent v3.11.2); vps-h1 already had root-mezmo-agent-1 running; all 6 servers now R1-compliant	Claude DevOps
2026-06-25	Verified bms-2 native `logdna-agent` live in Mezmo (`host:bms-2` ~190 lines/hr, current ts) via `/v2/export` query — confirms R1-compliant. Issue #1373 was a false alarm: a vestigial `/opt/mezmo-agent/docker-compose.yml` (un-runnable, no Docker on bms-2) was mistaken for the agent. No install needed; doc note clarified.	Claude DevOps (worker, bms-4)

p24-infra Docs

Explorer

server-monitoring-standard