p24-infra Docs

pinbox24-no-logs-mezmo-dual-agent

4 min read

Playbook: Pinbox24 “no logs” — Mezmo dual-agent conflict on bms-1

Symptom: P1 alert [P1] Pinbox24 no logs - bms-1 fires and re-fires (~every 30 min via mezmo-alert-router). Mezmo/LogDNA stops receiving Pinbox24 application logs even though the apps on bms-1 are healthy and producing logs.

First seen: 2026-07-08 → resolved 2026-07-09 (issue #3404). Host: bms-1 (OVH ns367522, 94.23.26.113).

Root cause

Two log-shipping agents were running on bms-1 at the same time:

AgentWhat it isState
systemd logdna-agent.service (/usr/bin/logdna-agent)Legacy LogDNA install (pre-Mezmo rebrand)Broken — requests_succeeded: 0, millions of requests_timed_out + rate_limits, ~1.2 GB RAM, ever-growing retry backlog. Shipped nothing for >20 h.
Docker mezmo-agent_mezmo-agent_1 (/opt/mezmo-agent, image logdna/logdna-agent:stable)Current managed agent; .env is distributed by secrets-sync.ymlHealthy — rate_limits: 0, retries: 0, throughput climbing; tails the same Pinbox24 pm2 log paths.

Both agents tail the same files (/var/log/<service>/pm2/*.log, incl. v42-prod, v32-prod, s3-v2-v42-prod) and ship to the same Mezmo account. When the Docker agent replaced the legacy systemd agent, the systemd unit was never stopped/disabled. The dead systemd agent kept retrying millions of timed-out requests, self-inflicting a rate-limit storm and blocking delivery, so Mezmo saw no Pinbox24 logs → the alert fired.

Note: Pinbox24 on bms-1 is fully Dockerized (post-#3171). The alert’s legacy action “SSH → docker ps → pm2 status” is outdated: pm2 list for root is empty. Logs are shipped by the mezmo-agent, not root pm2. Investigate the mezmo-agent, not root pm2.

Diagnosis (read-only)

ssh -i ~/.ssh/vps_root_key root@94.23.26.113   # from bms-4; developer id_ed25519 from windows-dev
 
# 1. Is the app producing logs?  (pm2 log files should have recent mtime)
ls -lt /var/log/*/pm2/*_production_out.log | head
 
# 2. How many log agents are running?  (should be exactly ONE — the Docker one)
systemctl is-active logdna-agent            # legacy — should be 'inactive'
docker ps --filter name=mezmo-agent         # managed — should be 'Up'
 
# 3. Legacy systemd agent health — the smoking gun:
journalctl -u logdna-agent --since '1 hour ago' --no-pager \
  | grep -iE 'metrics|timed out' | tail
#    Look for: "requests_succeeded":0  with huge "requests_timed_out"/"rate_limits"
 
# 4. Docker agent health — should be clean:
docker logs --since 5m mezmo-agent_mezmo-agent_1 2>&1 | grep 'INFO  metrics' | tail -2
#    Healthy = "ingest":{... "rate_limits":0,"retries":0} and throughput > 0

Never cat /opt/mezmo-agent/.env, /etc/logdna.env, or run full docker inspect — they contain the ingestion key. Use docker inspect --format '{{...}}' for single non-secret fields only.

Fix

Decommission the broken legacy systemd agent; keep the healthy Docker agent as the sole shipper.

# PLAYBOOK: infra-task-executor.md   (mutating prod op — annotation required by pre-bash hook)
systemctl stop logdna-agent
systemctl disable logdna-agent
# verify:
systemctl is-active logdna-agent    # -> inactive
docker ps --filter name=mezmo-agent # -> still Up

This is safe and reversible (systemctl start logdna-agent restores it) and frees ~1.2 GB RAM. Logs resume flowing to Mezmo within minutes; the mezmo-alert-router alert stops re-firing.

Permanent decommission (follow-up — #3569)

stop + disable survives reboot but the unit file /lib/systemd/system/logdna-agent.service still exists and could be re-enabled by a re-provision. To fully remove:

Ansible audit — DONE (#3569). The IaC side is already safe: no Ansible role installs the logdna-agent apt package or systemd unit. bms-1 is provisioned by bms-servers.yml (audit-daemon role only; the Docker/Pinbox24 stack is explicitly out of scope for every role). The managed Docker mezmo-agent (/opt/mezmo-agent, .env synced by secrets-sync.yml) is the sole log shipper, and ansible/inventory/host_vars/bms-1.yml now documents this and the decommission. A re-provision therefore will not re-introduce the legacy systemd agent.

Server-side cleanup — DONE (#3569, 2026-08-05) by an infra-task worker on bms-4 (a dev-issue PR cannot run these). apt-get purge -y logdna-agent removed the package, the /usr/bin/logdna-agent binary, and the package-owned unit file /lib/systemd/system/logdna-agent.service in one step; the non-package-owned /etc/logdna.conf + /etc/logdna.env (the .env held an ingestion key — never printed) were rm’d separately, then systemctl daemon-reload. Verified: systemctl status logdna-agent → “could not be found”; the only remaining logdna-agent process on the host is the healthy Docker mezmo-agent container (docker inspect PID match), which stayed Up throughout.

bms-1 SSH note (2026-08-05): bms-4 cannot reach bms-1:22 directly (source-IP SSH firewall). Jump via vps-i1 (allowlisted): ssh -o ProxyJump=vps-i1-jump -i ~/.ssh/vps_root_key root@94.23.26.113 (the bms-1 alias in ~/.ssh/config references a missing id_bms1 key — use vps_root_key).

Original checklist (all items now complete):

  • Delete/rename /lib/systemd/system/logdna-agent.service then systemctl daemon-reload. ✅
  • Delete /etc/logdna.conf + /etc/logdna.env (the .env holds an ingestion key — do not print it). ✅
  • Remove the logdna-agent apt package if installed. ✅ (apt purge also removed the unit file above)
  • bms-1 is P0 EOL and under active migration (#3171) — coordinate before touching the host.
  • docs/playbooks/infra-task-executor.md — infra-task worker flow (this issue was handled by one)
  • docs/playbooks/alert-response-runbook.md — general alert response
  • Managed agent: /opt/mezmo-agent/ (compose + .env synced by secrets-sync.yml)

Graph View