p24-infra Docs

mezmo-logdna-key-deploy

3 min read

Playbook: Deploy / rotate a per-host Mezmo (logdna-agent) ingestion key

Status: ACTIVE (created 2026-08-04 during #5453 diagnosis) Applies to: systemd logdna-agent hosts (bms-2, bms-3). Docker-based mezmo-agent hosts (vps-i1, bms-1) use a different deploy path (compose env) — not covered here.

Background — the #1915 per-host key migration

The Mezmo/LogDNA ingestion key was historically a single shared org key. #1915 migrates each host to its own per-host ingestion key (p24-<host>), minted via the Mezmo API and stored in secrets/bms-servers.env.sops as MEZMO_INGESTION_KEY_<HOST> (e.g. MEZMO_INGESTION_KEY_BMS2).

On systemd hosts the agent reads its key from /etc/logdna.env as MZ_INGESTION_KEY (3.x MZ_* env schema; the older LOGDNA_* names are aliases). Config there also sets MZ_HOSTNAME, MZ_TAGS, MZ_LOG_DIRS, MZ_AUTOUPDATE. There is no MZ_HOST override — the agent uses the default ingest endpoint logs.mezmo.com (ALPN h2, AWS ELB us-east-1).

Symptom of a wrong/rejected key (#5453)

journalctl -u logdna-agent shows, with the agent active (running):

WARN h2::proto::streams::streams: locally-reset streams reached limit (1024)
WARN logdna_agent::_main: failed sending http request, retrying: http2 error:
     stream error detected: unspecific protocol error detected
INFO metrics: {... "ingest":{... "requests_succeeded":0, "requests_failed":<millions>,
     "rate_limits":<n>, ...}, "retry":{"pending":<large>,"storage_used":<~1GB>}}

This LOOKS like a transport/HTTP2 failure but is NOT. The server rejects every request (403 / rate-limit); the Rust h2 crate’s rapid-reset mitigation caps locally-reset streams at 1024 per connection, so once the agent resets 1024 rejected streams it tears the whole connection down as an “unspecific protocol error” — masking the underlying 403. requests_succeeded:0 with a huge requests_failed and multi-hundred-MB retry.storage_used is the fingerprint.

Diagnose before deploying (prove it is the key, not the network)

Transport-vs-auth is decided by two probes (run from any host with egress — key never printed):

# 1. Transport healthy? DNS + TLS + ALPN h2 must all succeed:
getent hosts logs.mezmo.com
echo | openssl s_client -alpn h2 -connect logs.mezmo.com:443 -servername logs.mezmo.com 2>/dev/null \
  | grep -iE 'ALPN|Verify return'          # expect: ALPN protocol: h2  /  Verify return code: 0 (ok)
 
# 2. Auth — POST a one-line batch with the CANDIDATE key, print STATUS CODE ONLY:
NOW=$(date +%s000)
curl -s -o /dev/null -w '%{http_code}\n' --http2 \
  -H "apikey: $KEY" -H 'Content-Type: application/json' \
  -X POST "https://logs.mezmo.com/logs/ingest?hostname=diag&now=$NOW" \
  -d "{\"lines\":[{\"line\":\"diag\",\"app\":\"diag\",\"level\":\"INFO\",\"timestamp\":$NOW}]}"
# 200/202 = key accepted · 403 = key rejected (deploy the correct one) · 429 = rate-limited

If transport probes pass and the deployed key returns 403 while the new SOPS key returns 200 → the fix is a key deploy, not a network/agent change.

Deploy the key (safe, no value printed)

export SOPS_AGE_KEY_FILE=/home/claude-runner/.age/p24-infra-keys.txt
NEWKEY=$(sops -d --input-type dotenv --output-type dotenv \
  /opt/p24-infra/secrets/bms-servers.env.sops | grep '^MEZMO_INGESTION_KEY_BMS2=' | cut -d= -f2-)
 
# Back up, then rewrite ONLY the MZ_INGESTION_KEY line in place — value passed via env, never argv/echo:
ssh -i <key> root@<host> "cp /etc/logdna.env /etc/logdna.env.bak.\$(date +%Y%m%d%H%M%S)"
ssh -i <key> root@<host> "MZK='$NEWKEY' sh -c 'sed -i \"s|^MZ_INGESTION_KEY=.*|MZ_INGESTION_KEY=\$MZK|\" /etc/logdna.env'"  # PLAYBOOK: mezmo-logdna-key-deploy.md
unset NEWKEY
ssh -i <key> root@<host> "systemctl restart logdna-agent"  # PLAYBOOK: mezmo-logdna-key-deploy.md

Verify (must confirm before closing)

Wait ~30–60s, then read the metrics line — requests_succeeded must be climbing and the http2 ... protocol error warnings must stop:

journalctl -u logdna-agent --since '90 sec ago' | grep -oE '"requests_succeeded":[0-9]+' | tail -1
# expect a value > 0 and rising on the next metrics tick (~60s apart)

The 1GB retry backlog (retry.storage_used) drains gradually as the agent flushes buffered lines.

After deploying

  • Record the operation in infra_operations (log_op "claude" "config_change" ...).
  • The plaintext live value now diverges from any stale /etc/logdna.env.bak; SOPS remains the source of truth. Do not commit /etc/logdna.env.
  • Update the #1915 migration log — the host is “migrated” only once requests_succeeded > 0.

Graph View