p24-infra Docs

pinbox24-no-logs-bms1

7 min read

Playbook: Pinbox24 “No Logs” P1 — REAL outage (bms-1 mezmo-agent 403 key mismatch)

Trigger: P1 GitHub issue titled [P1] Pinbox24 no logs - bms-1 (auto-generated by the mezmo-alert-router n8n workflow, Sc2AKCdslinnSvWs). Backed by the Prometheus alert Pinbox24NoLogs (monitoring/prometheus/rules/pinbox24.yml, per-backend pinbox24_backend_log_lines_5m_total{container=...} == 0, for: 15m, severity warning — #5605 generalised it from the v42-prod-only unlabeled gauge to cover W3/v32-prod too) and the Mezmo absence alert on pm2_v42-prod from bms-1. Host of record: bms-1 (94.23.26.113, OVH hostname ns367522) — Pinbox24 production. Time budget: P1 — resolve within 1h.

First decide false-positive vs. real outage. Most of these alerts fire spuriously (storm tracked in #1310 / PR #1379). Run the Mezmo confirmation query in pinbox24-no-logs-false-positive.md first. Use THIS playbook only once that query confirms no pm2_v42-prod lines from bms-1 in the last 2h — a genuine log-forwarding outage. The concrete root cause documented here was identified in #1425 (2026-06-28).

1. Root cause (#1425)

Pinbox24 application containers on bms-1 were healthy and running (docker ps showed v32-prod, v42-prod, etc. all UP). The break was purely in log forwarding: the mezmo-agent Docker container on bms-1 was failing to ship logs with:

403 Forbidden: "Account not found"

403 "Account not found" from the Mezmo ingestion endpoint means the ingestion key the agent is sending is no longer valid for the account — i.e. the agent’s key has drifted from the live key.

Confirmed by comparing (SHA-256, value never printed):

  • LOGDNA_AGENT_KEY in /opt/mezmo-agent/.env on bms-1, against
  • MEZMO_INGESTION_KEY in secrets/monitoring.env.sops (the source of truth)

The two hashes did not match → the agent was authenticating with a stale ingestion key.

Why it drifted (systemic cause — fix this to stop recurrence)

secrets-sync.yml auto-deploys SOPS values only to vps-i1 and bms-4. bms-1 is NOT on the auto-sync path (it is also unreachable from bms-4 / the AI runners — no agent SSH key; see #1270). So when MEZMO_INGESTION_KEY was last rotated in SOPS, the new value was never propagated to /opt/mezmo-agent/.env on bms-1, and the agent kept sending the revoked key until Mezmo rejected it with 403.

2. Who can run the fix

The remediation requires SSH to bms-1 as root. Update (#5719, 2026-08-06): the bms-4 infra-task worker now holds the bms-1 root key and can run these steps directly — bms-1 is Pinbox24 production and the infra-task worker carries the stack SSH grant (see docs/w3-w4-stack-operations.md). The old “bms-4 / AI-Dev runners cannot reach bms-1 (Permission denied (publickey), #1270)” claim is stale. The human workstation (C:\Users\konar\.ssh\id_ed25519) still works too; either host that can both decrypt SOPS and SSH bms-1 can perform the key-sync fix below.

Secret safety: the ingestion key is a credential. Never echo it, never paste it into a chat, commit, or issue comment. Reference the key NAME (MEZMO_INGESTION_KEY / LOGDNA_AGENT_KEY) only. Use the read-into-variable pattern below so the value never lands in stdout or shell history.

3. Fix — sync the ingestion key and restart the agent

Run from a host that can both decrypt SOPS and SSH bms-1 (human workstation).

cd /opt/p24-infra      # or your local repo checkout
export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
 
# 1. Read the live ingestion key silently into a variable — NEVER echo $MEZMO_KEY
MEZMO_KEY=$(sops --decrypt --input-type dotenv --output-type dotenv \
  secrets/monitoring.env.sops | grep '^MEZMO_INGESTION_KEY=' | cut -d= -f2-)
[ -n "$MEZMO_KEY" ] && echo "key loaded" || { echo "MEZMO_INGESTION_KEY missing in SOPS"; exit 1; }
 
# 2. Push it to bms-1's mezmo-agent .env via stdin (value never appears in argv / ps / history).
#    IMPORTANT: docker-compose v1.25.3 on bms-1 includes trailing \n from env_file values,
#    producing a 33-char key instead of 32 → 403 "Account not found" from Mezmo.
#    Write WITHOUT a trailing newline (printf without \n at end).
printf '%s' "$MEZMO_KEY" | ssh root@94.23.26.113 \
  'KEY=$(cat | tr -d "\n\r")
   printf "LOGDNA_AGENT_KEY=%s" "$KEY" > /opt/mezmo-agent/.env
   echo "updated ($(wc -c < /opt/mezmo-agent/.env) bytes — expect 49)"'
unset MEZMO_KEY    # 3. Clear immediately
 
# 4. Restart the agent to pick up the new key (bms-1 uses docker-compose v1, NOT docker compose)
ssh root@94.23.26.113 \
  'cd /opt/mezmo-agent && docker-compose up -d --force-recreate mezmo-agent'

If the compose service name differs, discover it first: ssh root@94.23.26.113 'cd /opt/mezmo-agent && docker compose config --services'.

4. Verify

# a) Agent container is up and the 403 is gone (filter out any secret-bearing lines)
ssh root@94.23.26.113 \
  'cd /opt/mezmo-agent && docker compose ps && \
   docker compose logs --tail=40 mezmo-agent 2>&1 | grep -iE "403|forbidden|account not found|error|connected|ingest" || true'
#   Expect: container "Up"; NO "403 / Account not found"; ideally a "connected"/ingest-OK line.
 
# b) Pinbox24 app containers still healthy (no app impact from the agent restart)
ssh root@94.23.26.113 "docker ps --format '{{.Names}}\t{{.Status}}' | grep -i prod"

Then confirm logs are actually arriving in Mezmo (authoritative check — run from bms-4/vps-i1): use the /v2/export query in pinbox24-no-logs-false-positive.md §Step 1. Recent pm2_v42-prod lines (age < ~30 min) = forwarding restored.

When confirmed, post a resolution comment with the evidence and close the issue:

gh issue comment 1425 --repo radieu/p24-infra --body-file resolution.md
gh issue close   1425 --repo radieu/p24-infra --reason completed

5. Prevention

The durable fix is to keep bms-1’s mezmo-agent key in step with SOPS, since bms-1 is off the secrets-sync.yml auto-deploy path:

  1. Add bms-1 to the MEZMO_INGESTION_KEY rotation distribution checklist. In docs/secrets-rotation-log.md and the rotation procedure (CLAUDE.md §“When you rotate an existing infra secret”), record that rotating MEZMO_INGESTION_KEY MUST also update /opt/mezmo-agent/.env (LOGDNA_AGENT_KEY) on bms-1 and restart the agent — it is not covered by secrets-sync.yml.
  2. Consider a periodic drift check (e.g. credential-exporter or a cron) that compares the SHA-256 of bms-1’s LOGDNA_AGENT_KEY against SOPS MEZMO_INGESTION_KEY and alerts on mismatch — this would have surfaced the drift before Mezmo rejected the key.
  3. Separately, Pinbox24NoLogs can still false-positive because v42-prod logs go via the Docker JSON driver rather than /var/log (tracked in #1339, the PM2 volume-mount fix). That is a distinct issue from this key drift and does not need an alert-rule change here — the rule is already tuned to warning / for: 2h.

6. Escalation

If the key sync + restart does not restore forwarding (still 403, or no lines in Mezmo after ~15 min):

  • Verify the ingestion key itself is valid against the Mezmo API (it may have been revoked, not just drifted) — see docs/mezmo-operations.md §Troubleshooting and rotate per its §5 if revoked.
  • Check Mezmo ingestion quota / plan limits.
  • Per the Error Notification Standard, send a Discord alert via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (red) and keep the GH issue open with label human-action.

References

  • pinbox24-no-logs-false-positive.md — false-positive triage
    • the authoritative Mezmo /v2/export confirmation query (always run first)
  • docs/mezmo-operations.md — Mezmo agent config, key inventory, troubleshooting, rotation (§5)
  • docs/n8n-operations.md §mezmo-alert-router — alert → Discord/GH routing
  • monitoring/prometheus/rules/pinbox24.yml — Pinbox24NoLogs alert definition
  • #1425 — this incident (mezmo-agent 403 key mismatch, root cause 2026-06-28)
  • #1310 / PR #1379 — no-logs alert storm + dedup fix
  • #1339 — v42-prod Docker-JSON-driver log forwarding (PM2 volume mount) — separate root cause
  • #1270 — bms-1 unreachable from bms-4 / AI runners (no agent SSH key)

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="pinbox24-bms1",
    result="success",  # "success" | "failed" | "skipped"
    detail="Pinbox24 no-logs issue resolved — logging config fixed on bms-1",
    env="bms-1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'pinbox24-bms1', 'success', 'Pinbox24 no-logs issue resolved — logging config fixed on bms-1', 'bms-1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks