p24-infra Docs

monitoring-stack-operations

3 min read

Playbook — Monitoring Stack Operations (vps-i1)

Status: ACTIVE Target: vps-i1 (217.154.82.162) — monitoring stack only (Prometheus, Grafana, Alertmanager, exporters)

Stack location

/opt/p24-infra/monitoring/
├── docker-compose.yml
├── .env
├── grafana/
│   └── provisioning/
│       └── dashboards/       # JSON dashboards auto-provisioned by Grafana
└── prometheus/
    └── rules/

Git remote: radieu/p24-infra, branch: dev. Grafana dashboard JSON changes land here via normal PR → merge → pull cycle.

1. Pull repo updates (routine — after dashboard/config PR merged)

ssh root@217.154.82.162
cd /opt/p24-infra
git fetch origin
git pull origin main   # or: git merge origin/main

Verify no merge conflicts — if dirty, stash or reset as appropriate.

2. Reload Grafana dashboards after JSON change

Grafana auto-provisions dashboards from monitoring/grafana/provisioning/dashboards/. After a git pull that changed dashboard JSON:

cd /opt/p24-infra/monitoring
docker compose restart grafana

Wait ~10 s and verify:

docker compose ps grafana    # State: Up
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/api/health
# Expected: 200

Panel changes are visible immediately after login — no browser cache clear needed.

3. Reload Prometheus config (rules / scrape config change)

cd /opt/p24-infra/monitoring
curl -s -X POST http://localhost:9090/-/reload
# or: docker compose restart prometheus

4. Full monitoring stack restart (emergency only)

cd /opt/p24-infra/monitoring
docker compose down && docker compose up -d

Wait 30–60 s for all containers to stabilise. Check:

docker compose ps

5. Escalation

  • Stack not coming up after restart: check .env for missing vars (docker compose config)
  • Grafana dashboard not updating: confirm git pull ran, check provisioner logs (docker compose logs grafana | grep provision)
  • Alertmanager silences lost: re-apply via Grafana Alerting → Silences UI

Prevention

  • Never git checkout a non-dev branch on /opt/p24-infra — disrupts other processes reading config
  • Never edit dashboard JSON directly on the server — changes are lost on next pull
  • Commit dashboard JSON to the repo and let the pull cycle deploy it

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="monitoring-stack",
    result="success",  # "success" | "failed" | "skipped"
    detail="Monitoring stack operation — git pull / docker compose restart on vps-i1",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'monitoring-stack', 'success', 'Monitoring stack operation — git pull / docker compose restart on vps-i1', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks