p24-infra Docs

monitoring-stack-operations

4 min read

Playbook — Monitoring Stack Operations (vps-i1)

Status: ACTIVE Target: vps-i1 (217.154.82.162) — monitoring stack only (Prometheus, Grafana, Alertmanager, exporters)

Stack location

/opt/p24-infra/monitoring/
├── docker-compose.yml
├── .env
├── grafana/
│   └── provisioning/
│       └── dashboards/       # JSON dashboards auto-provisioned by Grafana
└── prometheus/
    └── rules/

Git remote: radieu/p24-infra, branch: dev. Grafana dashboard JSON changes land here via normal PR → merge → pull cycle.

1. Pull repo updates (routine — after dashboard/config PR merged)

ssh root@217.154.82.162
cd /opt/p24-infra
git fetch origin
git pull origin main   # or: git merge origin/main

Verify no merge conflicts — if dirty, stash or reset as appropriate.

2. Reload Grafana dashboards after JSON change

Grafana auto-provisions dashboards from monitoring/grafana/provisioning/dashboards/. After a git pull that changed dashboard JSON:

cd /opt/p24-infra/monitoring
docker compose restart grafana

Wait ~10 s and verify:

docker compose ps grafana    # State: Up
curl -s -o /dev/null -w "%{http_code}" http://localhost:3000/api/health
# Expected: 200

Panel changes are visible immediately after login — no browser cache clear needed.

3. Reload Prometheus config (rules / scrape config change)

cd /opt/p24-infra/monitoring
curl -s -X POST http://localhost:9090/-/reload
# or: docker compose restart prometheus

3b. Caddyfile route change (new/changed vhost) — MUST recreate the caddy container

deploy-monitoring-config.yml only git pulls config and reloads Prometheus/Alertmanager — it does not touch Caddy. And a plain caddy reload is not enough for a Caddyfile change either.

monitoring/Caddyfile is bind-mounted into the container as a single file (/opt/p24-infra/monitoring/Caddyfile -> /etc/caddy/Caddyfile). Docker pins a single-file bind mount to the inode at container-start time. git pull / git checkout writes the new Caddyfile via a rename → new inode, so a long-running caddy container keeps serving the old file. Symptom (seen on the #6808 report-query-api deploy, 2026-09-02): the host file has the new vhost, but docker exec monitoring-caddy-1 grep <newhost> /etc/caddy/Caddyfile returns 0, caddy reload logs config is unchanged, and the new host gets no cert (openssl s_client -servername <newhost> → no cert, HTTPS returns 000).

Fix — recreate the caddy container so it re-binds the current inode:

cd /opt/p24-infra/monitoring
docker compose up -d --no-deps --force-recreate caddy
# verify the container now sees the new vhost + obtains its LE cert:
docker exec monitoring-caddy-1 grep -c '<newhost>' /etc/caddy/Caddyfile        # expect 1
docker logs --since 60s monitoring-caddy-1 2>&1 | grep -iE 'certificate obtained|acme'
curl -sk --resolve <newhost>:443:127.0.0.1 https://<newhost>/health -o /dev/null -w '%{http_code}\n'

A --force-recreate caddy briefly (~2–3 s) drops all monitoring vhosts (grafana/prometheus/ alertmanager) — acceptable, but do it deliberately. docker restart also re-binds the inode, but --force-recreate is the unambiguous choice.

4. Full monitoring stack restart (emergency only)

cd /opt/p24-infra/monitoring
docker compose down && docker compose up -d

Wait 30–60 s for all containers to stabilise. Check:

docker compose ps

5. Escalation

  • Stack not coming up after restart: check .env for missing vars (docker compose config)
  • Grafana dashboard not updating: confirm git pull ran, check provisioner logs (docker compose logs grafana | grep provision)
  • Alertmanager silences lost: re-apply via Grafana Alerting → Silences UI

Prevention

  • Never git checkout a non-dev branch on /opt/p24-infra — disrupts other processes reading config
  • Never edit dashboard JSON directly on the server — changes are lost on next pull
  • Commit dashboard JSON to the repo and let the pull cycle deploy it

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="monitoring-stack",
    result="success",  # "success" | "failed" | "skipped"
    detail="Monitoring stack operation — git pull / docker compose restart on vps-i1",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'monitoring-stack', 'success', 'Monitoring stack operation — git pull / docker compose restart on vps-i1', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks