p24-infra Docs

monitoring-standby-vps-h1

3 min read

Monitoring Stack Standby on vps-h1

When to use

When grafana.vps-i1.infra.zintegrowana.online is unreachable and the CF Worker watchdog has confirmed vps-i1 is down (not just a network blip). The watchdog auto-triggers the standby deploy, but this playbook covers manual activation and teardown.

Automatic activation

The CF Worker p24-monitoring-watchdog detects vps-i1 failure within 5–10 min and:

  1. Sends Discord alert with standby Grafana URL
  2. Enqueues deploy-monitoring-standby P0 job → bms-4 worker picks it up
  3. Worker SSHes to vps-h1 and runs deploy-monitoring-standby.sh

Manual activation

If the auto-trigger fails:

# From local machine or any server with SSH key and sops access
ssh root@72.60.32.61  # vps-h1
cd /opt/p24-infra
git pull --ff-only origin main
export MONITORING_HOST="vps-h1"
export MONITORING_DOMAIN="vps-h1.infra.zintegrowana.online"
sops exec-env /opt/p24-infra/secrets/monitoring.env.sops \
  'docker compose \
     -f monitoring/docker-compose.yml \
     -f monitoring/docker-compose.standby.yml \
     up -d --build'

Grafana available at: https://grafana.vps-h1.infra.zintegrowana.online

Pre-requisites (must be in place before first real failover)

  • DNS A records created: grafana.vps-h1.infra.zintegrowana.online + monitor.vps-h1.infra.zintegrowana.online → 72.60.32.61 (Cloudflare, proxied)
  • CF Worker KV namespace WATCHDOG_KV created and bound
  • sops installed on vps-h1 (apt install sops or binary)
  • docker compose V2 available on vps-h1 (docker compose version)
  • p24-infra git repo cloned to /opt/p24-infra on vps-h1

What runs on standby vs primary

ServiceStandby (vps-h1)Notes
Prometheus✅Fresh scrape — no local history
Thanos-query✅Reads history from Wasabi (same bucket)
Alertmanager✅Sends alerts from vps-h1
Grafana✅At grafana.vps-h1.infra.zintegrowana.online
Loki✅Fresh — log history not migrated
Exporters✅queue, cost, pg-stats, vercel, backup, mezmo, n8n
gotenberg❌Disabled on standby
pdf-service❌Disabled on standby
p24-infra-mcp❌Disabled on standby
audit-engine❌Disabled on standby
credential-exporter❌Disabled (no /var/lib/p24 on vps-h1)

Teardown (after vps-i1 recovers)

ssh root@72.60.32.61
cd /opt/p24-infra/monitoring
docker compose -f docker-compose.yml -f docker-compose.standby.yml down

The watchdog auto-triggers teardown when vps-i1 Grafana recovers.

Data gap

Prometheus on standby starts fresh. Any metrics scraped during vps-i1 downtime that were NOT yet uploaded to Wasabi by Thanos sidecar (<2h window) are lost. Data older than 2h is fully available via Thanos-query reading Wasabi.

Loki log history during downtime: not available on standby. Mezmo retains full logs.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="deploy",
    resource="monitoring-standby-vps-h1",
    result="success",  # "success" | "failed" | "skipped"
    detail="Monitoring standby deployed or promoted on vps-h1",
    env="vps-h1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'deploy', 'monitoring-standby-vps-h1', 'success', 'Monitoring standby deployed or promoted on vps-h1', 'vps-h1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks