Monitoring Stack Standby on vps-h1

Issue: TBD (set after issue creation) Status: Plan — awaiting review before implementation Author: orchestrating session, 2026-06-30 Labels: feat, plans-review, monitoring

Planning document. Solves the meta-SPOF: if vps-i1 goes down, all monitoring visibility is lost along with alerting about the outage itself. No code ships until plan review passes.

1. Problem statement

The entire monitoring stack (Prometheus, Grafana, Alertmanager, Thanos, Loki, exporters) runs exclusively on vps-i1. If vps-i1 becomes unavailable:

Prometheus stops scraping → no new metrics
Grafana goes offline → operators are blind
Alertmanager stops routing → no alerts, including no alert about vps-i1 being down
The entity that would normally page us about the outage is the one that’s down

There is currently no detection mechanism outside vps-i1 and no runbook for spinning up monitoring elsewhere.

What already exists (reuse, don’t rebuild)

Asset	Relevant to this design
`monitoring/docker-compose.yml`	Full stack as Docker Compose; already parameterised via env vars from SOPS.
Thanos sidecar → Wasabi `p24-infra` bucket	Prometheus TSDB blocks uploaded continuously. A standby Thanos-query can read history from Wasabi immediately.
SOPS `secrets/monitoring.env.sops`	Single source of truth for all monitoring secrets.
`scripts/deploy-ionos.sh`	Existing deploy script for vps-i1 — model for a standby equivalent.
`p24-meta-dispatcher` CF Worker	Already HA (Cloudflare edge); can host a scheduled cron for health pings.
`dev_r_worker_queue`	P0 job mechanism — can carry `deploy-monitoring-standby`.

2. Root cause analysis — what blocks portability today

Running docker compose up -d from monitoring/ on vps-h1 would fail for four reasons identified by audit of docker-compose.yml:

Blocker 1 — hostname hardcoded in three places

# grafana service
GF_SERVER_ROOT_URL: https://grafana.vps-i1.infra.zintegrowana.online
GF_SERVER_DOMAIN:   grafana.vps-i1.infra.zintegrowana.online
 
# mezmo-agent service
LOGDNA_HOSTNAME: vps-i1
 
# promtail service volume mount
- ./promtail/config-vps-i1.yml:/etc/promtail/config.yml:ro

Blocker 2 — external Docker network not present on vps-h1

networks:
  traccar_traccar-net:
    external: true   # Traccar runs on vps-i1 — network doesn't exist on vps-h1

Caddy uses this network to proxy Traccar. On standby, Traccar is unreachable — this network must not be required.

Blocker 3 — application services bundled into monitoring compose

gotenberg, pdf-service, p24-infra-mcp, audit-engine are not monitoring services and should not run on a monitoring standby host. They add build-time complexity, consume RAM, and have their own dependencies (../infra-src/ relative paths, /var/lib/p24 host mounts).

Blocker 4 — Prometheus binds to 127.0.0.1 only

ports: ["127.0.0.1:9090:9090"]

An external health check (CF Worker) cannot reach Prometheus directly. The health check surface must be the Grafana public URL, which routes through Cloudflare and Caddy.

3. Design

3.1 `docker-compose.standby.yml` — override file for vps-h1

A Docker Compose override file that patches the three blockers without duplicating the base file. Deploy on vps-h1 with:

docker compose \
  -f monitoring/docker-compose.yml \
  -f monitoring/docker-compose.standby.yml \
  up -d

Contents:

# monitoring/docker-compose.standby.yml
# Override: makes the monitoring stack portable to any host other than vps-i1.
# Usage: docker compose -f docker-compose.yml -f docker-compose.standby.yml up -d
# Required env: MONITORING_HOST (e.g. 'vps-h1'), MONITORING_DOMAIN (e.g. 'vps-h1.infra.zintegrowana.online')
 
services:
  grafana:
    environment:
      - GF_SERVER_ROOT_URL=https://grafana.${MONITORING_DOMAIN}
      - GF_SERVER_DOMAIN=grafana.${MONITORING_DOMAIN}
 
  mezmo-agent:
    environment:
      - LOGDNA_HOSTNAME=${MONITORING_HOST}
      - LOGDNA_TAGS=p24-infra,${MONITORING_HOST},monitoring,standby
 
  promtail:
    volumes:
      - ./promtail/config-generic.yml:/etc/promtail/config.yml:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /var/log:/var/log:ro
 
  # Disable application services not relevant to monitoring standby
  gotenberg:
    profiles: ["disabled"]
  pdf-service:
    profiles: ["disabled"]
  p24-infra-mcp:
    profiles: ["disabled"]
  audit-engine:
    profiles: ["disabled"]
 
  # Disable credential-exporter: requires /var/lib/p24 host dir not present on vps-h1
  credential-exporter:
    profiles: ["disabled"]
 
  # Caddy: use standby Caddyfile (no traccar proxy, different domain)
  caddy:
    volumes:
      - ./Caddyfile.standby:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
      - ./status:/opt/p24-infra/monitoring/status:ro
 
networks:
  # Override: do not require the traccar network (not present on vps-h1)
  traccar_traccar-net:
    external: false
    name: monitoring_traccar_local

3.2 `monitoring/promtail/config-generic.yml`

A promtail config with ${MONITORING_HOST} substitution instead of the hardcoded vps-i1 label used in config-vps-i1.yml. Promtail supports env var expansion natively via --config.expand-env (already used via config.expand-env=true in loki).

# monitoring/promtail/config-generic.yml
server:
  http_listen_port: 9080
  grpc_listen_port: 0
 
positions:
  filename: /tmp/positions.yaml
 
clients:
  - url: http://loki:3100/loki/api/v1/push
 
scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        target_label: container
      - replacement: ${MONITORING_HOST}
        target_label: host

3.3 `monitoring/Caddyfile.standby`

Stripped Caddyfile without Traccar proxy and with parameterised domain. Serves only Grafana and the p24-status dashboard:

# monitoring/Caddyfile.standby
# Used when monitoring stack runs on a standby host (not vps-i1).
# MONITORING_DOMAIN must be set in environment (e.g. vps-h1.infra.zintegrowana.online).
 
grafana.{$MONITORING_DOMAIN} {
    reverse_proxy grafana:3000
}
 
monitor.{$MONITORING_DOMAIN} {
    root * /opt/p24-infra/monitoring/status
    file_server
}

3.4 CF Worker monitoring-watchdog — external health check

A new scheduled handler added to p24-meta-dispatcher (or a dedicated CF Worker p24-monitoring-watchdog):

Schedule: every 5 minutes via Cloudflare Workers Cron Trigger

Logic:

// infra-src/monitoring-watchdog/index.js
const PRIMARY_URL   = "https://grafana.vps-i1.infra.zintegrowana.online/api/health";
const STANDBY_HOST  = "vps-h1";
 
export default {
  async scheduled(event, env, ctx) {
    let healthy = false;
    for (let attempt = 0; attempt < 2; attempt++) {
      try {
        const res = await fetch(PRIMARY_URL, {
          signal: AbortSignal.timeout(8000),
          headers: { "User-Agent": "p24-monitoring-watchdog/1.0" }
        });
        if (res.ok) { healthy = true; break; }
      } catch (_) {}
    }
 
    const prev = await env.WATCHDOG_KV.get("monitoring_primary_healthy");
    await env.WATCHDOG_KV.put("monitoring_primary_healthy", healthy ? "1" : "0");
 
    if (!healthy && prev !== "0") {
      // First failure transition → alert + enqueue standby deploy
      await sendDiscordAlert(env, "vps-i1 monitoring DOWN — enqueueing standby deploy");
      await enqueueStandbyDeploy(env, STANDBY_HOST);
    } else if (healthy && prev === "0") {
      // Recovery
      await sendDiscordAlert(env, "vps-i1 monitoring RECOVERED");
      await enqueueStandbyTeardown(env, STANDBY_HOST);
    }
  }
};

enqueueStandbyDeploy inserts into dev_r_worker_queue with:

job_type = 'deploy-monitoring-standby'
priority = 1 (highest)
target_server = 'bms-4' (dispatcher picks it up, SSHes to vps-h1)

WATCHDOG_KV is a Cloudflare Workers KV namespace used to persist state between 5-min invocations (avoids false-positive flap on single request timeout).

3.5 P0 job handler — `deploy-monitoring-standby`

A new job type in the worker queue. When the dispatcher on bms-4 picks up this job:

Worker script: scripts/deploy-monitoring-standby.sh

#!/usr/bin/env bash
# Deploys monitoring stack to vps-h1 as standby.
# Called by queue worker when monitoring-watchdog detects vps-i1 is down.
# PLAYBOOK: monitoring-standby-vps-h1.md
set -euo pipefail
 
TARGET_HOST="72.60.32.61"   # vps-h1
REPO_DIR="/opt/p24-infra"
 
ssh -i /home/claude-runner/.ssh/id_ed25519 root@${TARGET_HOST} bash <<'REMOTE'
  set -euo pipefail
  cd /opt/p24-infra
  git pull --ff-only origin main
  export MONITORING_HOST="vps-h1"
  export MONITORING_DOMAIN="vps-h1.infra.zintegrowana.online"
  sops exec-env /opt/p24-infra/secrets/monitoring.env.sops \
    'docker compose -f monitoring/docker-compose.yml \
                    -f monitoring/docker-compose.standby.yml \
                    up -d --build'
REMOTE

3.6 DNS failover

When standby activates, Grafana becomes available at: https://grafana.vps-h1.infra.zintegrowana.online

No automatic DNS switch is needed — each host has its own permanent subdomain. Operators access the standby Grafana URL directly. The watchdog Discord alert includes the standby URL.

Cloudflare DNS records required (pre-provisioned, not created at failover time):

grafana.vps-h1.infra.zintegrowana.online → 72.60.32.61 (A record, proxied)
monitor.vps-h1.infra.zintegrowana.online → 72.60.32.61 (A record, proxied)

3.7 Thanos — historical data on standby

Thanos sidecar on vps-i1 uploads completed TSDB blocks to Wasabi p24-infra bucket every 2 hours (--storage.tsdb.max-block-duration=2h). The standby Thanos-query (running on vps-h1) reads directly from the same Wasabi bucket.

Data gap: at most the current unuploaded 2-hour block. Prometheus on vps-h1 starts fresh for new scrapes; historical data > 2h old is fully available via Thanos-query.

4. Files changed

File	Change
`monitoring/docker-compose.standby.yml`	New override file
`monitoring/promtail/config-generic.yml`	New generic promtail config
`monitoring/Caddyfile.standby`	New stripped Caddyfile for standby
`infra-src/monitoring-watchdog/index.js`	New CF Worker scheduled cron
`infra-src/monitoring-watchdog/wrangler.toml`	CF Worker config + KV binding + cron schedule
`scripts/deploy-monitoring-standby.sh`	New deploy script
`scripts/teardown-monitoring-standby.sh`	New teardown script
`docs/playbooks/monitoring-standby-vps-h1.md`	New ops playbook
`monitoring/prometheus/prometheus.yml`	Add scrape target for vps-h1 node-exporter (existing — already scraped?)

DNS records to create manually in Cloudflare (human action):

grafana.vps-h1.infra.zintegrowana.online
monitor.vps-h1.infra.zintegrowana.online

5. Out of scope

Automatic DNS cutover (Grafana URL stays as-is per host; operators use standby URL)
Loki log history on standby (Loki uses local storage; log history is lost on vps-i1 failure — acceptable, Mezmo has full log retention)
Dual-primary monitoring (two Prometheuses scraping all targets simultaneously) — too complex for current fleet size
vps-h1 becoming permanent home of monitoring stack — standby only; vps-i1 remains primary

6. Risks

Risk	Mitigation
False-positive watchdog trigger (transient CF → vps-i1 routing blip)	Two consecutive ping failures required before triggering; KV state prevents flap
vps-h1 also down when standby deploy is triggered	Job goes to bms-4 worker, SSH to vps-h1 fails → worker marks job failed → alert fires
Docker not installed on vps-h1	Traefik runs there now, so Docker exists; verify docker compose V2 version in pre-deploy check
`sops` not installed on vps-h1	Add to vps-h1 provisioning; verify in pre-deploy check
Grafana on standby shows wrong `server_url` in alert emails	Alert email links point to grafana.vps-i1 (stale). Acceptable — links don’t work anyway since vps-i1 is down. Fix: make alert `source` URL a `MONITORING_DOMAIN` env var
standby not torn down when vps-i1 recovers	Recovery watchdog transition triggers teardown job; manual teardown playbook documented

7. Test plan

docker compose -f docker-compose.yml -f docker-compose.standby.yml config — validates merged compose with no errors
Deploy standby on vps-h1 manually (non-emergency): MONITORING_HOST=vps-h1 MONITORING_DOMAIN=vps-h1.infra.zintegrowana.online sops exec-env ...
Grafana accessible at https://grafana.vps-h1.infra.zintegrowana.online within 3 min of deploy
Thanos-query on vps-h1 returns data from Wasabi for queries > 2h old
CF Worker watchdog: stop Caddy on vps-i1 temporarily → confirm watchdog fires within 10 min → Discord alert received → standby job enqueued
Recovery: restart Caddy on vps-i1 → confirm watchdog detects recovery → teardown job enqueued
DNS records for vps-h1 subdomains pre-created in Cloudflare (human action before first real failover)

p24-infra Docs

Explorer

monitoring-standby-vps-h1

Monitoring Stack Standby on vps-h1

1. Problem statement

What already exists (reuse, don’t rebuild)

2. Root cause analysis — what blocks portability today

Blocker 1 — hostname hardcoded in three places

Blocker 2 — external Docker network not present on vps-h1

Blocker 3 — application services bundled into monitoring compose

Blocker 4 — Prometheus binds to 127.0.0.1 only

3. Design

3.1 `docker-compose.standby.yml` — override file for vps-h1

3.2 `monitoring/promtail/config-generic.yml`

3.3 `monitoring/Caddyfile.standby`

3.4 CF Worker monitoring-watchdog — external health check

3.5 P0 job handler — `deploy-monitoring-standby`

3.6 DNS failover

3.7 Thanos — historical data on standby

4. Files changed

5. Out of scope

6. Risks

7. Test plan

Graph View

Table of Contents

p24-infra Docs

Explorer

monitoring-standby-vps-h1

Monitoring Stack Standby on vps-h1

1. Problem statement

What already exists (reuse, don’t rebuild)

2. Root cause analysis — what blocks portability today

Blocker 1 — hostname hardcoded in three places

Blocker 2 — external Docker network not present on vps-h1

Blocker 3 — application services bundled into monitoring compose

Blocker 4 — Prometheus binds to 127.0.0.1 only

3. Design

3.1 docker-compose.standby.yml — override file for vps-h1

3.2 monitoring/promtail/config-generic.yml

3.3 monitoring/Caddyfile.standby

3.4 CF Worker monitoring-watchdog — external health check

3.5 P0 job handler — deploy-monitoring-standby

3.6 DNS failover

3.7 Thanos — historical data on standby

4. Files changed

5. Out of scope

6. Risks

7. Test plan

Graph View

Table of Contents

3.1 `docker-compose.standby.yml` — override file for vps-h1

3.2 `monitoring/promtail/config-generic.yml`

3.3 `monitoring/Caddyfile.standby`

3.5 P0 job handler — `deploy-monitoring-standby`