p24-infra Docs

monitoring-stack-operations

12 min read

Monitoring Stack — Operations Workbook

Covers: Prometheus, Thanos (sidecar + query), Alertmanager, Loki, Promtail, Blackbox Exporter, Caddy — all running on IONOS VPS (vps-i1).

Architecture

IONOS VPS (217.154.82.162)  ─── Caddy (443 TLS) ─── public HTTPS endpoints
│
├── prometheus:9090          Metrics collection, 15d local TSDB
│   └── thanos-sidecar:10901 Uploads 2h TSDB blocks → Wasabi ecotrans-monitoring
│
├── thanos-query:10904       Unified PromQL: local + Wasabi long-term
├── alertmanager:9093        Alert routing → email (Mailgun EU)
├── loki:3100                Log aggregation (14-day retention)
├── promtail                 Ships Docker logs → Loki
├── blackbox-exporter:9115   HTTP/HTTPS probes (synthetic checks)
└── caddy:80/443             TLS termination for all above
 
Long-term storage: Wasabi s3://ecotrans-monitoring (eu-central-1)

Compose file: /opt/p24-infra/monitoring/docker-compose.yml
Public URLs:

ServiceURL
Prometheushttps://prometheus.vps-i1.infra.zintegrowana.online
Alertmanagerhttps://alertmanager.vps-i1.infra.zintegrowana.online

Both are protected by Caddy basic_auth. The credential is SOPS-managed, not hard-coded in the Caddyfile: the prometheus/alertmanager vhosts reference {$PROMETHEUS_BASIC_AUTH_USER} {$PROMETHEUS_BASIC_AUTH_HASH}, sourced from the on-server monitoring/.env that Caddy loads (env_file: .env). The trio lives in secrets/monitoring.env.sops:

KeyPurpose
PROMETHEUS_BASIC_AUTH_USERbasic_auth username (admin)
PROMETHEUS_BASIC_AUTH_HASHbcrypt hash Caddy verifies against
PROMETHEUS_BASIC_AUTH_PASSWORDplaintext password (for human/curl access)

A single trio is shared by both vhosts (they previously used an identical hard-coded hash). To read the password for a manual curl, decrypt the one key silently — never print it:

PW=$(sops -d --input-type dotenv --output-type dotenv secrets/monitoring.env.sops \
  | grep '^PROMETHEUS_BASIC_AUTH_PASSWORD=' | cut -d= -f2-)
curl -u "admin:$PW" https://prometheus.vps-i1.infra.zintegrowana.online/api/v1/status/config
unset PW

To rotate: regenerate password + bcrypt hash, sops set all three keys in secrets/monitoring.env.sops (use --input-type dotenv --output-type dotenv), let CI sync the on-server monitoring/.env from SOPS before reloading Caddy, then docker compose up -d caddy. Grafana datasources reach Prometheus over the internal docker network (http://prometheus:9090, no basic_auth) and are unaffected by a password change.

Config Management

FileIn repo?Purpose
prometheus/prometheus.yml✅Scrape targets
prometheus/rules/*.yml✅Alert rules
prometheus/blackbox.yml✅Blackbox probe config
alertmanager/alertmanager.yml✅Alert routing + receivers
loki/loki-config.yml✅Loki storage + retention
promtail/config-vps-i1.yml✅Log scrape config
Caddyfile✅Reverse proxy + TLS
thanos/s3.yml✅ templateWasabi S3 config (from template + .env)
.env❌ (.env.example)Secrets

Updating alert rules (hot reload)

# Edit monitoring/prometheus/rules/*.yml → commit → on vps-i1:
git pull
curl -X POST http://localhost:9090/-/reload
# No restart needed

Updating Alertmanager config

# Edit monitoring/alertmanager/alertmanager.yml → commit → on vps-i1:
git pull
curl -X POST http://localhost:9093/-/reload

Deployment

Full stack bring-up

cd /opt/p24-infra/monitoring
docker compose up -d

Restart individual service

docker compose restart prometheus
docker compose restart alertmanager
docker compose restart caddy

Check stack health

cd /opt/p24-infra/monitoring
docker compose ps
docker compose logs --tail=30 prometheus

Backup

What needs backing up

DataBackup methodScheduleDestination
Prometheus TSDBThanos sidecar → WasabiContinuous (2h blocks)s3://ecotrans-monitoring/
Alertmanager silencesNot backed up—Gap — silences are ephemeral
Caddy TLS certscaddy_data volume — not backed up—Gap — auto-renewed via ACME
Prometheus config + rulesGit repoOn pushGitHub

Caddy certs note: If caddy_data is lost, Caddy will re-request Let’s Encrypt certificates automatically on restart. Brief downtime (~1 min) during renewal. Not a data-loss risk.

Manual Prometheus backup (emergency — force Thanos upload)

# Trigger Thanos compaction to flush any pending blocks
docker run --rm \
  -v /opt/p24-infra/monitoring/thanos/s3.yml:/s3.yml:ro \
  quay.io/thanos/thanos:v0.36.1 \
  compact --objstore.config-file /s3.yml --wait

Restore

Prometheus — Restore from Wasabi

# 1. List available blocks
docker run --rm \
  -v /opt/p24-infra/monitoring/thanos/s3.yml:/s3.yml:ro \
  quay.io/thanos/thanos:v0.36.1 \
  tools bucket ls --objstore.config-file /s3.yml
 
# 2. Stop Prometheus and Thanos sidecar
cd /opt/p24-infra/monitoring
docker compose stop thanos-sidecar prometheus
 
# 3. Restore specific block
docker run --rm \
  -v /opt/p24-infra/monitoring/thanos/s3.yml:/s3.yml:ro \
  -v prometheus_data:/prometheus \
  quay.io/thanos/thanos:v0.36.1 \
  tools bucket rewrite --objstore.config-file /s3.yml \
  --id <BLOCK_ULID> --output-dir /prometheus
 
# 4. Start Prometheus (Thanos will continue uploading)
docker compose up -d prometheus thanos-sidecar

Caddy — Fresh cert after volume loss

# Just restart — Caddy auto-renews
docker compose up -d caddy
# Monitor logs during first start
docker compose logs -f caddy

Loki — Data loss is acceptable

Loki stores logs with 14-day retention. On a fresh start, log history is empty — only new logs will appear. This is acceptable by design.

Healthchecks

All services have Docker healthcheck: directives (added 2026-05-14):

ServiceCheck endpointInterval
prometheus/-/healthy30s
thanos-sidecar/-/healthy (port 10902)30s
thanos-query/-/healthy (port 10904)30s
alertmanager/-/healthy30s
loki/ready30s
promtail/ready (port 9080)30s
blackbox-exporter/health30s
caddy/config/ (admin port 2019)30s

External probes: Prometheus infrastructure.yml rules fire ServerDown within 2 min.

Alert Rules Reference

Rule fileKey alerts
infrastructure.ymlServerDown, ContainerCrashLooping, LowDisk, HighMemory, HighCPU
backups.ymlBackupStale (>26h), BackupSizeRegression
synthetic.ymlEndpointDown, EndpointSlow
security.ymlSSHAuthFailures
costs.ymlVercelApproachingFreeTier, SupabaseDbSizeApproachingPro
queues.ymlTranscriptionQueueCritical
loki.ymlLokiIngestionStopped
n8n.ymlN8nWorkflowFailed, N8nSnapshotStale

Password Rotation

basic_auth (Prometheus + Alertmanager public URLs)

Caddy basic_auth for these two routes is SOPS-managed via the PROMETHEUS_BASIC_AUTH_{USER,HASH,PASSWORD} trio in secrets/monitoring.env.sops (the Caddyfile references {$PROMETHEUS_BASIC_AUTH_USER} {$PROMETHEUS_BASIC_AUTH_HASH} — no hash is hard-coded). It is independent of GRAFANA_ADMIN_PASSWORD. Rotate via:

# 1. Generate a new password + matching bcrypt hash, write all three keys into SOPS
#    (values stay in-memory — never echo them):
NEW_PASS=$(python3 -c 'import secrets; print(secrets.token_urlsafe(24))')
NEW_HASH=$(python3 -c 'import bcrypt,os; print(bcrypt.hashpw(os.environ["NEW_PASS"].encode(), bcrypt.gensalt(rounds=14, prefix=b"2a")).decode())' NEW_PASS="$NEW_PASS")
# (or: docker run --rm caddy:2.8-alpine caddy hash-password --plaintext "$NEW_PASS")
sops set --input-type dotenv --output-type dotenv secrets/monitoring.env.sops '["PROMETHEUS_BASIC_AUTH_PASSWORD"]' "$(python3 -c 'import json,os;print(json.dumps(os.environ["NEW_PASS"]))' NEW_PASS="$NEW_PASS")"
sops set --input-type dotenv --output-type dotenv secrets/monitoring.env.sops '["PROMETHEUS_BASIC_AUTH_HASH"]'     "$(python3 -c 'import json,os;print(json.dumps(os.environ["NEW_HASH"]))' NEW_HASH="$NEW_HASH")"
unset NEW_PASS NEW_HASH
 
# 2. Let CI secrets-sync regenerate the on-server monitoring/.env from SOPS BEFORE reloading Caddy.
# 3. On vps-i1:
docker compose up -d caddy

No Caddyfile edit is needed — only the SOPS keys change. Grafana datasources use the internal docker network (no basic_auth) and are unaffected.

Troubleshooting

SymptomCauseFix
Prometheus targets show DOWNExporter container restarteddocker compose restart <exporter>
Thanos upload stalledWasabi connectivity issueCheck docker compose logs thanos-sidecar; verify S3 creds in .env
Alertmanager not sending emailSMTP config wrongcurl -X POST http://localhost:9093/-/reload; check Mailgun dashboard
Caddy TLS renewal failedRate limit or DNS not resolvingCheck Caddy logs; verify DNS wildcard record
Loki not receiving logsPromtail cannot reach Lokidocker compose restart promtail; check loki_data volume space
blackbox-exporter probe failsTarget unreachableVerify URL + Caddy config for the target service
queue-exporter scrapes 0 queuesAll rows active=false in registrySELECT * FROM dev_r_exporters_queues; — set active=true for desired rows
queue-exporter query errorRLS policy on target table blocks grafana_readonlyGrant SELECT on the table; check for RLS policies referencing other tables
pg-stats-exporter connection errorIPv6 DNS / wrong DB hostEnsure SUPABASE_DB_HOST is the session pooler (aws-1-eu-central-1.pooler.supabase.com), not db.*.supabase.co

Custom Exporters

All custom exporters run as Docker containers on vps-i1 (IONOS), built from monitoring/exporters/. Each exposes a /metrics endpoint scraped by Prometheus every 60s.

ExporterPortSourceWhat it publishes
queue-exporter:9200Supabase DB (psycopg2, grafana_readonly)Queue depths by status for tables in dev_r_exporters_queues
pg-stats-exporter:9201Supabase DB (psycopg2, grafana_readonly)Top-200 slowest queries from extensions.pg_stat_statements
cost-exporter:9210Vercel API + Supabase mgmt API + Wasabi S3Monthly spend / usage per service (daily refresh)
vercel-exporter:9202Vercel APIDeployment state per project (every 5m)
backup-exporter:9220/opt/backups/backup-status.promBackup age and size freshness

Rebuild after code change

cd /opt/p24-infra/monitoring
git pull
docker compose up -d --no-deps --build queue-exporter   # or whichever exporter

Queue Exporter — Managing Monitored Tables

The queue-exporter does not have a hardcoded list of tables. It reads dev_r_exporters_queues from Supabase on every poll cycle (60s). Changing which tables are monitored requires only a SQL row change — no code change, no redeploy.

Table schema

SELECT id, table_name, schema_name, label, status_column, active, notes
FROM dev_r_exporters_queues
ORDER BY label;
ColumnPurpose
table_namePostgres table to GROUP BY status_column
schema_nameSchema, default public
labelPrometheus label value for the queue dimension
status_columnColumn to group by, default status
activetrue = scrape each cycle; false = skip
notesFree text — why it’s there or why it’s paused

Add a new queue

INSERT INTO dev_r_exporters_queues (table_name, schema_name, label, status_column, active, notes)
VALUES ('my_jobs', 'public', 'my_jobs', 'status', true, 'Job processing queue added YYYY-MM-DD');

Also grant grafana_readonly SELECT on the table:

GRANT SELECT ON public.my_jobs TO grafana_readonly;

The exporter picks it up within 60s — no restart needed.

Pause a queue (keep row, stop scraping)

UPDATE dev_r_exporters_queues SET active = false WHERE table_name = 'my_jobs';

Remove a queue permanently

DELETE FROM dev_r_exporters_queues WHERE table_name = 'my_jobs';

Current registered queues

SELECT table_name, label, active, notes FROM dev_r_exporters_queues ORDER BY active DESC, label;

Permissions note

The exporter connects as grafana_readonly via the Supabase session pooler. If a table has RLS policies that reference other tables (e.g. profiles), the query will fail with permission denied. Fix: either grant SELECT on the referenced table too, or create a SECURITY DEFINER view and query that instead.

NC-Alert Real-Time Dispatch Architecture

Issues labelled p24-infra-nc-alert are automatically investigated by the nc-alert orchestrator worker skill. This section documents the end-to-end pipeline.

Flow overview

Alert fires (Prometheus / Cloudflare / DB-maintenance / nightly-infra-check)
   │
   ▼
Script/workflow creates GitHub issue
  - Labels: p24-infra-nc-alert, bug
  - Body contains:
      <!-- affected-service: {slug} -->   ← normalized service slug for sibling detection
      <!-- alert-source: prometheus|cloudflare|db-maintenance|nightly-infra-check -->
   │
   ▼
GH Actions: nc-alert-instant-dispatch.yml triggers on issues:labeled (p24-infra-nc-alert)
  - 5-minute Supabase debounce: checks dev_r_worker_queue for active nc-alert-batch rows
  - If none active: dispatches nc-alert-batch (issue_number=0 sentinel) to meta-dispatcher
    with server_preference=vps-i1 (#2188)
  - Meta-dispatcher CF Worker handles issue_number=0 via isNcAlertBatch guard (no GitHub lookup)
   │
   ▼
Worker picks up nc-alert-batch job → runs /nc-alert-orchestrator skill
  - Pinned to vps-i1 (#2188): the queue-dispatcher only lets vps-i1 claim nc-alert-batch rows,
    so the orchestrator's primary check `GET http://localhost:9090/api/v1/alerts` always works
    (no auth). bms-4 cannot reach Prometheus — port 9090 firewalled, and the public Caddy
    basic_auth password has no decryptable SOPS key — so it is excluded at claim time.
  - Claims all open p24-infra-nc-alert issues (excluding human-action)
  - Investigates each by alert-source (Prometheus API / CF API / Supabase REST)
  - Posts structured Investigation comment per issue
  - Capped at 3 iterations (metadata.iteration field in queue row)

Alert sources and metadata tags

SourceFileaffected-service formatalert-source tag
Prometheusscripts/prometheus-alerts-ai-triage.py{server}-{job} (e.g. vps-i1-node-exporter)prometheus
Cloudflare.github/workflows/cloudflare-security-check.ymlcloudflare-securitycloudflare
DB maintenance.github/workflows/db-maintenance-check.ymlsupabase-fleet-positionsdb-maintenance
Nightly infra check.claude/commands/nightly-infra-check.md{server}-{component}nightly-infra-check

Sibling detection

When a p24-infra-nc-alert issue already exists for the same affected-service slug, alert sources comment on the existing issue instead of opening a duplicate. The Cloudflare workflow implements sibling detection explicitly (checks open issues for matching <!-- affected-service: cloudflare-security -->). The Prometheus script deduplicates via sha1(alertname|instance|job) in the issue body.

Meta-dispatcher fix: issue_number=0 sentinel

The meta-dispatcher CF Worker previously rejected issue_number=0 due to a falsy check. The fix in infra-src/meta-dispatcher/src/index.ts explicitly checks body.issue_number === undefined || body.issue_number === null (not !body.issue_number). The isNcAlertBatch guard skips the GitHub issue API lookup and routes to insertQueueRow with synthetic metadata.

The dedup guard in insertQueueRow detects batch jobs by job_type === "nc-alert-batch" and calls shouldSkipBatchJobDispatch instead of the per-issue dedup — preventing duplicate orchestrators without requiring a valid issue number.

NC-alert orchestrator worker skill

File: .claude/commands/nc-alert-orchestrator.md

The skill is invoked by the worker queue when job_type=nc-alert-batch. It:

  1. Reads its own queue row to get metadata.iteration (caps at 3)
  2. Lists all open p24-infra-nc-alert issues (excluding human-action)
  3. Parses alert-source and affected-service tags from each issue body
  4. Calls the appropriate API for each source (Prometheus /api/v1/alerts, CF security insights, Supabase pg_stat_user_tables)
  5. Posts a structured ### Investigation comment — never closes issues
  6. After processing, re-checks for new nc-alert issues that arrived during investigation and cascades if needed

GH Secrets required

SecretPurpose
QUEUE_API_URLMeta-dispatcher CF Worker URL
QUEUE_API_KEYMeta-dispatcher auth bearer token
SUPABASE_URLDebounce check in nc-alert-instant-dispatch.yml
SUPABASE_SERVICE_KEYDebounce check auth

QUEUE_API_URL and QUEUE_API_KEY must be added to radieu/p24-infra GH Secrets after merge (human action).

Worker-Failure Alert → GitHub Bridge (#1819, #1960)

n8n workflow on bms-4 (alertmanager-to-gh-comment, id aR156ACui4i6lqgE) that converts permanent worker-queue failures into actionable GitHub state: a comment on the affected issue plus the human-action label.

#1960: the legacy Path B (Discord-bot polling of #infra-alerts every 5 min using DISCORD_BOT_TOKEN) has been removed. It was never deployed on bms-4 (DISCORD_BOT_TOKEN was missing) and added a ~5-min delay plus fragile embed parsing. It is replaced by Path C: a native Alertmanager → n8n webhook push. DISCORD_BOT_TOKEN no longer has any consumer.

JSON source of truth: bms-4/n8n-workflows/alertmanager-to-gh-comment.json

Flow overview

permanent worker-queue failure
   │
   ├──(A) queue-retry-worker.py push ────────────────────────┐  ← token-free, instant
   │     POST .../webhook/retry-worker-permanent-failure      │
   │     body: {issue_number, retry_count, error_message,     │
   │            source:"retry-worker"}                         │
   │                                                           ▼
   ├──(C) Alertmanager webhook push (#1960) ─────────────►  Has Alert? (IF, has_alert == "yes")
   │     receiver `worker-failure-gh` POSTs native           │ yes
   │     Alertmanager JSON to                                ▼
   │     .../webhook/alertmanager-worker-failure       Build GH Comment (Code)
   │     Parse Alertmanager Alert reads alerts[],            │  fan-out (quirk 3)
   │     extracts labels.issue_number / annotations          ├──► POST /issues/{n}/comments
   │                                                          └──► POST /issues/{n}/labels ['human-action']
   │
   On any node error ──► On Error (Error Trigger) ──► Build Discord Error ──► infra-scripts error webhook

Two ingestion paths feed one shared comment/label chain:

  • Path A — retry-worker webhook push. queue-retry-worker.py POSTs the failure JSON to the retry-worker-permanent-failure webhook. Unchanged by #1960.

  • Path C — Alertmanager webhook push (#1960). The Alertmanager worker-failure-gh receiver (route category = worker-failure, continue: true, group_wait: 0s) POSTs the native Alertmanager payload to the alertmanager-worker-failure webhook. Parse Alertmanager Alert iterates alerts[], keeps status == "firing", and extracts labels.issue_number (fallback annotations.issue_number), retry_count, and the error text from annotations.description || summary. The Alertmanager per-alert fingerprint is used as the GitHub idempotency marker. No Discord bot token, no polling latency. continue: true means the same alert still reaches the email / n8n-incident receivers, so Discord delivery is unchanged.

    For Path C to fire end-to-end, the worker-permanent-failure alert rule must carry the labels category: worker-failure and issue_number: "<n>" (and may set annotations.retry_count).

Environment variables (bms-4 n8n)

VarUsed byStatus
GH_TOKENGitHub comment + label callspresent in secrets/n8n-bms4.env.sops
GH_ALERT_REPOtarget repo (optional)defaults to radieu/p24-infra if unset
P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URLself-failure error embedpresent

DISCORD_BOT_TOKEN and DISCORD_INFRA_ALERTS_CHANNEL_ID are no longer used by this workflow (#1960). DISCORD_BOT_TOKEN is now deprecated — see docs/discord-notifications.md.

Activation prerequisites (workflow imported inactive)

The workflow is imported but not active. Before activating:

  1. #1811 deployed — queue-retry-worker.py must be live so failure alerts actually fire.
  2. Path A — ✅ implemented (#1887): queue-retry-worker.py now POSTs each permanently-failed row (retry_count >= max_retries) to the retry-worker-permanent-failure webhook (notify_permanent_failure(), env-overridable via N8N_RETRY_FAILURE_WEBHOOK_URL). The push is idempotent — guarded by a metadata.permanent_failure_alerted flag — so the 10-min timer fires the alert exactly once per row instead of re-commenting every cycle (the workflow does no GitHub-side dedup). No secrets needed. Still requires the workflow to be activated in n8n.
  3. Path C (#1960): deploy the updated alertmanager.yml.tpl to vps-i1 (adds the worker-failure-gh receiver + category = worker-failure route) and ensure the worker-permanent-failure alert rule emits the category/issue_number labels above. No secrets needed — the n8n webhook is unauthenticated and internal-only.
  4. Error workflow: to route this workflow’s own failures to the Error Trigger, set its Settings → Error Workflow to itself in the n8n UI (or via the API settings.errorWorkflow).

Operations

# Import / update from the JSON source of truth (bms-4):
KEY=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops | grep '^BMS4_N8N_API_KEY=' | cut -d= -f2-)
HOST=$(sops -d --input-type dotenv --output-type dotenv secrets/n8n-bms4.env.sops | grep '^BMS4_N8N_HOST=' | cut -d= -f2-)
curl -s -X PUT -H "X-N8N-API-KEY: $KEY" -H "Content-Type: application/json" \
  --data @bms-4/n8n-workflows/alertmanager-to-gh-comment.json \
  "${HOST%/}/api/v1/workflows/aR156ACui4i6lqgE"; unset KEY HOST
 
# Smoke-test Path C with a synthetic Alertmanager payload (substitute a real test issue number):
curl -s -X POST https://n8n.bms-4.infra.zintegrowana.online/webhook/alertmanager-worker-failure \
  -H "Content-Type: application/json" \
  -d '{"status":"firing","alerts":[{"status":"firing","fingerprint":"test123","labels":{"alertname":"WorkerPermanentFailure","category":"worker-failure","issue_number":"0"},"annotations":{"description":"synthetic test"}}]}'

n8n 2.26.x quirks applied (see docs/playbooks/n8n/n8n-http-request-quirks.md): HTTP body uses specifyBody=json + jsonBody; the routing IF node is typeVersion: 1 with string equality (has_alert == "yes"); comment and label calls fan out in parallel from the Code node to avoid $json being overwritten by the first HTTP response.

Graph View

Backlinks