vps-i1-operations

vps-i1 — Operations Workbook

IONOS VPS (vps-i1). Primary infrastructure host: monitoring stack, Traccar GPS, OpenClaw WhatsApp gateway, GitHub Actions runner.

---

Architecture

IONOS VPS (217.154.82.162)   AlmaLinux 9.7
  CPU: AMD EPYC-Milan, 6 vCPUs
  RAM: 8 GB
  Role: monitoring host, GPS tracking, WhatsApp gateway, GH Actions runner
 
  Caddy (443/80) ─── TLS termination for all public endpoints
  │
  ├── monitoring-prometheus-1       :9090  (127.0.0.1 only)
  ├── monitoring-thanos-sidecar     :10901 (uploads blocks → Wasabi)
  ├── monitoring-thanos-query       :10904 (unified PromQL)
  ├── monitoring-alertmanager-1     :9093  (127.0.0.1 only)
  ├── monitoring-loki-1             :3100  (127.0.0.1 only)
  ├── monitoring-promtail-1         (log shipper, no public port)
  ├── monitoring-grafana-1          :3000  (127.0.0.1 only)
  ├── monitoring-blackbox-exporter  :9115
  ├── monitoring-queue-exporter     :9200
  ├── monitoring-pg-stats-exporter  :9201
  ├── monitoring-cost-exporter      :9210
  ├── monitoring-vercel-exporter    :9202
  ├── monitoring-backup-exporter    :9220
  ├── node_exporter                 :9100  (host network)
  ├── openclaw-openclaw-gateway-1   :18789-18790 (proxied via Caddy)
  ├── traccar                       :8082  (web), 5027/UDP (GPS)
  ├── traccar-db                    (MySQL, internal)
  └── status.vps-i1 (Uptime Kuma)  (proxied via Caddy)

Compose files:

Stack	File on server	File in repo
Monitoring	/opt/p24-infra/monitoring/docker-compose.yml	monitoring/docker-compose.yml
OpenClaw	/opt/p24-infra/openclaw/docker-compose.yml	openclaw/docker-compose.yml
Traccar	/opt/traccar/docker-compose.yml	not tracked

Public URLs:

Service	URL	Auth
Grafana	https://grafana.vps-i1.infra.zintegrowana.online	Grafana login
Prometheus	https://prometheus.vps-i1.infra.zintegrowana.online	basic_auth (PROMETHEUS_BASIC_AUTH_USER / PROMETHEUS_BASIC_AUTH_PASSWORD)
Alertmanager	https://alertmanager.vps-i1.infra.zintegrowana.online	basic_auth (PROMETHEUS_BASIC_AUTH_USER / PROMETHEUS_BASIC_AUTH_PASSWORD)
OpenClaw	https://openclaw.vps-i1.infra.zintegrowana.online	API key
Traccar	https://traccar.vps-i1.infra.zintegrowana.online	Traccar login
Status	https://status.vps-i1.infra.zintegrowana.online	Kuma login

---

SSH Access

User	Key	Scope
root	C:\Users\konar\.ssh\id_ed25519 (local workstation)	Full admin
claude-admin	GH Secret VPS_SSH_PRIVATE_KEY (ed25519)	Passwordless sudo: docker, systemctl, mkdir, chown, cp, tee

# Python paramiko — non-interactive SSH from Windows
import paramiko
client = paramiko.SSHClient()
client.set_missing_host_key_policy(paramiko.AutoAddPolicy())
client.connect("217.154.82.162", port=22, username="root",
               key_filename=r"C:\Users\konar\.ssh\id_ed25519", timeout=15)
stdin, stdout, stderr = client.exec_command("docker compose -f /opt/p24-infra/monitoring/docker-compose.yml ps")
print(stdout.read().decode())

# Direct SSH
ssh -i C:\Users\konar\.ssh\id_ed25519 root@217.154.82.162

---

Container Overview

Container	Purpose	Ports	Stack
monitoring-caddy-1	TLS reverse proxy	80, 443	monitoring
monitoring-grafana-1	Dashboards	127.0.0.1:3000	monitoring
monitoring-prometheus-1	Metrics collection, 15d TSDB	127.0.0.1:9090	monitoring
monitoring-thanos-sidecar	Uploads TSDB blocks → Wasabi	10901	monitoring
monitoring-thanos-query	Unified PromQL (local + Wasabi)	10904	monitoring
monitoring-alertmanager-1	Alert routing → email	127.0.0.1:9093	monitoring
monitoring-loki-1	Log aggregation (14d retention)	127.0.0.1:3100	monitoring
monitoring-promtail-1	Ships Docker logs → Loki	—	monitoring
monitoring-blackbox-exporter	HTTP/S probes	9115	monitoring
monitoring-queue-exporter	Supabase queue depths → Prometheus	9200	monitoring
monitoring-pg-stats-exporter	Supabase slow-query metrics → Prometheus	9201	monitoring
monitoring-cost-exporter	Vercel/Supabase/Wasabi spend → Prometheus	9210	monitoring
monitoring-vercel-exporter	Vercel deployment state → Prometheus	9202	monitoring
monitoring-backup-exporter	Backup freshness → Prometheus	9220	monitoring
node_exporter	Host OS metrics	9100 (host net)	systemd
openclaw-openclaw-gateway-1	WhatsApp gateway	18789-18790	openclaw
traccar	GPS tracking web UI	8082	traccar
traccar-db	MySQL for Traccar	internal	traccar

---

Config Management

File / Directory	In repo?	Notes
monitoring/docker-compose.yml	Yes	Full stack definition
monitoring/prometheus/	Yes	Scrape config + alert rules
monitoring/alertmanager/alertmanager.yml	Yes	Alert routing
monitoring/Caddyfile	Yes	Reverse proxy + TLS
monitoring/.env	No (.env.example)	Secrets — on server only
openclaw/docker-compose.yml	Yes	OpenClaw stack
ansible/playbooks/provision-new-vps.yml	Yes	Full server provisioning

Apply config change

# On vps-i1 — pull latest and reload affected service
cd /opt/p24-infra
git pull
 
# Hot-reload Prometheus rules (no restart)
curl -X POST http://localhost:9090/-/reload
 
# Hot-reload Alertmanager
curl -X POST http://localhost:9093/-/reload
 
# Caddy config reload (no restart)
docker compose -f monitoring/docker-compose.yml exec caddy caddy reload --config /etc/caddy/Caddyfile
 
# Full restart of a service
cd /opt/p24-infra/monitoring
docker compose restart <service>

---

Backup

Data	Method	Schedule	Destination
Prometheus TSDB	Thanos sidecar continuous upload	Every 2h (block upload)	s3://ecotrans-monitoring/ (Wasabi eu-central-1)
Config + rules	Git push	On every commit	GitHub radieu/p24-infra
SSH root key	GH Secret VPS_ROOT_SSH_KEY (base64-encoded)	Manual, on rotation	GitHub Secrets
Traccar DB	Backup script (if configured)	See traccar-operations.md	Wasabi
OS-level config	Not backed up — rebuild from Ansible	—	Ansible playbook in repo
Docker volumes (stateless)	N/A — ephemeral by design	—	—
Caddy TLS certs (caddy_data)	Not backed up — auto-renewed	—	Re-provisioned on restart
Alertmanager silences	Not backed up	—	Ephemeral — acceptable gap

Server rebuild source of truth: ansible/playbooks/provision-new-vps.yml — provisions full OS baseline, installs Docker, sets up claude-admin, deploys systemd units.

---

Restore

Scenario 1: Service crash (server intact)

# Monitoring stack
cd /opt/p24-infra/monitoring
docker compose up -d
 
# OpenClaw
cd /opt/p24-infra/openclaw
docker compose up -d

Scenario 2: Full server rebuild

# 1. Provision new IONOS VPS (same or replacement IP)
# 2. Run Ansible playbook from local workstation
ansible-playbook ansible/playbooks/provision-new-vps.yml -i <new-ip>,
 
# 3. Clone repo on new server
ssh root@<new-ip> "git clone https://github.com/radieu/p24-infra /opt/p24-infra"
 
# 4. Restore .env files from local .env.local
scp -i C:\Users\konar\.ssh\id_ed25519 monitoring/.env root@<new-ip>:/opt/p24-infra/monitoring/.env
 
# 5. Start stacks
ssh root@<new-ip> "cd /opt/p24-infra/monitoring && docker compose up -d"
ssh root@<new-ip> "cd /opt/p24-infra/openclaw && docker compose up -d"
 
# 6. Restore Prometheus TSDB from Wasabi (if needed — see monitoring-stack-operations.md)

Estimated RTO: ~30 minutes for full service restore (Ansible ~10min, stacks up ~5min, Prometheus data pull optional).

Scenario 3: Prometheus data loss only

See docs/monitoring-stack-operations.md — Restore from Wasabi section.

---

Healthcheck / Monitoring

Check	Method	Alert
Host reachability	Prometheus job node scrapes 217.154.82.162:9100	ServerDown fires after 5m
Container health	Docker healthcheck: directives on all monitoring containers	ContainerCrashLooping rule
Disk usage	node_exporter → LowDisk rule (< 10% free)	LowDisk fires
Memory usage	node_exporter → HighMemory rule (> 90%)	HighMemory fires
CPU usage	node_exporter → HighCPU rule (> 80% 5m avg)	HighCPU fires
SSH auth failures	/var/log/secure via node_exporter	SSHAuthFailures rule

Manual check:

# Container status
ssh -i C:\Users\konar\.ssh\id_ed25519 root@217.154.82.162 \
  "cd /opt/p24-infra/monitoring && docker compose ps"
 
# node_exporter up?
curl http://217.154.82.162:9100/metrics | head -5

---

Password Rotation

SSH key rotation (root + claude-admin)

Rotation frequency: 365 days. Last rotated: see docs/secrets-rotation-log.md.

# 1. Generate new key pair (on local workstation)
ssh-keygen -t ed25519 -f C:\Users\konar\.ssh\id_ed25519_new -C "vps-i1-root-$(date +%F)"
 
# 2. Add new public key to authorized_keys on vps-i1
ssh -i C:\Users\konar\.ssh\id_ed25519 root@217.154.82.162 \
  "echo '<new-public-key>' >> /root/.ssh/authorized_keys"
 
# 3. Verify new key works
ssh -i C:\Users\konar\.ssh\id_ed25519_new root@217.154.82.162 "hostname"
 
# 4. Remove old key
ssh -i C:\Users\konar\.ssh\id_ed25519_new root@217.154.82.162 \
  "sed -i '/<old-key-fingerprint>/d' /root/.ssh/authorized_keys"
 
# 5. Replace local key
mv C:\Users\konar\.ssh\id_ed25519_new C:\Users\konar\.ssh\id_ed25519
mv C:\Users\konar\.ssh\id_ed25519_new.pub C:\Users\konar\.ssh\id_ed25519.pub
 
# 6. Update GH Secret VPS_ROOT_SSH_KEY (base64)
$key = [Convert]::ToBase64String([IO.File]::ReadAllBytes("C:\Users\konar\.ssh\id_ed25519"))
gh secret set VPS_ROOT_SSH_KEY -b $key -R radieu/p24-infra
 
# 7. Update VPS_SSH_PRIVATE_KEY for claude-admin separately if different key
# 8. Log in docs/secrets-rotation-log.md

Grafana admin / Prometheus basic_auth password

See docs/grafana-operations.md and docs/monitoring-stack-operations.md.

---

Known Limitations

No IPv6 — permanent constraint

IONOS CoreVPS (this product tier) does not support IPv6. Confirmed 2026-06-20 via IONOS CloudPanel → Network → Public IP: only IPv4 (217.154.82.162) is available. There is no IPv6 option in the panel. This is a product-level limitation, not a configuration gap — it cannot be fixed without migrating to a different hosting provider.

Impact: Supabase’s direct database host (db.mwkqmgadqnkkihjdeqsi.supabase.co) resolves to an IPv6-only address (2a05:d014:1c06:5f0d:1845:c885:3940:16ec). vps-i1 cannot reach it directly.

---

Supabase DB Connectivity (via bms-4 socat proxy)

vps-i1 has no IPv6. bms-4 (OVH) has native IPv6 and acts as a TCP proxy:

Grafana / pg-stats-exporter (vps-i1, IPv4 only)
  → TCP IPv4 → bms-4:15432
    → socat → db.mwkqmgadqnkkihjdeqsi.supabase.co:5432 (IPv6)
      (SSL is end-to-end — bms-4 sees only encrypted bytes)

Verified working 2026-06-27: grafana_readonly auth succeeds, SELECT current_user returns grafana_readonly.

Connection details

Setting	Value
Host (from vps-i1)	54.36.123.110 (bms-4)
Port	15432 (socat listener)
Username	grafana_readonly
SSL	sslmode=require (TLS terminates at Supabase, not bms-4)

Grafana datasource config: monitoring/grafana/provisioning/datasources/supabase.yml.

bms-4 proxy service

The canonical :15432 proxy is socat-supabase.service — repo-tracked at bms-4/socat-supabase.service and protected by the p24-socat-watchdog.timer watchdog (see docs/playbooks/socat-supabase-watchdog.md).

Retired duplicate (#2309): supabase-pg-proxy.service was a second unit binding the same :15432 listener and the same upstream. Because socat-supabase.service already holds the port, it could never start and crash-looped (52k+ restarts). The watchdog now disables + masks it automatically. Do not re-create it — manage the proxy via socat-supabase.service only.

Item	Value
Service	socat-supabase.service (systemd, bms-4)
Command	socat TCP4-LISTEN:15432,bind=0.0.0.0,reuseaddr,fork TCP6:db.mwkqmgadqnkkihjdeqsi.supabase.co:5432,connect-timeout=10
Firewall	iptables: port 15432 allowed from 217.154.82.162 (vps-i1) only
Status	active; watchdog-protected (port + zombie check every 5 min)

# Manage on bms-4
ssh root@54.36.123.110 "systemctl status socat-supabase"
ssh root@54.36.123.110 "systemctl restart socat-supabase"
ssh root@54.36.123.110 "journalctl -u socat-supabase --tail=20"
# If 'Address already in use': fuser -k 15432/tcp && systemctl restart socat-supabase

Deployment steps (after merging PR #1739)

1. Merge PR #1739 (datasource config updated; credentials already rotated in SOPS).
2. Restart Grafana on vps-i1 to pick up the new supabase.yml (provisioning is read at startup): ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose restart grafana"

Why not Supavisor shared pooler

aws-1-eu-central-1.pooler.supabase.com:5432 accepts IPv4 but caches pg_authid for custom roles — a ALTER ROLE ... WITH PASSWORD change is invisible to the pooler until Supabase Support restarts it. Not acceptable for operational autonomy.

---

Troubleshooting

Symptom	Likely cause	Fix
docker compose ps shows container Exit 1	Bad config or missing env var	docker compose logs <container>
Prometheus targets all DOWN	Prometheus itself restarted	docker compose restart prometheus
Caddy 502	Upstream container not running	docker compose up -d <service>
SSH connection refused	sshd crashed or firewall changed	Console login via IONOS panel → systemctl restart sshd
Disk full	Log accumulation or Prometheus TSDB	docker system prune -f; extend volume if needed
node_exporter unreachable	Systemd service stopped	systemctl restart node_exporter