p24-infra Docs

bms1-blackbox-config-deploy

4 min read

Playbook: Deploy / reload the bms-1 blackbox-exporter config

Status: ACTIVE (promoted from DRAFT 2026-07-12) Created: 2026-07-12 Origin incident: #4023 — WkhtmlProbeExporterDown fired because the tcp_connect module change from #4004 was never deployed to bms-1. Related: pinbox24-wkhtml-network-isolation.md, p4-ovh-bms-1-ns367522-operations.md

Why this playbook exists

bms-1/blackbox-bms1.yml and bms-1/docker-compose.yml are the two files bind-mounted into the /opt/mezmo-agent/ compose stack on bms-1. Since #4025 (2026-07-12) they are auto-deployed by .github/workflows/deploy-bms1-configs.yml on every push to main that touches either path — the workflow scps the file to bms-1, installs it into /opt/mezmo-agent/ with a timestamped .bak, runs the compose recreate on the affected service, and verifies the blackbox health endpoint returns HTTP 200 before exiting green.

bms-1 Compose version (#4594): bms-1 runs Docker 20.10.2 with only legacy docker-compose v1 (/usr/local/bin/docker-compose, 1.25.3) — it has no Compose v2 plugin, so docker compose (spaced) fails there with exit 125 ('compose' is not a docker command). The workflow detects the available command at runtime (docker compose version probe → falls back to docker-compose); the manual commands below hardcode docker-compose for the same reason.

Before #4025 there was no automated deployment path:

  • deploy-monitoring-config.yml ships only monitoring/prometheus/** to vps-i1.
  • secrets-sync.yml’s sync-bms1-all job ships only env files to bms-1, not this config.

That gap is what caused #4023 (http_connected → tcp_connect module change in #4004 was committed to git but never reached the running exporter → HTTP 400 on /probe → alert fired).

This playbook is now the manual fallback — use it when the CI path itself is broken (runner offline, VPS_ROOT_SSH_KEY invalid, etc.) or when patching an out-of-band change that never made it into main.

Where things actually live on bms-1

  • The bms-1 monitoring compose project is deployed at /opt/mezmo-agent/ (NOT /opt/p24-infra/, which is not a git checkout on bms-1).
  • The exporter reads its config from a bind mount: ./blackbox-bms1.yml:/etc/blackbox_exporter/config.yml:ro → host path /opt/mezmo-agent/blackbox-bms1.yml.
  • Prometheus on vps-i1 scrapes it remotely at 94.23.26.113:9115, job blackbox_bms1, requesting module=tcp_connect.

Symptom → this playbook

  • WkhtmlProbeExporterDown firing; up{job="blackbox_bms1"} = 0.
  • The exporter is reachable but the scrape fails with HTTP 400 (server returned HTTP status 400 Bad Request) — blackbox-exporter returns 400 for /probe when the requested module is not defined in its loaded config. This is drift, not an unreachable/down exporter.

Diagnosis (read-only)

ssh -i ~/.ssh/vps_root_key root@94.23.26.113 \
  "curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:9115/probe?module=tcp_connect&target=wkhtml-v42-prod:80'"
# 400 = module missing from loaded config → drift, apply the fix below
# 200 = healthy, module present → look elsewhere
 
# Confirm which modules the running container actually loaded:
ssh -i ~/.ssh/vps_root_key root@94.23.26.113 \
  "docker exec blackbox-exporter-bms1 sed -n '/^modules:/,\$p' /etc/blackbox_exporter/config.yml | grep -E 'prober:|^  [a-z_]+:'"

Fix — redeploy the repo config and reload the exporter

Run from a p24-infra worker checkout (the repo copy is the source of truth):

# 1. Back up the current (drifted) host config, then push the repo version.
scp -i ~/.ssh/vps_root_key bms-1/blackbox-bms1.yml root@94.23.26.113:/opt/mezmo-agent/blackbox-bms1.yml.new
ssh -i ~/.ssh/vps_root_key root@94.23.26.113 '
  cd /opt/mezmo-agent
  cp -a blackbox-bms1.yml "blackbox-bms1.yml.bak.$(date +%Y%m%d%H%M%S)"
  mv blackbox-bms1.yml.new blackbox-bms1.yml
  # 2. Recreate so the exporter re-reads the bind-mounted config.
  # NOTE: bms-1 has Docker 20.10.2 with only legacy Compose v1 — use
  # `docker-compose` (hyphenated), NOT `docker compose` (v2, absent here → exit 125, #4594).
  docker-compose up -d --force-recreate blackbox-exporter-bms1
'
 
# 3. Verify — should now return 200, not 400.
ssh -i ~/.ssh/vps_root_key root@94.23.26.113 \
  "curl -s -o /dev/null -w '%{http_code}\n' 'http://localhost:9115/probe?module=tcp_connect&target=wkhtml-v42-prod:80'"

docker kill -s HUP blackbox-exporter-bms1 also reloads config without recreate, but --force-recreate is preferred here so the container’s mounted file is guaranteed fresh.

Within ~1–2 scrape intervals (30s) up{job="blackbox_bms1"} returns to 1 and WkhtmlProbeExporterDown clears.

Post-flight

  • Log the operation to infra_operations (op_type=config_change, resource=blackbox-exporter-bms1, server=bms-1). See docs/infra-operations-audit-operations.md.
  • If the drift recurs after a main push that touched bms-1/blackbox-bms1.yml, first check the deploy-bms1-configs workflow run for that commit — it should have deployed automatically. If it failed, fix the workflow (or re-dispatch it manually: gh workflow run deploy-bms1-configs.yml) rather than repeating this manual runbook.

Graph View