Purpose: step-by-step pattern for adding the Nth custom exporter to the p24-infra monitoring
stack. Written while building mailgun-pipeline-exporter (issue #3688) — the first exporter that
needed a host other than vps-i1, which surfaced a network-adjacency decision this playbook now
documents explicitly.
When to use this
Any time a new metric needs to come from polling an external API, a database, or log source that
none of the 11+ existing exporters already cover. Check first whether an existing exporter already
touches the same data source (e.g. mezmo-exporter already derives several pinbox24_* metrics
from Mezmo-shipped logs — adding a metric there may be simpler than a new exporter).
Step 1 — Decide where the exporter runs
This is the step most likely to be skipped, and the one that caused the most design work for
#3688. Prometheus itself always runs on vps-i1, but the exporter container does not have to.
vps-i1 (monitoring/docker-compose.yml) — the default. Use this unless the exporter needs
direct network access to something vps-i1 cannot reach.
bms-4 (bms-4/docker-compose.yml) — use when the exporter needs direct access to MongoDB
rs0 (bms-2/bms-3:27017). bms-4 is a replica-set member (arbiter) and is already allowlisted;
vps-i1 is not, and opening 27017 to a non-replica-set host would violate the ufw allowlist
policy (CLAUDE.md §Do NOT). This is exactly why mongodb-exporter-bms2/-bms3
(percona/mongodb_exporter) run on bms-4, not vps-i1.
bms-1 (bms-1/docker-compose.yml) — use when the exporter needs direct access to a
bms-1-hosted service (v42-prod, v32-prod, mailgun-v42-prod) or needs the same MongoDB
network path those containers already use. mailgun-pipeline-exporter lives here for exactly
this reason (#3688).
Any other host — check whether that host already has network access to the data source
your exporter needs (grep docker-compose.yml/prometheus.yml for existing jobs touching the
same host/service) before assuming vps-i1 will work.
Whichever host you pick, Prometheus scrapes it remotely over its public IP — bind the
exporter’s port to 0.0.0.0, not 127.0.0.1 (vps-i1-local exporters use 127.0.0.1:PORT:PORT
because Prometheus reaches them over the Docker network directly; remote-host exporters need
0.0.0.0:PORT:PORT and a static_configs target of <public-ip>:PORT in prometheus.yml, same
as the mongodb job).
Step 2 — Scaffold the exporter code
Location: monitoring/exporters/<name>-exporter/ regardless of which host it deploys to (source
code lives centrally in this repo; only the deployment docker-compose file differs).
Copy the structure of the most similar existing exporter — mezmo-exporter (simple polling +
gauges/counters) is the best starting template:
Conventions to follow (all established, don’t reinvent):
Sync code, not async — FastAPI is used only for /metrics and /health; the actual
collection runs in a daemon thread started from FastAPI’s lifespan.
POLL_INTERVAL_S env var for collection cadence (default varies by exporter, 60s is typical).
Metrics as module-level Gauge/Counter objects; label series that can “go missing” should be
explicitly initialised to 0 at import time so Grafana/alerts never see a stale-empty series.
Fail-soft, not fail-hard: any external API/DB call failure should log a warning, increment a
dedicated <name>_exporter_api_errors_total{source=...} counter, and return an empty result —
never a dict of zeros. A collection cycle that “succeeds” with real zeros and one that
fails must be distinguishable by the caller, or a broken exporter will falsely trigger whatever
alert reads the primary metric as if it were a real outage (this exact mistake cost ~20h on
issue #2492 — see pinbox24.yml’s Pinbox24NoLogs/MezmoExporterApiErrors comments).
Bounded timeouts on every external call, especially database connections. The #3688
incident this exporter itself was built to detect was caused by an app-side MongoDB call with
no timeout hanging forever — do not repeat that mistake in the exporter that watches for it.
Use serverSelectionTimeoutMS/connectTimeoutMS/socketTimeoutMS for MongoDB,
timeout= for requests.
Add the test directory to pyproject.toml’s testpaths so python -m pytest -v --tb=short
(the root command) picks it up automatically.
Run ruff check monitoring/exporters/<name>-exporter/ before committing (line-length 100,
E/F/I/B/W rules, E501 ignored).
Step 3 — Wire into Prometheus
Add a job_name block to monitoring/prometheus/prometheus.yml, near other jobs targeting the
same host (keeps the file organized by physical topology, not just by service name).
If the exporter is bms-1/bms-4-hosted, target its public IP (see CLAUDE.md §Server IPs).
(requires Docker Desktop running locally; if unavailable, at minimum run
python -c "import yaml; yaml.safe_load(open('monitoring/prometheus/prometheus.yml'))"
as a syntax sanity check before pushing, and let CI’s promtool step catch semantic errors.)
Step 4 — Add alert rules
Add to the most thematically appropriate file in monitoring/prometheus/rules/ (create a new file
only if no existing one fits). Conventions:
severity: critical / severity: warning labels drive Alertmanager routing — no category
label needed unless the alert requires a dedicated n8n branch (see Step 5).
Heavy inline YAML comments explaining why the threshold/window was chosen are the house style
— every alert in this repo has one. Include the issue number that motivated the alert.
Add a companion <Name>ExporterErrors alert (mirrors MezmoExporterApiErrors /
MailgunPipelineExporterErrors) if the exporter’s primary metric could otherwise be
misread as a real outage during an exporter-side failure — see the fail-soft rule in Step 2.
Add a standard <Name>ExporterDown alert using up{job="<job_name>"} == 0 (see
MongoExporterDown, SupabasePgStatsExporterDown) so a fully-dead exporter pages too.
Step 5 — Alertmanager routing
Nothing to do for the common case: severity: critical and severity: warning labels already
route through the existing route.routes matchers in
monitoring/alertmanager/alertmanager.yml.tpl to the critical/warning-digest receivers, which
already fan out to both the email webhook and the n8n Discord-bridge webhook. Do not add a new
receiver or webhook unless the alert needs a distinct downstream action (e.g. auto-creating a
dedicated GitHub issue via an n8n branch, like pm2-restart-loop) — that requires a category
label plus a manual n8n workflow branch (UI-only, not versioned in this repo — see
docs/playbooks/n8n/n8n-alertmanager-incidents.md), which is real extra work and should be a deliberate
choice, not a default.
Step 6 — Deploy (separate from writing the code)
Do not deploy from a coding/PR session. Writing the exporter, compose service, scrape job, and
alert rules is a dev-coder task and belongs in a PR. Actually standing the container up on the
target server (building the image, populating its live .env, opening any firewall exception,
docker-compose up -d) is a sys-admin operation and should happen in its own follow-up session
after the PR merges — call this out explicitly in the PR description so it isn’t silently skipped.
If the exporter needs credentials not already present on its target host’s SOPS-synced .env
(check with the safe key-name-only listing pattern in CLAUDE.md §SOPS), that’s a
secret-manager task — flag the specific missing key + target file, do not add it yourself from a
dev-coder session (see CLAUDE.md §Role Enforcement).
Related
docs/playbooks/mailgun-mongodb-stale-credential-hang.md — the incident this pattern was
extracted from (issue #3688)
docs/playbooks/n8n/n8n-alertmanager-incidents.md — how to add a dedicated n8n receiver branch if
Step 5’s default routing isn’t enough
docs/playbooks/mongodb-exporter-uri-special-chars.md — a real gotcha hit by the existing
MongoDB exporters (password special characters breaking URI parsing) worth knowing before
writing any exporter that builds a MongoDB URI from a raw password