p4-ovh-bms-4-ns3101999 — Operations Workbook

Label: p4-ovh-bms-4-ns3101999 Host: 54.36.123.110 Hostname: ns3101999 Provider: OVH / Kimsufi (ns3101999.ip-54-36-123.eu) Hardware: 8 vCPU · 32 GB RAM · 1.8 TB RAID1 (2× NVMe → /dev/md3) OS: Ubuntu 22.04.5 LTS (kernel 5.15.0-181-generic) Role: MongoDB rs0 arbiter + Docker host (n8n, Traefik, AI-Dev-BMS4-1) Status: Active — fully operational as of 2026-06-17 Inventoried: 2026-06-14

Server Role

Dual-purpose server:

MongoDB arbiter — arbiter-only member of rs0 replica set. Participates in elections for quorum but stores no data. Replaced dead arbiter at 51.83.132.99 (removed 2026-06-10).
Docker host — runs Traefik + n8n (with 3 workers + Redis queue) + monitoring exporters.
AI Dev environment — AI-Dev-BMS4-1 Claude Code agent (max 4 parallel sessions).

RAM: ~31 GB total · 2.5 GB used · 27 GB available. Disk: 1.8 TB total · 21 GB used (2%).

SSH Access

Method	Command
Human (radieu)	`ssh root@54.36.123.110` (uses `~/.ssh/id_ed25519`)
Password fallback	`root` + password in `.env.local` → `bare_metal_server_4_root_password`

Keys installed:

id_ed25519 (radieu) → /root/.ssh/authorized_keys — 2026-06-14

Users:

root — primary shell access (OVH bare metal default)
claude-runner (uid 1001) — Claude Code agent; docker group member
claude-admin (uid 1002) — scoped sudo for docker/systemctl ops

MongoDB

Setting	Value
Version	7.0.37
Replica set	`rs0`
Role	Arbiter only — no data stored, participates in elections
Port	`27017` (bindIp `0.0.0.0`)
Data dir	`/var/lib/mongodb` (minimal — no actual data for arbiter)
Log	`/var/log/mongodb/mongod.log`
Auth	keyFile `/etc/mongodb-keyfile` + `authorization: enabled`
keyFile md5	`34abe562b9b7ecdeaa0e2edb321ec4a4` (matches bms-3 source)
Status	Active since 2026-06-10

Replica Set Members (verified 2026-06-17)

Member	IP	Role
ns3087638 (bms-2)	`145.239.133.104:27017`	PRIMARY
ns3129867 (bms-3)	`51.68.155.224:27017`	SECONDARY
ns3101999 (bms-4)	`54.36.123.110:27017`	ARBITER
~~dead arbiter~~	~~`51.83.132.99:27017`~~	~~Removed 2026-06-10~~

All 3 members healthy (health: 1). Dead arbiter removed successfully.

mongod.conf

storage:
  dbPath: /var/lib/mongodb
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log
net:
  port: 27017
  bindIp: 0.0.0.0
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
replication:
  replSetName: "rs0"
security:
  keyFile: /etc/mongodb-keyfile
  authorization: enabled

Check mongod status

systemctl status mongod
mongosh --quiet --eval 'rs.status().members.map(m => ({name:m.name,stateStr:m.stateStr,health:m.health}))'

Docker

Docker CE 29.5.3 installed via get.docker.com. Enabled via systemd.

systemctl status docker
docker ps

Running Containers (as of 2026-06-17)

Container	Image	Status	Ports
`bms-4-traefik-1`	`traefik:v3.7.5`	Up	0.0.0.0:80, 0.0.0.0:443
`bms-4-n8n-1`	`n8nio/n8n:2.26.3`	Up (healthy)	0.0.0.0:5678
`bms-4-n8n-worker-1-1`	`n8nio/n8n:2.26.3`	Up	0.0.0.0:5679
`bms-4-n8n-worker-2-1`	`n8nio/n8n:2.26.3`	Up	0.0.0.0:5680
`bms-4-n8n-worker-3-1`	`n8nio/n8n:2.26.3`	Up	0.0.0.0:5681
`bms-4-n8n-postgres-1`	`postgres:16.9-alpine`	Up (healthy)	5432 (internal)
`bms-4-redis-1`	`redis:7-alpine`	Up (healthy)	6379 (internal)
`bms-4-redis-exporter-1`	`oliver006/redis_exporter:v1.67.0`	Up	0.0.0.0:9121
`bms-4-cadvisor-1`	`ghcr.io/google/cadvisor:v0.57.0`	Up (healthy)	0.0.0.0:8080

Docker Compose — bms-4/docker-compose.yml

Repo file: bms-4/docker-compose.yml

Deployed to /root/docker-compose.yml on the server.

Architecture: n8n in queue mode with 3 dedicated worker containers + Redis as queue broker. Traefik handles TLS termination via Let’s Encrypt.

Deploy / update

ssh root@54.36.123.110
cd /root
docker compose pull
docker compose up -d
docker compose ps

n8n

Configuration

Setting	Value
Version	`n8nio/n8n:2.26.3`
URL	`https://n8n.bms-4.infra.zintegrowana.online`
Mode	Queue mode (EXECUTIONS_MODE=queue)
Queue	Redis on bms-4
Database	PostgreSQL 16.9 on bms-4
Workers	3 worker containers (ports 5679–5681)
Timezone	`Europe/Warsaw`
Webhook URL	`https://n8n.bms-4.infra.zintegrowana.online/`

Migrated from vps-h1 on 2026-06-15. All workflows from n8n.vps-h1.infra.zintegrowana.online were migrated and activated on bms-4. n8n on vps-h1 was stopped after successful verification.

Health check

curl -s -o /dev/null -w '%{http_code}' https://n8n.bms-4.infra.zintegrowana.online/healthz
# Expected: 200

Known issues / gotchas

specifyBody=keypairs broken in n8n 2.26.x — use specifyBody=json with jsonBody expression
Publishing workflows requires POST /activate with versionId (not just PATCH)
Workers share n8n_data volume with main node (filesystem binary mode)

DNS

Cloudflare DNS record added 2026-06-14:

*.bms-4.infra.zintegrowana.online → 54.36.123.110 (A record, TTL auto, not proxied)

Active URLs:

https://n8n.bms-4.infra.zintegrowana.online — n8n workflow automation

Monitoring

node_exporter (systemd)

prometheus-node-exporter installed via apt-get install prometheus-node-exporter. Active since 2026-06-14.

Service	Method	Status	Port
`prometheus-node-exporter`	systemd	active (running) since 2026-06-14	`:9100`

systemctl status prometheus-node-exporter
curl http://54.36.123.110:9100/metrics | head -5

Prometheus scrape targets (in prometheus.yml)

# In 'node' job:
- targets: ['54.36.123.110:9100']
  labels: { env: production, server_type: baremetal, server: p4-ovh-bms-4-ns3101999, location: ovh-fr }
 
# In 'cadvisor' job:
- targets: ['54.36.123.110:8080']
  labels: { server: p4-ovh-bms-4-ns3101999 }

Both targets active in monitoring/prometheus/prometheus.yml. Prometheus hot-reload run after each config change.

Redis

redis:7-alpine, internal-only (6379, not published). Backs n8n’s Bull queue and the p24auth:* keys written by redis-bridge. Password comes from REDIS_PASSWORD in /opt/p24-infra/bms-4/.env (deployed from secrets/n8n-bms4.env.sops by secrets-sync.yml).

Healthcheck must authenticate via REDISCLI_AUTH, never redis-cli -a (#4595). The service sets REDISCLI_AUTH=${REDIS_PASSWORD} in its environment: block and the probe is a bare ["CMD", "redis-cli", "ping"] — redis-cli picks the credential up from the environment. With the old -a <value> form the resolved credential appeared in the probe’s argv every 10s and was readable by any process with docker access on bms-4 via docker events (exec_start), ps aux, or docker inspect .Config.Healthcheck. Do not reintroduce -a.

Verify after any change that recreates the container:

# Must NOT contain a credential value — only ["CMD","redis-cli","ping"]
docker inspect bms-4-redis-1 --format '{{json .Config.Healthcheck}}'
# Must be: healthy
docker inspect bms-4-redis-1 --format '{{.State.Health.Status}}'

Consumers gate on depends_on: condition: service_healthy (n8n, n8n-worker-1..3, redis-exporter, redis-bridge), so a broken healthcheck stops the whole n8n stack from starting.

Known residual exposure: command: still passes --requirepass ${REDIS_PASSWORD}, so the value is present in redis-server’s own argv (docker inspect .Config.Cmd). Removing that needs a mounted redis.conf or an entrypoint wrapper — not done as part of #4595, tracked in #4597.

Applying a compose change to redis (the container must be recreated):

cd /opt/p24-infra && git pull origin main
cd /opt/p24-infra/bms-4 && set -a && . ./.env && set +a
docker compose up -d --no-deps redis

secrets-sync.yml’s sync-bms-4 job does exactly this (gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=bms-4), including a 60s wait for healthy. Note its push trigger is path-filtered to secrets/*.env.sops, so a compose-only change is not deployed by merging to main — dispatch the workflow or apply manually.

Redis exporter

oliver006/redis_exporter:v1.67.0 running as Docker container on :9121. Reports Redis queue metrics for n8n queue monitoring.

Prometheus Pushgateway

prom/pushgateway running as a Docker container, bound 0.0.0.0:9091. Acts as the push target for short-lived / batch jobs that cannot be scraped directly. vps-i1 Prometheus scrapes it remotely at 54.36.123.110:9091 (job pushgateway, honor_labels: true).

Service	Method	Bind	Scraped by
`pushgateway`	Docker container	`0.0.0.0:9091`	vps-i1 Prometheus (`54.36.123.110:9091`, job `pushgateway`)

Producers:

n8n brand-credential-health-check workflow pushes brand_credential_status gauge metrics here on each run.

# Verify the pushgateway is up and listing pushed metrics
curl -s http://54.36.123.110:9091/metrics | grep -E '^brand_credential_status' | head

Added in PR #1116; bind port corrected to 9091 in PR #1119. honor_labels: true on the Prometheus side preserves the instance/job labels set by the pushing workflow rather than overwriting them with the pushgateway’s own.

Promtail (syslog → Loki) — #1336

Promtail runs as a systemd service (no Docker), installed by the Ansible promtail-syslog role via ansible/playbooks/bms-4.yml. It scrapes /var/log/auth.log + /var/log/syslog (which carry the auditd admin-action trail) and ships them to Loki on vps-i1, so bms-4 logs are centrally searchable alongside the rest of the fleet (#1316).

Service	Method	Port	Ships to
`promtail`	systemd (`promtail-syslog` role)	`:9080` (local)	`loki.vps-i1.infra.zintegrowana.online`

systemctl is-active promtail
journalctl -u promtail -n 50 --no-pager

Host label: promtail_host_label: bms-4 (set in host_vars/bms-4.yml) → logs queryable in Grafana Explore (Loki) as {host="bms-4", job="syslog"}.

Secret: LOKI_BASIC_AUTH_PASSWORD is written to /etc/promtail/promtail.env (mode 0600) at apply time, sourced from secrets/monitoring.env.sops — never hardcoded. Deploy:

export SOPS_AGE_KEY_FILE="$HOME/.age/p24-infra-keys.txt"
LOKI_PW=$(sops --decrypt --input-type dotenv --output-type dotenv \
  secrets/monitoring.env.sops | grep '^LOKI_BASIC_AUTH_PASSWORD=' | cut -d= -f2-)
cd ansible && ansible-playbook playbooks/bms-4.yml --check --diff \
  -e "loki_basic_auth_password=$LOKI_PW"   # drop --check to apply

Verify after deploy: run sudo -v on bms-4, then in Grafana Explore: {job="syslog", host="bms-4"} |= "sudo" — the entry should appear within ~60s.

Disk Layout

/dev/nvme0n1  ~1.9 TB  (NVMe disk 1)
/dev/nvme1n1  ~1.9 TB  (NVMe disk 2)
/dev/md3       1.8 TB   mounted at /  — 21G used (2%)

Disk in excellent state — 1.7 TB free. No disk pressure concerns.

AI Dev Agent — AI-Dev-BMS4-1

Setting	Value
GitHub user	`AI-Dev-BMS4-1`
Max parallel agents	4
Linux user	`claude-runner` (uid 1001, docker group)
SSH user	`claude-admin` (uid 1002, scoped sudo)
Claude Code path	`/usr/bin/claude`
Auth	OAuth subscription (Claude Max)
Credentials	`/home/claude-runner/.claude/`
Re-auth script	`python d:\tmp\reauth-bms4.py` (run locally when OAuth expires)

The agent is registered in dev_r_services as AI-Dev-BMS4-1 with compliance_workbook='yes'.

Firewall (UFW)

UFW active. Allowed inbound rules:

22/tcp — SSH
80/tcp — Traefik HTTP (redirect to HTTPS)
443/tcp — Traefik HTTPS
27017/tcp — MongoDB (from bms-2: 145.239.133.104, bms-3: 51.68.155.224)
9100/tcp — node_exporter (from vps-i1: 217.154.82.162)
8080/tcp — cAdvisor (from vps-i1)
Docker bridge → host access allowed for n8n workers
9998/tcp — gmail-session-manager, restricted to 172.18.0.0/16 (n8n’s Docker bridge) via iptables -A INPUT -s 172.18.0.0/16 -p tcp -m tcp --dport 9998 -j ACCEPT, mirroring the existing 9997 rule (#4813)

Provisioning Log

Date	Action	By
2026-06-14	Server inventoried (Ubuntu 22.04.5, 0% disk)	Claude Code
2026-06-14	apt upgrade completed	Claude Code
2026-06-14	MongoDB 7.0.37 installed, keyFile from bms-3, mongod.conf written, mongod started	Claude Code
2026-06-14	Docker CE 29.5.3 installed	Claude Code
2026-06-14	Cloudflare DNS `*.bms-4.infra.zintegrowana.online` → `54.36.123.110` created	Claude Code
2026-06-14	`prometheus-node-exporter` installed and started (systemd)	Claude Code
2026-06-14	Added to `monitoring/prometheus/prometheus.yml` (node + cadvisor jobs)	Claude Code
2026-06-14	`bms-4/docker-compose.yml` created in repo	Claude Code
2026-06-10	`rs.addArb("54.36.123.110:27017")` — bms-4 joined rs0 as arbiter	radieu
2026-06-10	`rs.remove("51.83.132.99:27017")` — dead arbiter removed from rs0	radieu
2026-06-15	n8n migrated from vps-h1 — all workflows activated on bms-4	Claude Code
2026-06-15	n8n stopped on vps-h1 after successful bms-4 verification	Claude Code
2026-06-15	claude-runner + claude-admin users provisioned, AI-Dev-BMS4-1 registered	Claude Code
2026-06-17	Ops doc updated to reflect complete provisioning state	Claude Code

Acceptance Criteria — All Met

Criterion	Status
rs0 has 3 healthy members (bms-2 PRIMARY + bms-3 SECONDARY + bms-4 ARBITER)	Done
Dead arbiter `51.83.132.99` removed from rs0	Done
n8n accessible at `https://n8n.bms-4.infra.zintegrowana.online` (HTTP 200)	Done
Prometheus scrapes node_exporter at `54.36.123.110:9100`	Done
All elements registered in `dev_r_services` with `compliance_workbook='yes'`	Done
Ops doc exists at `docs/servers/p4-ovh-bms-4-ns3101999-operations.md`	Done

Claude Session Manager

Python HTTP service for Telegram bot control:

Port: :9997
Auth: X-Session-Key header (CLAUDE_SESSION_MANAGER_KEY from SOPS secrets/n8n-bms4.env.sops)
Systemd: claude-session-manager.service (runs as claude-runner)
iptables: 172.18.0.0/16 -> port 9997 ACCEPT (allows N8N Docker container to reach host service)

Gmail Session Manager

Python HTTP service backing the Telegram “Gmail assistant” bot (#4813) — a fully isolated sibling of Claude Session Manager, not an extension of it:

Port: :9998, bound to the Docker bridge address 172.18.0.1 only (not 0.0.0.0)
Auth: X-Session-Key header (GMAIL_SESSION_MANAGER_KEY from SOPS secrets/n8n-bms4.env.sops, compared with hmac.compare_digest)
OS user: gmail-runner (uid 1004) — unprivileged, no sudoers entry, no SOPS age key, no SSH keys, no GitHub token, no access granted to /opt/p24-infra or /home/claude-runner
Code: /opt/gmail-session-manager/ (server.py, mcp.json, PROMPT.md), source in infra-src/gmail-session-manager/
State: /var/lib/gmail-telegram/{sessions,pending,creds} (mode 0700, owner gmail-runner)
Clone: /home/gmail-runner/telegram-workdir/gmail-tools (radieu/gmail-tools, npm ci --omit=dev)
Systemd: gmail-session-manager.service (User=gmail-runner, NoNewPrivileges=yes, ProtectSystem=strict, ProtectHome=yes, PrivateTmp=yes, ReadWritePaths=/var/lib/gmail-telegram) — installed but not enabled/started until wave 4 (Google OAuth for gmail-runner) completes
iptables: 172.18.0.0/16 -> port 9998 ACCEPT (mirrors the 9997 rule)
n8n workflow telegram-gmail-bot (id YjtA6n33pWvpReIo, 21 nodes) — built 2026-08-04, inactive (service has no valid Google credentials yet). Credential Telegram Bot - Gmail Assistant (id C9Y4XJhOzmEvCEDU) bound to ={{ $env.TELEGRAM_GMAIL_BOT_API_KEY }}.
Wave 4 verified still open 2026-08-04: no token.json in /var/lib/gmail-telegram/creds/. Shared OAuth client gcp-oauth.keys.json (id/secret only) staged there from /opt/gmail-tools/credentials/, owner gmail-runner:gmail-runner, mode 0600.
Full ops doc: docs/telegram-gmail-bot-operations.md
Do not confuse with google-worker-bot / gmail-worker-bot (an earlier, separate /opt/gmail-tools + claude-runner based prototype, see docs/gmail-tools-telegram-bot-operations.md) — the two are intentionally unrelated services with different OS users, ports, and security models.

p24-infra Docs

Explorer

p4-ovh-bms-4-ns3101999-operations