bms-4 Automation Platform — n8n Operations

n8n workflow automation engine running on OVH bms-4 (n8n.bms-4.infra.zintegrowana.online). Handles fleet GPS sync, WhatsApp routing, brand video publishing, and daily reporting. All workflows migrated from vps-h1 on 2026-06-15; vps-h1 n8n decommissioned same day.

Migration (2026-06-15): All workflows moved from vps-h1 → bms-4 (PostgreSQL backend, Traefik TLS). vps-h1 n8n containers stopped and removed — vps-h1 now runs only WAHA + Traefik + monitoring exporters.

Architecture

OVH bms-4 (54.36.123.110)
├── Container: n8n                    image: docker.n8n.io/n8nio/n8n:2.26.3
│   ├── port 5678                    web UI + webhook receiver (internal only)
│   ├── EXECUTIONS_MODE=queue        enqueues jobs to Redis; does NOT execute them
│   ├── DB: n8n-postgres container
│   └── Redis: redis container (job broker)
│
├── Container: n8n-worker-1           image: docker.n8n.io/n8nio/n8n:2.26.3
│   ├── port 5679                    Prometheus scrape endpoint
│   ├── cpus: 1.5 · N8N_RUNNERS_MAX_CONCURRENCY=3
│   └── dequeues jobs from Redis, executes, writes results to PostgreSQL
│
├── Container: n8n-worker-2           image: docker.n8n.io/n8nio/n8n:2.26.3
│   ├── port 5680                    Prometheus scrape endpoint
│   ├── cpus: 1.5 · N8N_RUNNERS_MAX_CONCURRENCY=3
│   └── same as n8n-worker-1
│
├── Container: n8n-worker-3           image: docker.n8n.io/n8nio/n8n:2.26.3
│   ├── port 5681                    Prometheus scrape endpoint
│   ├── cpus: 1.5 · N8N_RUNNERS_MAX_CONCURRENCY=3
│   └── same as n8n-worker-1
│
├── Container: redis                  image: redis:7-alpine
│   ├── AOF persistence (appendonly yes)
│   ├── maxmemory 256mb (noeviction policy)
│   └── password-protected (REDIS_PASSWORD)
│
├── Container: redis-exporter         image: oliver006/redis_exporter:v1.67.0
│   └── port 9121                    Prometheus scrape (vps-i1 → 54.36.123.110:9121)
│
└── Container: traefik (bms-4)        TLS termination → n8n:5678

Public URL: https://n8n.bms-4.infra.zintegrowana.online
Compose file on server: /opt/p24-infra/bms-4/docker-compose.yml
Compose file in repo: bms-4/docker-compose.yml (canonical — keep in sync)

Active Workflows

Workflow	ID	Trigger	Purpose
ATRAX GPS Sync	`CCx9UMdphmGficDX`	Cron every 3min	Fetches vehicle positions from ATRAX API → Supabase `fleet_positions`
wa-router	—	Webhook (WAHA)	Routes WhatsApp messages to AI / Supabase
wa-ai-to-inbox	—	Webhook	WhatsApp AI reply generation
wa-processing-watchdog	—	Cron every 5min	Alerts if WhatsApp processing stalls
GPS Sync Watchdog	—	Cron every 30min	Alerts if last GPS sync >30min ago
Daily Fleet Report	—	Cron 07:00 UTC	Generates and emails daily fleet status report
brand-heygen-video	`GEfh5Ykv9A7XpHYy`	Webhook (radekkonarski-personal-brand)	Receives HeyGen video completion → uploads to Wasabi → posts to LinkedIn. Credentials: `WASABI_ACCESS_KEY`, `WASABI_SECRET_KEY`, `LINKEDIN_ACCESS_TOKEN`, `LINKEDIN_PERSON_URN` from SOPS `secrets/n8n-bms4.env.sops` (deployed via `secrets-sync.yml`).
mezmo-alert-router	`Sc2AKCdslinnSvWs`	Webhook (Mezmo)	Receives Mezmo alert webhooks → routes to Discord embed + optional GH issue by alert type

Migrated business-automation workflows (ex-Hostinger)

These workflows carry the legacy n8n-hstgr/ registry prefix (dev_r_services) from when they ran on the Hostinger n8n instance. They were migrated to this bms-4 n8n instance on 2026-06-15 and host = 'bms-4' in dev_r_services. This section is their ops workbook (workbook_url).

Common operational properties (all rows below):

Backup / restore: gitbackup workflow commits workflow JSON to git nightly (first auto-run 23:15; cannot be triggered manually in queue mode).
Monitoring: error-flow (centralized error handler, activated 2026-06-15) routes any failure to Discord.
Secrets: injected at runtime via start.sh + SOPS+age (see Config Management below).

Workflow (`n8n-hstgr/…`)	Trigger / schedule	Purpose	Status
`wa-router`	Webhook (WAHA)	Main WhatsApp message router	Active
`wa-group-sync`	Webhook / cron	WhatsApp group membership sync	Active
`wa-processing-watchdog`	Cron every 5min	Restarts stalled WhatsApp pipelines	Active
`webhooks-for-fleet-register`	Webhook	Fleet registration event handlers	Active
`fleet-update-v2-batch`	Cron every 10min	Batch fleet GPS upsert (RPC `update_fleet_gps_batch`, migrations 014/015)	Active
`update-fleet-from-driver-reports`	Webhook	Updates fleet from driver reports (Google Sheets cred)	Active
`atrax-daily-stats-collector`	Daily	Atrax stats collection	Active
`atrax-drivers-daily-snapshot`	Daily	Atrax driver snapshot	Active
`atrax-ecodriving-daily`	Daily	Atrax ecodriving data	Active
`get-intercars-token`	Scheduled	Fetch/refresh Intercars API token	Active
`interrcars-payments-3h-update`	Cron every 3h	Intercars payments sync	Active
`invoices-registration-2-pinbox`	Webhook	Invoice registration → Pinbox / faktury@eco-trans.eu	Active
`vin-analyse-car-spec`	Webhook	VIN → car spec lookup	Active
`et-furpak-car-webapp`	Webhook	Furpak car webapp integration	Active
`et-furpak-driver-litle-checkreport`	Webhook	Furpak driver check report	Active
`hu-sp-report-email`	Scheduled	HU/SP inspection report email sender	Active
`mobile-weekly-report-generator`	Cron weekly	Mobile weekly report (events)	Active
`p24-weekly-mobile-report-generator`	Cron weekly	P24 weekly mobile report	Active
`compliance-gaps-to-incidents`	Scheduled	Creates Supabase incidents from compliance gaps	Active
`imap-inbox-sync-ai-classifier`	IMAP poll	IMAP inbox → AI classifier (Gmail `ecotrans.automation`)	Active
`tronik-gps-kierowcy`	Gmail trigger	Tronik GPS driver report (AI via SSH `claude` worker)	Active
`tronik-gps-serwisant-mobilny`	Gmail trigger	Tronik GPS mobile-technician report	Active
`github-auto-trigger`	GitHub webhook	Auto-trigger handler — `auto` branch runs `claude` CLI via SSH (`vps-i1-root-ssh`), see `n8n-github-automation.md`	Active
`et-chatbot-produkcja`	Webhook	Telegram chatbot, RAG + pgvector	In development — not yet activated on bms-4
`alertmanager-to-incidents`	Webhook (Alertmanager)	Alertmanager → Supabase incidents	Superseded by `n8n-bms4/alertmanager-to-incidents` (2026-06-15)

The infra-only workflows error-flow and gitbackup are the monitoring/backup backbone for every row above; both are documented in the bms-4 server workbook.

Config Management

File	Location	In repo?	Contains secrets?
`docker-compose.yml`	`/opt/p24-infra/bms-4/docker-compose.yml`	✅ `bms-4/docker-compose.yml`	No
PostgreSQL data	Docker volume `n8n_postgres_data` on bms-4	❌	Yes (credential vault)
n8n workflow exports	`infra-src/n8n-workflows/*.json`	✅	No (credentials stripped)

Secret Injection — start.sh + SOPS+age

⚠️ ALWAYS start/restart n8n via start.sh — never run docker compose up -d directly. Raw docker compose bypasses secrets sourcing and the containers will start with empty secrets, causing all workflows that depend on env vars to fail silently.

Startup flow:

/opt/p24-infra/bms-4/start.sh
  └── sources /opt/p24-infra/bms-4/.env (SOPS-deployed by secrets-sync.yml)


  └── docker compose -f /opt/p24-infra/bms-4/docker-compose.yml up -d
        └── ${VAR} substitution uses shell env

Note: secrets are deployed by secrets-sync.yml to /opt/p24-infra/bms-4/.env. Keep .env.bak up-to-date whenever secrets rotate (see regen-n8n-envbak.sh).

To add a new secret to n8n containers:

Add the secret to secrets/n8n-bms4.env.sops (see docs/secrets-management.md)
Add - NEW_VAR=${NEW_VAR} to the environment: block in bms-4/docker-compose.yml (for n8n, n8n-worker-1, n8n-worker-2, n8n-worker-3)
Commit and push to main
On bms-4: bash /opt/p24-infra/bms-4/start.sh
Verify: docker exec bms-4-n8n-1 printenv | grep NEW_VAR

Secrets source: secrets/n8n-bms4.env.sops — see docs/secrets-management.md for rotation procedures.

Secrets injected at runtime

n8n credentials are stored encrypted in PostgreSQL (AES-256 via N8N_ENCRYPTION_KEY).
Key env vars (all sourced from SOPS secrets/n8n-bms4.env.sops via start.sh):

Variable	Purpose
`N8N_ENCRYPTION_KEY`	Master encryption key for credential vault — must be set in shell env on bms-4, not just in docker-compose.yml, so workers can decrypt credentials at startup
`BMS4_N8N_API_KEY`	n8n REST API key (external access)
`N8N_DB_PASSWORD`	PostgreSQL password for n8n database
`REDIS_PASSWORD`	Redis auth password (shared by main + workers + redis-exporter)
`QUEUE_BULL_REDIS_HOST`	Redis hostname — `redis` (Docker network)
`QUEUE_BULL_REDIS_PORT`	Redis port — `6379`
`QUEUE_BULL_REDIS_PASSWORD`	Same value as `REDIS_PASSWORD`
`QUEUE_BULL_REDIS_DB`	Redis DB index — `0`

N8N_ENCRYPTION_KEY advisory: Workers read this key directly from the shell environment when they start. If it is only declared inside docker-compose.yml environment: and not exported in the host shell, workers may fail to decrypt credentials and executions will error. Always ensure it is present in /opt/bms4-services/.env and exported in the session before running docker compose up.

Queue Mode

Added 2026-06-15. n8n runs in queue mode (EXECUTIONS_MODE=queue) with Redis as the job broker.

Why queue mode

Single-container n8n was blocking on long-running workflows (Atrax GPS sync, WhatsApp AI, daily report). Queue mode offloads execution to dedicated workers, keeping the main container responsive to webhooks.

Capacity

Component	Count	CPU limit	Max concurrent jobs
n8n main	1	0.5	0 (webhook + scheduling only)
n8n-worker-1	1	1.5	3 (`N8N_RUNNERS_MAX_CONCURRENCY=3`)
n8n-worker-2	1	1.5	3
n8n-worker-3	1	1.5	3
Total	—	—	9 concurrent executions

Redis configuration

Setting	Value	Reason
Image	`redis:7-alpine`	Stable LTS
Persistence	AOF (`appendonly yes`)	Job queue survives container restart
Max memory	`256mb` (noeviction)	Prevents OOM; noeviction keeps queue intact
Auth	`requirepass ${REDIS_PASSWORD}`	Required — all clients pass password

Scaling workers

To add a third worker, clone n8n-worker-2 block in bms-4/docker-compose.yml, assign a unique port (e.g. 5681:5678), and run docker compose up -d n8n-worker-3.

Checking worker health

ssh root@54.36.123.110
cd /opt/p24-infra/bms-4
 
# All containers running?
docker compose ps
 
# Worker logs
docker compose logs --tail=50 n8n-worker-1
docker compose logs --tail=50 n8n-worker-2
 
# Redis queue depth
docker compose exec redis redis-cli -a "${REDIS_PASSWORD}" llen bull:jobs:wait

Troubleshooting queue mode

Symptom	Cause	Fix
Workers exit immediately	`N8N_ENCRYPTION_KEY` not in shell env	Export key in host shell; `docker compose up -d` again
Executions stuck in “running”	Worker crashed mid-job	Restart workers; Redis retains job state
Redis OOM	Queue flooded	Check for runaway workflow; `docker compose restart redis` (jobs lost)
Worker can’t connect to Redis	`REDIS_PASSWORD` mismatch	Verify `.env` has matching `REDIS_PASSWORD`; restart all

Deployment

Starting / restarting n8n

Always use start.sh — it sources SOPS-deployed secrets before starting containers.

ssh root@54.36.123.110
bash /opt/p24-infra/bms-4/start.sh
# Output: "[start.sh] Sourced secrets from .env"
# Then docker compose recreates only changed containers

Fresh install

# On bms-4 — assumes docker-compose.yml is deployed and secrets-sync.yml has run
ssh root@54.36.123.110
bash /opt/p24-infra/bms-4/start.sh

Update n8n version

Update image tag in bms-4/docker-compose.yml
Commit and push

On bms-4:

ssh root@54.36.123.110
cd /opt/p24-infra/bms-4
docker compose pull n8n
docker compose up -d n8n

Verify: curl -s https://n8n.bms-4.infra.zintegrowana.online/healthz

Import a workflow

# Via REST API (no UI needed)
curl -X POST https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows \
  -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @infra-src/n8n-workflows/wa-router.json

Backup

What needs backing up

Data	Method	Schedule	Destination
Workflow definitions	`n8n-backup.yml` GH Action → Wasabi	Nightly 03:00 UTC	`s3://p24-infra/n8n/workflows-YYYY-MM-DD.json`
n8n credentials (encrypted)	`n8n-backup.yml` → Wasabi	Nightly	`s3://p24-infra/n8n/credentials-YYYY-MM-DD.json`
PostgreSQL DB (`n8n_postgres_data`)	Not directly backed up	—	Gap — execution history only; workflows/creds covered by API export

Note: Workflows and credentials are exported via REST API nightly. If the PostgreSQL volume is lost, workflows can be re-imported but execution history will be gone. Credentials must be re-entered manually (values are never exported).

Manual backup

# Export workflows via API
curl -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows > /tmp/n8n-workflows.json
 
# Export credentials (encrypted, no values)
curl -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/credentials > /tmp/n8n-credentials.json

Restore

Target RTO: 45 minutes (includes workflow re-activation)

Scenario 1: Container crash

ssh root@54.36.123.110
docker compose -f /opt/p24-infra/bms-4/docker-compose.yml up -d n8n
# n8n_postgres_data volume preserved — no data loss

Scenario 2: Fresh install (n8n_postgres_data volume lost)

# 1. Download latest workflow backup from Wasabi
aws s3 cp s3://p24-infra/n8n/workflows-YYYY-MM-DD.json /tmp/ \
  --endpoint-url https://s3.eu-central-2.wasabisys.com
 
# 2. Start fresh n8n (PostgreSQL will be initialized on first start)
ssh root@54.36.123.110
docker compose -f /opt/p24-infra/bms-4/docker-compose.yml up -d n8n-postgres n8n
 
# 3. Wait for n8n to be ready
curl -s https://n8n.bms-4.infra.zintegrowana.online/healthz
 
# 4. Import workflows via API
curl -X POST https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows/import \
  -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d @/tmp/n8n-workflows.json
 
# 5. Re-enter credentials manually in the n8n UI (cannot be restored from export)
 
# 6. Activate workflows: set active=true for each
 
# 7. Verify ATRAX sync: check Supabase fleet_positions for new entries

Note: Credentials (API keys, passwords stored in n8n) must be re-entered manually after a fresh restore. Keep the credential list in docs/elements.md § Credentials Index.

Healthcheck

Docker healthcheck: wget http://localhost:5678/healthz — defined in bms-4/docker-compose.yml

External probe: blackbox-exporter via https://n8n.bms-4.infra.zintegrowana.online/healthz — EndpointDown alert.

Manual check:

curl -s https://n8n.bms-4.infra.zintegrowana.online/healthz
# Expected: {"status":"ok"}

Password Rotation

BMS4_N8N_API_KEY

# 1. Generate new key in n8n UI: Settings → API → Create API Key
 
# 2. Update /opt/bms4-services/.env on bms-4:
ssh root@54.36.123.110 "sed -i 's/BMS4_N8N_API_KEY=.*/BMS4_N8N_API_KEY=<new-key>/' /opt/bms4-services/.env"
 
# 3. Restart n8n to pick up new key:
ssh root@54.36.123.110 "docker compose -f /opt/p24-infra/bms-4/docker-compose.yml restart n8n"
 
# 4. Update GH Secret (BMS4_N8N_API_KEY) and .env.local
 
# 5. Log rotation in docs/secrets-rotation-log.md

GPS Sync Workflow — atrax, kravag-scheduled-fleet-updates (CCx9UMdphmGficDX)

Production host: bms-4 (n8n.bms-4.infra.zintegrowana.online) — migrated from vps-h1 on 2026-06-15.
DB: PostgreSQL on bms-4 (not SQLite — see docs/n8n-postgresql-operations.md).
Status: Active, production, criticality = high.

Triggers

Name	Type	Schedule / Path	Notes
`3 minutes1`	Schedule	Every 3 min	Primary trigger — main fleet sync
`Webhook atrax sync`	Webhook	`/webhook/atrax-sync` (prod) / `/webhook-test/atrax-sync` (test)	On-demand sync
`5 morning`	Schedule	05:00 UTC daily	Disabled 2026-06-15 — was causing double executions
`When clicking 'Execute workflow'`	Manual	—	Dev/test only

What it writes

p24_gps_current_state — vehicle GPS positions (via upsert_gps_current_state node)
p24_l_cars_atrax — vehicle metadata (via update-atrax-pojazdy-w car-atrax node)
p24_l_cars — mileage via rpc/update_atrax_rt_batch; driver info via rpc/update_atrax_drivers_batch (both single batch calls per cycle — see issue #629)

Freshness is monitored by .github/workflows/atrax-data-freshness.yml (runs every 10 min).

Batch update pattern (IMPORTANT — do not revert)

All writes to p24_l_cars MUST use batch RPCs, never individual row upserts via REST API. Individual POST /rest/v1/p24_l_cars?on_conflict=... calls (one per vehicle) caused AccessExclusiveLock contention blocking all reads (issue #629, 2026-06-17).

Node	RPC called	Fields written
`update atraxId na rej floty by plate1`	`rpc/update_atrax_rt_batch`	`current_mileage`, `km_upd_date`
`batch_drivers_rpc`	`rpc/update_atrax_drivers_batch`	`atrax_driver1`, `atrax_driver1_name`, `atrax_driver2`, `atrax_driver2_name`

The aggregate_driver_updates node (Aggregate type) collapses all per-vehicle items into a single item before batch_drivers_rpc runs, ensuring exactly one HTTP call per sync cycle regardless of fleet size.

Known failure modes (discovered 2026-05-19, issue #201)

Failure	Symptom	Fix
Workflow deactivated in n8n	`active=0` in SQLite, no executions	See “Re-activating a workflow” below
`activeVersionId` NULL in SQLite	Workflow shows `active=1` but never appears in startup activation log	See “Re-activating a workflow” below
Missing `gps_atrax_installed` column on `p24_l_cars`	`supabase upsert query` node fails, GPS branch never runs	`ALTER TABLE p24_l_cars ADD COLUMN IF NOT EXISTS gps_atrax_installed boolean DEFAULT false`
Ambiguous `update_atrax_rt_batch` function overload	PGRST203 error on `update atraxId na rej floty by plate1` node	Drop old single-arg overload: `DROP FUNCTION public.update_atrax_rt_batch(vehicles jsonb)`
Missing `status_atrax` column on `p24_l_cars`	`upsert_gps_current_state` fails — trigger `sync_gps_state_to_cars` references it	`ALTER TABLE p24_l_cars ADD COLUMN IF NOT EXISTS status_atrax text`

Re-activating a workflow after deactivation

If n8n was restarted and the startup log does NOT show Activated workflow "atrax, kravag-scheduled-fleet-updates":

# SSH to bms-4 as ubuntu (n8n now on bms-4, using PostgreSQL — not SQLite)
ssh root@54.36.123.110
 
# Check active state via n8n API
curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows/CCx9UMdphmGficDX \
  | jq '{id,name,active}'
 
# Re-activate via API
curl -X PATCH \
  -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"active": true}' \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows/CCx9UMdphmGficDX
 
# Or restart the n8n container on bms-4
docker compose -f /opt/p24-infra/bms-4/docker-compose.yml restart n8n
 
# Verify: check executions appear
curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  "https://n8n.bms-4.infra.zintegrowana.online/api/v1/executions?workflowId=CCx9UMdphmGficDX&limit=3" \
  | jq '[.data[] | {id,status,startedAt}]'

n8n 2.x Upgrade (2026-05-24, issue #220)

What changed

n8n was upgraded from 1.123.44 to 2.22.0 to fix two critical CVEs:

CVE	CVSS	Description	Fixed in
CVE-2025-68668	9.9	Python Code Node RCE via Pyodide	n8n 2.0.0
CVE-2026-25115	10.0	Python sandbox escape	n8n 2.4.8

Both CVEs required the Python Code Node (Pyodide-based). None of our workflows use Python — all Code nodes use JavaScript — so the real-world exposure was zero, but the CVSS scores demanded the upgrade.

Workflow compatibility audit

All workflows were audited before upgrade. All Code nodes use JavaScript (not Python):

Workflow	Code nodes	Language	2.x compatible
wa-router	2	JavaScript	Yes
wa-ai-to-inbox	3	JavaScript	Yes
wa-group-sync	1	JavaScript	Yes
wa-processing-watchdog	1	JavaScript	Yes
fleet-update-v2-batch	4	JavaScript	Yes
alertmanager-to-incidents	0	—	Yes
sentry-to-github	2	JavaScript	Yes
github-auto-trigger	1	JavaScript	Yes
hu-sp-report-email	1	JavaScript	Yes

n8n 2.x breaking changes (1.x → 2.x)

Key breaking changes in n8n 2.0.0 relevant to our stack:

Python Code Node removed — Pyodide sandbox dropped entirely. Not used in our workflows.
N8N_BASIC_AUTH_ACTIVE deprecated — use N8N_COMMUNITY_PACKAGES_ALLOW_TOOL_USAGE for community nodes. Basic auth still works for API key auth (our setup uses X-N8N-API-KEY).
SQLite schema migration — n8n 2.x auto-migrates the SQLite DB on first start. Expect ~10–30s extra startup time. Migration is non-destructive.
Execution data schema — executionData column type changed. Old executions remain readable but storage format differs.
Webhook node v2 — our workflows already use typeVersion: 2 (confirmed in audit above).
httpRequest node v4.2 — our workflows use this version (confirmed in audit above). No changes needed.
Code node v2 — our Code nodes use typeVersion: 2. Compatible with 2.x.

Deployment procedure

n8n 2.x upgrade is performed via a standard image tag bump + rolling restart:

# On bms-4 (PostgreSQL backend — no SQLite migration needed)
ssh root@54.36.123.110
cd /opt/p24-infra/bms-4
docker compose pull n8n
docker compose up -d n8n
 
# Monitor startup logs for workflow activation
docker compose logs -f n8n | grep -E "Activated|error|ERROR"
 
# Verify healthz
curl -s http://localhost:5678/healthz
# Expected: {"status":"ok"}
 
# Verify all workflows are active
curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  http://localhost:5678/api/v1/workflows | jq '[.data[] | {name,active}]'

Post-upgrade checklist

n8n container starts and /healthz returns {"status":"ok"}
All workflows show active: true in API response
wa-router webhook receives test event from WAHA
fleet-update-v2-batch (ATRAX sync) runs on schedule — check fleet_positions freshness
Trivy scan passes clean (CVE-2025-68668 and CVE-2026-25115 no longer reported)

Rollback procedure

If issues arise, roll back to previous version:

# On bms-4
ssh root@54.36.123.110
# Edit bms-4/docker-compose.yml image tag back to previous version
cd /opt/p24-infra/bms-4
docker compose up -d n8n
 
# If rollback causes DB errors, restore from last Wasabi backup:
# aws s3 cp s3://p24-infra/n8n/workflows-YYYY-MM-DD.json /tmp/ \
#   --endpoint-url https://s3.eu-central-2.wasabisys.com

Troubleshooting

Symptom	Cause	Fix
GPS positions stale (>10min)	ATRAX workflow deactivated or DB schema mismatch	See “GPS Sync Workflow — Known failure modes” above; check n8n startup log
GPS positions stale (>10min)	ATRAX API unreachable	Check `get_token` node output in latest execution; test Atrax API from bms-4
WhatsApp messages not routing	wa-router webhook timeout	Check WAHA logs on vps-h1; verify webhook URL points to `n8n.bms-4`
n8n UI unreachable	Traefik or container down	`ssh root@54.36.123.110 "docker compose -f /opt/p24-infra/bms-4/docker-compose.yml ps"`
`N8N_ENCRYPTION_KEY` changed	Credential vault corrupted	Must re-enter all credentials manually
n8n crashes on startup	`N8N_DB_PASSWORD` not set	Verify `N8N_DB_PASSWORD` is in `secrets/n8n-bms4.env.sops`; trigger secrets-sync.yml then run `bash /opt/p24-infra/bms-4/start.sh`
brand-heygen-video fails on Wasabi node	Missing env vars in container	Verify `WASABI_ACCESS_KEY` / `WASABI_SECRET_KEY` in `secrets/n8n-bms4.env.sops`; trigger secrets-sync.yml then restart via `start.sh`
brand-heygen-video fails on LinkedIn node	Token expired or missing	Rotate `LINKEDIN_ACCESS_TOKEN` in `secrets/n8n-bms4.env.sops`; trigger secrets-sync.yml then restart via `start.sh`
mezmo-alert-router webhook not firing	Workflow deactivated or Mezmo view channel URL wrong	Check workflow active status via API; verify view channel URL in Mezmo dashboard matches `https://n8n.bms-4.infra.zintegrowana.online/webhook/mezmo-alert`
Discord 400 from Send Discord node	n8n specifyBody bug — body sent as form key	Verify Send Discord node uses specifyBody=“json” + jsonBody param; see n8n quirks §above

brand-heygen-video — Brand Video Post-Production Pipeline (GEfh5Ykv9A7XpHYy)

Production host: bms-4 (n8n.bms-4.infra.zintegrowana.online)
Status: Active, production, criticality = medium.
Caller: scripts/heygen/produce.js (runs locally on developer machine — fires webhook after HeyGen renders the video).

What it does

produce.js handles the HeyGen web UI session and waits for the render. Once done, it POSTs the video URL to this n8n workflow, which takes over and runs the three-step post-production pipeline:

produce.js  ──webhook──▶  n8n: brand-heygen-video
                              │
                              ├─ 1. Download MP4 from HeyGen CDN URL
                              ├─ 2. Upload to Wasabi S3 (public-read)
                              └─ 3. Post notification to LinkedIn

Trigger

Type	Path	Caller
Webhook (POST)	`/webhook/brand-heygen-video`	`produce.js` — fires after HeyGen render completes

Payload from produce.js:

{
  "video_url": "https://cdn.heygen.com/...",
  "slot": "video-09",
  "title": "Pillar 09 — ...",
  "hmac": "<HMAC-SHA256 of payload body>"
}

The workflow verifies the HMAC using N8N_WEBHOOK_SECRET before processing.

Execution steps

Step	Node	Action
1	HTTP Request	GET `video_url` → binary MP4
2	AWS S3	PUT `s3://p24-infra/radekkonarski-brand/videos/{slot}.mp4` — ACL `public-read`
3	HTTP Request	POST LinkedIn Share API with video URL and caption

Wasabi public URL after upload: https://s3.eu-central-2.wasabisys.com/p24-infra/radekkonarski-brand/videos/{slot}.mp4

Required credentials (all in SOPS `secrets/n8n-bms4.env.sops`, injected via `start.sh`)

Variable	Used by	Notes
`WASABI_ACCESS_KEY`	S3 upload node	Shared p24-infra Wasabi key
`WASABI_SECRET_KEY`	S3 upload node
`LINKEDIN_ACCESS_TOKEN`	LinkedIn node	OAuth token — expires every 60 days, rotate in `secrets/n8n-bms4.env.sops`
`LINKEDIN_PERSON_URN`	LinkedIn node	`urn:li:person:<id>` — static, does not expire
`N8N_WEBHOOK_SECRET`	HMAC verify node	Shared secret with `produce.js` `.env.local`

Known failure modes

Failure	Symptom	Fix
`produce.js` fires but n8n returns 404	Workflow inactive or wrong webhook path	Re-activate via API: `curl -X PATCH ... {"active": true}`
S3 upload node errors `InvalidAccessKeyId`	`WASABI_ACCESS_KEY` empty in container	Verify in `secrets/n8n-bms4.env.sops`; trigger secrets-sync.yml then run `start.sh`
LinkedIn POST returns 401	`LINKEDIN_ACCESS_TOKEN` expired	Rotate token in `secrets/radekkonarski-brand.env.sops` AND `secrets/n8n-bms4.env.sops`; trigger secrets-sync.yml then run `start.sh`; token valid 60 days
HMAC verification fails	`N8N_WEBHOOK_SECRET` mismatch	Must match value in `.env.local` on developer machine and in `secrets/n8n-bms4.env.sops`
Download times out	HeyGen CDN URL expired	Re-run `produce.js` for that slot; HeyGen URLs expire after ~1h

Re-activating after deactivation

ssh root@54.36.123.110
 
# Check status
curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows/GEfh5Ykv9A7XpHYy \
  | jq '{id,name,active}'
 
# Re-activate
curl -X PATCH \
  -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"active": true}' \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows/GEfh5Ykv9A7XpHYy

mezmo-alert-router — Mezmo Alert Routing (Sc2AKCdslinnSvWs)

Production host: bms-4 (n8n.bms-4.infra.zintegrowana.online)
Status: Active, production, criticality = medium.
Webhook URL: https://n8n.bms-4.infra.zintegrowana.online/webhook/mezmo-alert
Workflow export: n8n-workflows/mezmo-alert-router.json

What it does

Receives Mezmo alert webhooks and routes them to Discord and/or GitHub Issues based on the alert name:

Alert name pattern	Discord color	GH Issue?	Label
`mongotimeout`	Yellow (warning)	Yes	`bug`
`nologs` / `no logs` / `pinbox24down`	Red (critical)	Yes	`bug`
`agent_timeout` / `agent timeout`	Red (critical)	Yes	`human-action`
`triage`	Orange (warning)	Yes	`human-action`
(anything else)	Grey	No	—

Architecture

Mezmo view alert fires
  → POST https://n8n.bms-4.infra.zintegrowana.online/webhook/mezmo-alert
  → Mezmo Alert Webhook node (trigger)
  → Nologs Suppression Guard node (Code — drops Pinbox24 "no logs" false positives; see below)
  → Prepare Alert node (Code — enriches payload, builds discord_payload + gh_payload)
  → ┌─ Send Discord (HTTP POST to Discord webhook URL)
    └─ IF GH Issue → Dedup GH Issue → Create GH Issue (HTTP POST to GitHub API)

The Prepare Alert Code node fans out to both Send Discord and IF GH Issue in parallel. This avoids the n8n 2.26.x issue where running an HTTP node first would overwrite $json with the API response (empty {}), making the downstream IF node unable to read the original Mezmo payload.

Nologs Suppression Guard (`#4130`, hardened `#5704`)

The first node after the webhook. Mezmo’s native “no logs” View alert for Pinbox24/bms-1 misfires while logs are in fact flowing, so this Code node cross-checks a ground-truth Prometheus metric before letting a nologs-type alert (nologs / no logs / pinbox24down) proceed:

Non-nologs alerts pass straight through, untouched.
For a nologs-type alert it queries the mezmo-exporter ground truth sum(pinbox24_backend_log_lines_5m_total) — the per-container gauge (#5605) summed across every Pinbox24 backend on bms-1 (v42-prod / W4 + v32-prod / W3) — on vps-i1, via prometheus.vps-i1.infra.zintegrowana.online/api/v1/query. (#5704 Option 2 repoint — was the legacy unlabeled pinbox24_log_lines_5m_total, which counted v42-prod / W4 stdout only.)
Metric > 0 (fresh logs confirmed) → the alert is suppressed (dropped) — no Discord, no GH issue.
Metric == 0 (genuine no-logs) → the alert passes through and opens a P1 GH issue.

Fail-open by design: if the ground truth cannot be established, the guard does not suppress — a real outage (or a broken exporter reporting 0) still alerts.

#5704 retry hardening: the guard was fail-open with no retry, so a single transient Prometheus query error/timeout at fire time was enough to defeat suppression and open a P1 GH issue while logs were healthily flowing (#5700). The query is now retried before failing open — PROM_MAX_ATTEMPTS = 3 attempts, PROM_BACKOFF_MS = 600 linear backoff (600ms then 1200ms), PROM_TIMEOUT_MS = 6000 per attempt. A definitive numeric answer (including 0) short-circuits the loop immediately; the guard only fails open after all attempts fail. Worst-case added latency (~20s) applies only to nologs-type alerts when Prometheus is fully unreachable.

#5704 Option 2 — repointed to the per-container ground truth (#5605). The guard now queries sum(pinbox24_backend_log_lines_5m_total) (summed across every BACKEND_CONTAINERS entry: v42-prod / W4 + v32-prod / W3) instead of the legacy unlabeled pinbox24_log_lines_5m_total, which counted v42-prod / W4 stdout only — so a W3-only-active quiet period on W4 read 0 and wrongly failed to suppress a host-level false positive.

Why summing is safe (does not mask a genuine per-backend outage): the native Mezmo “no logs” View this guard suppresses is host-level (bms-1) — it fires only when the host stops shipping any Pinbox24 logs, so a W3-only outage never triggers it (W4 keeps the host logging). Genuine per-backend logging gaps are detected independently by the Prometheus Pinbox24NoLogs rule (pinbox24_backend_log_lines_5m_total{container=...} == 0 per backend, #5605), which this guard never touches. A host-level sum() is therefore both correct for the host-level View and non-masking. (Widening to a max_over_time(...) window for Mezmo indexing lag was considered and deliberately not added: it would suppress a genuine short outage for up to the window length, and the #5704 Option 1 retry loop already absorbs the transient-blip class that was the real #5700 trigger.)

Live source of truth: the versioned export is n8n-workflows/mezmo-alert-router.json (secrets templated as {{ PROMETHEUS_BASIC_AUTH_USER }} / {{ PROMETHEUS_BASIC_AUTH_PASSWORD }}). Editing the JSON does not auto-deploy — the change must be imported into the live workflow Sc2AKCdslinnSvWs on bms-4 via the n8n REST API (a sys-admin/infra-task operation).

Required credentials

Variable	Source	Used by
`P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL`	`secrets/n8n-bms4.env.sops`	Discord embed POST
`GITHUB_PAT_ALL_WRITES`	`secrets/n8n-bms4.env.sops`	GitHub Issues API (`Authorization: token <pat>`)

Both variables are injected as environment variables into the n8n container via bms-4/docker-compose.yml and referenced directly inside the Prepare Alert Code node.

Known n8n 2.26.x quirks (discovered during this workflow’s development)

These are bugs/behaviour differences in n8n 2.26.x that affect HTTP Request and IF nodes:

1. `specifyBody` bug — use `specifyBody: "json"` + `jsonBody` parameter

Setting specifyBody: "keypairs" sends an empty body. Setting specifyBody: "string" sends the JSON string as a form field KEY (body becomes {"{\"embeds\":[...]}": ""}), which Discord rejects with 400.

Fix: always use specifyBody: "json" and set the body in the jsonBody parameter (not body). The value must be a JSON string, not a JavaScript object expression.

{
  "sendBody": true,
  "bodyContentType": "json",
  "specifyBody": "json",
  "jsonBody": "={{ $json.discord_payload }}"
}

Where discord_payload is produced by the Code node as JSON.stringify({embeds: [...]}).

2. `isTrue` operator bug — use typeVersion 1 with string equality

In IF nodes with typeVersion: 2.2, the isTrue operator does not reliably route boolean true values to the true branch. Cross-node references like $('PrepareAlert').item.json.create_gh_issue also return nothing after a runOnceForAllItems Code node (no pairedItem metadata).

Fix: use typeVersion: 1 and string comparison:

{
  "type": "n8n-nodes-base.if",
  "typeVersion": 1,
  "parameters": {
    "conditions": {
      "string": [{"value1": "={{ $json.create_gh_issue }}", "operation": "equal", "value2": "yes"}]
    }
  }
}

Output "yes" or "no" strings from the Code node (not boolean true/false).

3. HTTP node overwrites `$json`

After an HTTP Request node runs, $json in downstream nodes becomes the HTTP response body (e.g. {}), not the original upstream data. If you chain: Code → HTTP Discord → IF GH Issue, the IF node sees {} instead of the Code node’s output.

Fix: use parallel fanout — connect the Code node to both the HTTP node and the IF node simultaneously via the connections array:

"Prepare Alert": {
  "main": [[
    {"node": "Send Discord", "type": "main", "index": 0},
    {"node": "IF GH Issue", "type": "main", "index": 0}
  ]]
}

Re-activating after deactivation

ssh root@54.36.123.110
 
# Check status
curl -s -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows/Sc2AKCdslinnSvWs \
  | jq '{id,name,active}'
 
# Re-activate
curl -X PATCH \
  -H "X-N8N-API-KEY: ${BMS4_N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{"active": true}' \
  https://n8n.bms-4.infra.zintegrowana.online/api/v1/workflows/Sc2AKCdslinnSvWs

Testing

# Send a test MongoTimeoutError alert
curl -X POST https://n8n.bms-4.infra.zintegrowana.online/webhook/mezmo-alert \
  -H "Content-Type: application/json" \
  -d '{"body":{"alert":"MongoTimeoutError-test","host":"bms-1","lines":"MongoTimeoutError: connection timed out"}}'
 
# Send a test no-logs alert
curl -X POST https://n8n.bms-4.infra.zintegrowana.online/webhook/mezmo-alert \
  -H "Content-Type: application/json" \
  -d '{"body":{"alert":"No logs from Pinbox24 w3 longer than 24 hours","host":"w3","lines":""}}'

Expected: Discord message appears in the infra-scripts-errors channel within 5 seconds. GH issue created in radieu/p24-infra for MongoTimeoutError. No GH issue for the test fallback.

p24-infra Docs

Explorer

n8n-operations

bms-4 Automation Platform — n8n Operations

Architecture

Active Workflows

Migrated business-automation workflows (ex-Hostinger)

Config Management

Secret Injection — start.sh + SOPS+age

Secrets injected at runtime

Queue Mode

Why queue mode

Capacity

Redis configuration

Scaling workers

Checking worker health

Troubleshooting queue mode

Deployment

Starting / restarting n8n

Fresh install

Update n8n version

Import a workflow

Backup

What needs backing up

Manual backup

Restore

Scenario 1: Container crash

Scenario 2: Fresh install (n8n_postgres_data volume lost)

Healthcheck

Password Rotation

BMS4_N8N_API_KEY

GPS Sync Workflow — atrax, kravag-scheduled-fleet-updates (CCx9UMdphmGficDX)

Triggers

What it writes

Batch update pattern (IMPORTANT — do not revert)

Known failure modes (discovered 2026-05-19, issue #201)

Re-activating a workflow after deactivation

n8n 2.x Upgrade (2026-05-24, issue #220)

What changed

Workflow compatibility audit

n8n 2.x breaking changes (1.x → 2.x)

Deployment procedure

Post-upgrade checklist

Rollback procedure

Troubleshooting

brand-heygen-video — Brand Video Post-Production Pipeline (GEfh5Ykv9A7XpHYy)

What it does

Trigger

Execution steps

Required credentials (all in SOPS secrets/n8n-bms4.env.sops, injected via start.sh)

Known failure modes

Re-activating after deactivation

mezmo-alert-router — Mezmo Alert Routing (Sc2AKCdslinnSvWs)

What it does

Architecture

Nologs Suppression Guard (#4130, hardened #5704)

Required credentials

Known n8n 2.26.x quirks (discovered during this workflow’s development)

1. specifyBody bug — use specifyBody: "json" + jsonBody parameter

2. isTrue operator bug — use typeVersion 1 with string equality

3. HTTP node overwrites $json

Re-activating after deactivation

Testing

Graph View

Table of Contents

Backlinks

Required credentials (all in SOPS `secrets/n8n-bms4.env.sops`, injected via `start.sh`)

Nologs Suppression Guard (`#4130`, hardened `#5704`)

1. `specifyBody` bug — use `specifyBody: "json"` + `jsonBody` parameter

2. `isTrue` operator bug — use typeVersion 1 with string equality

3. HTTP node overwrites `$json`