p24-infra Docs

p4-ovh-bms-1-ns367522-operations

16 min read

p4-ovh-bms-1-ns367522 — Operations & Architecture Workbook

Label: p4-ovh-bms-1-ns367522 Host: 94.23.26.113 Hostname: ns367522 Provider: OVH / Kimsufi (ns367522.ip-94-23-26.eu) Hardware: Intel Core (8 vCPU) · 32 GB RAM · 440 GB RAID (/dev/md127) OS: Ubuntu 20.04.1 LTS ⚠️ EOL April 2025 — upgrade runbook ready, pending human execution (bms1-ubuntu-2204-upgrade.md, 1756). Committed target date: 2026-09-30 (gate table, set by #4540) Role: Pinbox24 SaaS production backend (multi-version) Last major review: 2026-06-18

CRITICAL RISKS

RiskSeverityStatus
Ubuntu 20.04 EOL — No security patches since April 2025. CVEs accumulate daily. Any exploited vulnerability = total production compromise. Also blocks #3725 (Docker Engine 20.10.2 cannot run current Debian images).P1 CRITICAL⏳ Runbook ready — pending human execution. In-place 22.04 upgrade documented in bms1-ubuntu-2204-upgrade.md (#1683, executed under #1756); requires maintenance window + OVH KVM. Committed target date 2026-09-30 (gate table); pre-condition #2389 due 2026-09-05. Flip to Resolved only after the upgrade is confirmed live.
Untagged Docker images — v32-prod-socket, v32-prod-reso run from local-only image ID 63225032f8f8 tagged v32-prod-old:old. If these containers are stopped and removed, they can only be restarted from old image. v41-prod references ECR but tag is absent locally — re-pull needed before restart.P1 CRITICALUnresolved
Private image registry DOWN — private-registry.dev.pinbox24.com (54.38.137.250, OVH) is unreachable (connection refused). Several running containers reference this registry. Their images are cached locally — do not prune! All images are also mirrored to ECR (verified 2026-06-18).P2 HIGHMonitor
Disk space — Cleaned 2026-06-18. Currently 17% used (65G / 440G). Daily logrotate + Docker log caps now active. Monitor weekly.P2 HIGHResolved 2026-06-18
Docker-UFW bypass — Docker bypassed UFW by default; ports 49xxx were internet-accessible. Fixed 2026-06-18 with DOCKER-USER iptables rules blocking 49100–65535. Portainer rebound to 127.0.0.1:9000. Rules persisted via netfilter-persistent.P2 HIGHResolved 2026-06-18

Disaster Recovery Readiness (2026-06-28, #1851)

Full DR runbook: bms1-dr-plan.md — total-loss-of-bms-1 recovery for Pinbox24 W3 + W4.

Key fact: bms-1 is a stateless application tier. The W3/W4 production database is MongoDB rs0 on bms-2 / bms-3 / bms-4 (the app connects out via MONGODB_URL), so losing bms-1 does not lose the database — recovery is “redeploy containers from ECR → point at surviving rs0 → DNS cutover”, not “restore a database”.

ItemStatus
DR runbook written + step-by-step✅ bms1-dr-plan.md
RTO target2–4 h (Scenario A: bms-1 only)
RPO — bms-1 only (rs0 intact)~0 (DB unaffected)
RPO — bms-1 + rs0 lostup to 7 days (weekly Wasabi backup — move to daily, #1851)
All container images in ECR✅ verified 2026-06-18
Off-host copy of /root/container-configs/*.json❌ gap — schedule off-host copy
Off-host copy of per-version .env secrets❌ CRITICAL gap — not in SOPS; managed by git-deploy/GitLab. Biggest RTO blocker
pinbox24.com Cloudflare DNS access confirmed⏳ verify before a disaster
Restore drill timings recorded⏳ run scripts/mongodb-restore-drill.sh

See bms1-dr-plan.md §Pre-disaster prep for the gap-closure task list.

Architecture — w3 / w4 Routing Chain

Internet
   │
   ▼ DNS (Cloudflare) → 94.23.26.113 (bms-1)
   │
nginx-proxy (Docker)
   │ routes by VIRTUAL_HOST env var
   ├── w3.pinbox24.com ──────────── v31-prod (172.17.0.5:80, bridge)
   ├── api.w3.pinbox24.com ──────── v32-prod (172.20.0.12:3000, test-net)
   ├── socket.w3.pinbox24.com ───── v32-prod-socket (172.19.0.11:3000, prod-v-3-net)
   │
   ├── w4.pinbox24.com ──────────── v41-prod (172.17.0.7:80, bridge)
   ├── api.w4.pinbox24.com ──────── v42-prod (172.20.0.13:3000, test-net)
   ├── api-notify.w4.pinbox24.com ─ v42-notify-prod (172.20.0.3:3000, test-net)
   ├── s3-api.w4.pinbox24.com ───── s3-v42-prod (172.20.0.6:3000, test-net)
   ├── s3-v2-api.w4.pinbox24.com ── s3-v2-v42-prod (172.20.0.2:3000, test-net)
   ├── mailgun-api.w4.pinbox24.com  mailgun-v42-prod (172.20.0.4:3000, test-net)
   ├── pdf-gen-api.w4.pinbox24.com  pdf-gen-v42-prod (172.20.0.5:3000, test-net)
   ├── git-deploy-api.w4.pinbox24.com  git-deploy-v42-prod (172.20.0.7:3000, test-net)
   └── portainer.pinbox24.com ───── portainer-pinbox24 (172.17.0.2:9000, bridge)

Key architectural insight: The main domains (w3.pinbox24.com, w4.pinbox24.com) are served by the frontend containers (v31, v41) on port 80. The api.* subdomains go to separate backend containers (v32, v42) on port 3000. This is a split frontend/API architecture — not a single monolith per version.

Docker networks

NetworkPurposeContainers
bridge (172.17.x.x)Default Docker bridgev31-prod, v41-prod, portainer, nginx-proxy
test-net (172.20.x.x)v4.x active productionv42-prod, v42-notify, s3-*, mailgun, pdf-gen, git-deploy, s3-v2
prod-v-4-net (172.18.x.x)v4.x legacy netv42-notify-prod, pdf-gen-v42-prod, s3-v42-prod-02-25-old
prod-v-3-net (172.19.x.x)v3.x productionv32-prod-socket, cron-v32-prod

Image Registry Status (verified 2026-06-18)

Primary registry: AWS ECR

563740926945.dkr.ecr.eu-central-1.amazonaws.com — all active images confirmed present

ECR imageLast pushLocal cacheUsed by
v31-prod:latest2022-05-31✓v31-prod
v32-prod:latest2026-02-04✓v32-prod
v41-prod:latest2023-07-31✓ (untagged locally)v41-prod
v42-prod:latest2025-11-06✓v42-prod
v42-notify-prod:latest2021-01-24✓v42-notify-prod
s3-v2-v42-prod:latest2026-02-26✓s3-v2-v42-prod
mailgun-v42-prod:latest2026-02-26✓mailgun-v42-prod
v4-s3:latest2020-02-21✓s3-v42-prod
old-s3:latest2021-01-24✓s3-v32-prod
v32-cron:latest2021-01-24✓cron-v32-prod
pdf-gen-v42-prod:latest2021-01-24✓pdf-gen-v42-prod
git-deploy-v42-prod:latest2021-04-06✓git-deploy-v42-prod

Secondary registry: private-registry.dev.pinbox24.com

STATUS: DOWN (connection refused, 2026-06-18). Host resolves to 54.38.137.250 (OVH server — probably the old bms-1 private registry that was shut down).

All images from this registry are mirrored to ECR (verified). Do NOT docker image prune on bms-1 — the locally cached private-registry images are the only local copies. If containers using private-registry images are stopped, they can be restarted using ECR equivalents.

Local-only images (no active registry)

Local tagImage IDUsed byRisk
v32-prod-old:old63225032f8f8v32-prod-socket, v32-prod-resoRecoverable from ECR v32-prod
(untagged)466afeafe154v41-prodRecoverable: docker pull 563740926945...v41-prod:latest

To authenticate with ECR before pulling:

aws ecr get-login-password --region eu-central-1 | \
  docker login --username AWS --password-stdin \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com

AWS credentials: SOPS secrets/bms-servers.env.sops. Never hardcode.

Running Application Stacks (2026-06-18)

w3 — Pinbox24 v3.x (legacy, ~4–5 years uptime)

ContainerDomainImageRegistryNote
v31-prodw3.pinbox24.comv31-prod:latestECR ✓Frontend, port 80
v32-prodapi.w3.pinbox24.comv32-prod:latestECR ✓Backend API
v32-prod-socketsocket.w3.pinbox24.comv32-prod-old:oldlocal only ⚠️WebSocket backend
v32-prod-resow3.reso-integration-addrecords.pinbox24.comv32-prod-old:oldlocal only ⚠️RESO integration
s3-v32-prod—old-s3:latestECR ✓S3 microservice
s3-v32-prod-renamed—old-s3:latestECR ✓S3 alias
s3-v32-prod-socket—old-s3:latestECR ✓S3 for socket
s3-v32-prod-reso—old-s3:latestECR ✓S3 for RESO
cron-v32-prod—v32-cron:latestECR ✓Cron jobs
cron-v32-prod-socket—v32-cron:latestECR ✓Cron for socket
cron-v32-prod-reso—v32-cron:latestECR ✓Cron for RESO

w4 — Pinbox24 v4.x (current production, ~3 months uptime)

ContainerDomainImageRegistryNote
v41-prodw4.pinbox24.comv41-prod:latestECR ✓ (re-pull needed)Frontend, port 80
v42-prodapi.w4.pinbox24.comv42-prod:latestECR ✓Backend API
v42-notify-prodapi-notify.w4.pinbox24.comv42-notify-prod:latestECR ✓Push notifications
s3-v42-prods3-api.w4.pinbox24.comv4-s3:latestECR ✓S3 microservice
s3-v2-v42-prods3-v2-api.w4.pinbox24.coms3-v2-v42-prod:latestECR ✓S3 v2
mailgun-v42-prodmailgun-api.w4.pinbox24.commailgun-v42-prod:latestECR ✓Email
pdf-gen-v42-prodpdf-gen-api.w4.pinbox24.compdf-gen-v42-prod:latestECR ✓PDF generation
git-deploy-v42-prodgit-deploy-api.w4.pinbox24.comgit-deploy-v42-prod:latestECR ✓GitLab CI webhook
wkhtml-v42-prod—openlabs/docker-wkhtmltopdf-aasDocker Hubwkhtmltopdf service

Infrastructure

ContainerDomain / PortRole
nginx-proxy:80, :443Auto-routing reverse proxy + TLS termination
nginx-proxy-letsencrypt—Let’s Encrypt cert automation
portainer-pinbox24portainer.pinbox24.com (via nginx-proxy); 127.0.0.1:9000 (local)Docker UI — v1 legacy

Deprecated (stopped 2026-06-18 — do not remove yet)

ContainerStoppedNote
s3-v42-prod-02-25-old2026-06-18Zombie od 2021-01-24. Zero requestów w logach od uruchomienia. Połączony z baza eat1 na zewnętrznym eat-hn1.artnet.pl:30071 (inny tenant, nie w4_db). nginx-proxy miał go jako drugi backend dla s3-api.w4.pinbox24.com (round-robin z nowym), ale v42-prod zawsze wołał nowy przez Docker DNS http://s3-v42-prod:3000. Bezpieczny do usunięcia — poczekaj 30 dni. Restore: docker start s3-v42-prod-02-25-old.

Host-Native Services (outside Docker)

ServicePort / BindStatusNotes
PM2 v5.1.00.0.0.0:3001Stopped 2026-06-18NodeChat v1.0.0 at /temp/p24-v-3.2 — zero connections ever confirmed, heartbeat-only since 2021. Stopped (not deleted) — restore: pm2 start 0. Scheduled for deletion 2026-08-18 if no issues.
PostgreSQL 12127.0.0.1:5432Running since 2025Only default system databases (postgres/template0/template1), 40 MB total, never used by any application. Likely installed as package dependency. No application data, no backup needed. Can be disabled.
node_exporter*:9100RunningPrometheus scrape from vps-i1 only (UFW + iptables)
Netdata v1.19.0127.0.0.1:19999RunningLocal only; not integrated with Prometheus. dbengine_global_errors alarm tuned 2026-07-17 (#4242) — see §Netdata Health Alarm Tuning. No netdatacli on this build → reload health with killall -USR2 netdata.
GitLab runner—ActiveCI/CD deploy runner
LogDNA agent—ActiveShips v42-prod / v42-notify-prod / s3-v2-v42-prod logs to Mezmo

Security Posture (2026-06-18)

Firewall (UFW + iptables)

LayerStatus
UFWActive — default deny incoming
Open from anywhereTCP 22, 80, 443
Open from vps-i1 onlyTCP 9100 (node_exporter)
All other portsBlocked

SSH

SettingValue
PasswordAuthenticationno — key-only
PermitRootLogin(default — key-only enforced)
fail2banActive — sshd jail

Docker-UFW bypass fix (applied 2026-06-18)

Docker bypasses UFW by routing container traffic through FORWARD chain, not INPUT. Fixed by adding rules to DOCKER-USER chain (evaluated before DOCKER chain):

# Current effective DOCKER-USER rules (persisted via netfilter-persistent)
1: ACCEPT  all  ctstate ESTABLISHED,RELATED   # allow return traffic
2: DROP    tcp  multiport dports 49100:65535  # block all ephemeral Docker ports
3: RETURN  all                                # pass through to Docker chain

Portainer rebound to 127.0.0.1:9000 only (was 0.0.0.0:49154). Rules persisted: netfilter-persistent save → /etc/iptables/rules.v4.

Remaining security issues

IssueSeverity
Ubuntu 20.04 EOL — no kernel/package patches available. Upgrade runbook ready (bms1-ubuntu-2204-upgrade.md, #1683), pending human execution under #1756. Committed target date 2026-09-30.P1 CRITICAL
PM2 port 3001 — host process on 0.0.0.0:3001, blocked by UFW but old codeP2
PostgreSQL purpose unknown — local only (OK) but no backup and unknown consumersP2
Portainer v1 — outdated version (2015 Docker UI)P3
wkhtml container — openlabs/docker-wkhtmltopdf-aas:latest from 2015P3

Backup Status (2026-06-18)

What is backed up (Wasabi p24-infra bucket, eu-central-2)

BackupLast verifiedSize
MongoDB w3 DB dump2026-06-157.3 GB
MongoDB w4 DB dump2026-06-152.8 GB
MongoDB admin/freeze2026-06-152.9 GB

Backup script: /opt/scripts/backup-mongodb.sh (or similar — verify with crontab -l).

What is NOT backed up

DataRisk
PostgreSQL /var/lib/postgresql/12/mainP2 — unknown purpose, no backup
Docker container volumesP3 — likely stateless
Application-level user uploads (S3 handled by app)Managed by app via S3 microservice

Restore Procedure

Authoritative restore scripts

GitLab: gitlab.com/pinbox24/p24-server-scripts Cloned on bms-1 at: /home/p24-server-scripts/

ScriptPurpose
main.shFull stack bootstrap: deps → Docker → LogDNA → v3 → v4
start-v3.shStarts entire v3 stack (v31 frontend + v32 backend + socket + reso + s3 + cron)
start-v4.shStarts entire v4 stack (v41 frontend + v42 backend + all microservices)
install-required-dependencies.shapt packages, Docker, AWS CLI
install-required-dockers.shPulls nginx-proxy + letsencrypt companion
create-user.shCreates pinbox24 system user

Note: start-v3.sh / start-v4.sh still reference private-registry.dev.pinbox24.com — that registry is DOWN. Run ECR login first (step 0) so Docker pulls from ECR instead. Skip login-docker.sh (references old private-registry, outdated).

0. Prerequisites — ECR login

AWS credentials in /root/.aws/credentials on bms-1 (SOPS secrets/bms-servers.env.sops — AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY).

aws ecr get-login-password --region eu-central-1 | \
  docker login --username AWS --password-stdin \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com

1. Fresh server restore (disaster recovery)

# New Ubuntu 22.04+ server, min 8 GB RAM
apt update && apt install -y git awscli docker.io docker-compose
 
# Configure ECR credentials
mkdir -p /root/.aws
# Write /root/.aws/credentials — from SOPS secrets/bms-servers.env.sops → AWS_ACCESS_KEY_ID / SECRET
 
# Clone restore scripts
cd /home && git clone https://gitlab.com/pinbox24/p24-server-scripts.git
cd p24-server-scripts && chmod +x *.sh
 
# ECR login (see step 0)
# Set LogDNA key (from SOPS secrets/bms-servers.env.sops or /etc/logdna.conf on old server)
export LOGDNA_KEY='<key from SOPS secrets/bms-servers.env.sops>'
 
./create-user.sh
./install-required-dependencies.sh
./install-required-dockers.sh
./install-logdna-agent.sh
./start-v3.sh
./start-v4.sh

2. Restore MongoDB from Wasabi

# Credentials from SOPS secrets/bms-servers.env.sops → P24_INFRA_WASABI_ACCESS_KEY / SECRET
export AWS_ACCESS_KEY_ID=<key>
export AWS_SECRET_ACCESS_KEY=<secret>
export AWS_DEFAULT_REGION=eu-central-2
 
# List available backups
aws s3 ls s3://p24-infra/ --endpoint-url https://s3.eu-central-2.wasabisys.com
 
# Download latest verified backup (2026-06-15)
aws s3 cp s3://p24-infra/w3_db_2026-06-15.tar.gz /root/ \
  --endpoint-url https://s3.eu-central-2.wasabisys.com
aws s3 cp s3://p24-infra/w4_db_2026-06-15.tar.gz /root/ \
  --endpoint-url https://s3.eu-central-2.wasabisys.com
 
tar -xzf w3_db_2026-06-15.tar.gz
mongorestore --host <mongo-rs0-primary>:27017 w3_db/

3. Fix v41-prod local tag (if untagged after image prune)

aws ecr get-login-password --region eu-central-1 | docker login --username AWS \
  --password-stdin 563740926945.dkr.ecr.eu-central-1.amazonaws.com
docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v41-prod:latest
# Now image has both ECR and private-registry tags — container can be restarted

4. Capture container configs before any maintenance

# Run BEFORE any container stop/rm to preserve restart parameters
mkdir -p /root/container-configs
for c in $(docker ps --format '{{.Names}}'); do
  docker inspect $c > /root/container-configs/$c.json
done

SSH Access

MethodCommand
Human (radieu)ssh root@94.23.26.113 — key ~/.ssh/id_ed25519
Claude agentssh root@94.23.26.113 — VPS_SSH_PRIVATE_KEY in /root/.ssh/authorized_keys
OVH panelIPMI/KVM via OVH manager — server ID 1823494
Portainerhttps://portainer.pinbox24.com (via nginx-proxy); direct local: 127.0.0.1:9000

Day-to-Day Operations

Check container status

docker ps
docker stats --no-stream

Check disk usage

df -h
docker system df
du -sh /var/log/* 2>/dev/null | sort -rh | head -10

View logs

docker logs --tail 100 <container-name>
docker logs --since 1h <container-name>
# LogDNA: https://app.logdna.com (ships v42-prod, v42-notify-prod, s3-v2-v42-prod)

Restart a container

# CAUTION: for v32-prod-socket, v32-prod-reso, v41-prod — capture config first!
docker inspect <container> > /tmp/<container>-config.json
docker restart <container>

Manual ECR deploy (if auto-deploy fails)

docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/<image>:<tag>
docker stop <container> && docker rm <container>
# Recreate from saved inspect config

Log Rotation (configured 2026-06-18)

Config fileTargetPolicy
/etc/logrotate.d/pm2-v42-prod/var/log/v42-prod/pm2/*.logdaily / 7d / 200M / copytruncate
/etc/logrotate.d/v32-prod/var/log/v32-prod/*.logdaily / 7d / 100M / copytruncate
/etc/logrotate.d/s3-v2-v42-prod/var/log/s3-v2-v42-prod/*.logdaily / 7d / 100M
/etc/systemd/journald.confsystemd journalSystemMaxUse=500M
/etc/docker/daemon.jsonDocker container logsmax-size 10m / max-file 5

Port Map

PortBindServiceExternal?
220.0.0.0SSHYes (key-only + fail2ban)
80 / 4430.0.0.0nginx-proxyYes
30010.0.0.0PM2 NodeChatBlocked by UFW
5432127.0.0.1PostgreSQL 12No
8125127.0.0.1Netdata StatsDNo
9000127.0.0.1Portainer (via nginx-proxy)Via nginx-proxy only
9100*node_exportervps-i1 only (UFW)
19999127.0.0.1Netdata UINo
492020.0.0.0 (Docker)v31-prod directBlocked by DOCKER-USER
492230.0.0.0 (Docker)v41-prod directBlocked by DOCKER-USER
172.17.0.1:6379bridgeRedis (disabled 2026-06-18)No

Monitoring

ComponentDetail
node_exporter :9100Prometheus scrape from vps-i1
Grafanagrafana.vps-i1.infra.zintegrowana.online — Servers Overview dashboard
LogDNA / Mezmohttps://app.logdna.com — last 7 days of v42 application logs
Uptime RobotHeartbeat cron every 5 min: */5 * * * * wget --spider <heartbeat-url>

Open Tasks

P1 Critical

  • OS upgrade — Ubuntu 20.04 EOL → 22.04 LTS. Committed target date: 2026-09-30 (gate table); pre-condition #2389 (ECR image backup) due 2026-09-05. Step-by-step runbook ready: bms1-ubuntu-2204-upgrade.md (#1683, executed under #1756). Human-executed: needs a maintenance window + OVH KVM console. Unblocks #1661 (OpenSSH post-quantum) and #3725 (Docker Engine).

P2 High

  • Re-tag v41-prod locally — Done 2026-06-18: docker pull 563740926945...v41-prod:latest succeeded.
  • v32-prod-socket/reso local image — Backed up to ECR as v32-prod:socket-backup-2026-06-18 (2026-06-18). Restart plan: see §v32-prod-socket/reso Migration Plan below.
  • PM2 NodeChat stopped — pm2 stop all executed 2026-06-18. Restore: pm2 start 0. Scheduled for permanent removal 2026-08-18 — Google Calendar reminder set.
  • Automated MongoDB backup — Cron installed 2026-06-19. Script: /usr/local/bin/mongo-backup.sh. Schedule: 0 2 * * * UTC. Logs: /var/log/mongo-backup.log. Creds: /root/backup-creds.env. Wasabi: s3://p24-infra/backups/bms-1/YYYY-MM-DD/. Retain 14 days. See issue #742.

P3 Medium

  • PostgreSQL: empty install — systemctl disable --now postgresql executed 2026-06-18.
  • s3-v42-prod-02-25-old stopped — 2026-06-18. Zero traffic since 2021, different tenant DB. Calendar reminder 2026-07-18 to delete.
  • Portainer v1 upgrade — Upgraded to portainer/portainer-ce:latest (v2) 2026-06-19. Port: 127.0.0.1:9000. Note: v2 starts fresh (no config migration from v1 — re-register Docker endpoint and admin password on first login).
  • v3.x sunset — Active traffic confirmed (see §v3.x Traffic Analysis). Client identification required before sunset. Issue #745.

Low / Future

  • v3.x client verification — DB query (2026-06-19) showed last users login 2019, p24-usages last active Oct 2025, abc3.w3.pinbox24.com hits were all bots. BUT owner believes 2–3 real clients still log in via w3.pinbox24.com — they may use a different auth path not captured in users.updatedAt. Do not sunset v3.x until this is confirmed. Next step: check Pinbox24 app-level session/login logs or ask clients directly.
  • Registry consolidation — Clarify private-registry.dev.pinbox24.com reachability and strategy.
  • Netdata integration — Expose Netdata to Prometheus or replace with node_exporter dashboards.

v3.x Traffic Analysis (2026-06-19)

nginx-proxy log analysis revealed w3 is actively used:

SubdomainHits (nginx log)TypeNote
w3.pinbox24.com~668Monitoring + usersHEAD requests (uptime) + real browsing
socket.w3.pinbox24.com~146Live WebSocketActive socket connections (v32-prod-socket)
abc3.w3.pinbox24.com68Client subdomainSpecific client account on v3.x
api.w3.pinbox24.com~18API callsIncluding real data responses

Conclusion: v3.x has real users. Cannot sunset without identifying and migrating them first. Next step: query Pinbox24 DB for client accounts on *.w3.pinbox24.com domains → Issue #745.

v32-prod-socket/reso Migration Plan (ECR image)

Current state: Both containers run from local v32-prod-old:old (SHA 63225032f8f8). ECR backup exists: v32-prod:socket-backup-2026-06-18.

Risk: socket.w3.pinbox24.com has active WebSocket users. Restart drops all open connections briefly (seconds). Schedule off-peak.

Steps when ready (off-peak window):

# 1. Pull the ECR backup image
aws ecr get-login-password --region eu-central-1 | \
  docker login --username AWS --password-stdin 563740926945.dkr.ecr.eu-central-1.amazonaws.com
docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod:socket-backup-2026-06-18
 
# 2. Recreate v32-prod-socket
docker stop v32-prod-socket
docker rm v32-prod-socket
# Restore from /root/container-configs/v32-prod-socket.json, replace image with ECR tag
# Use: docker inspect output saved at /root/container-configs/v32-prod-socket.json
 
# 3. Same for v32-prod-reso
docker stop v32-prod-reso
docker rm v32-prod-reso
# Restore from /root/container-configs/v32-prod-reso.json

Saved configs at: /root/container-configs/v32-prod-socket.json, v32-prod-reso.json

Known Limitations

  • No IaC — server was provisioned manually; no Ansible playbook to reproduce it.
  • Long-lived containers — some containers have never been recreated (4–5 years). Restarting requires current docker inspect config.
  • EOL OS — no kernel/package patches available for any CVE until OS upgrade. Upgrade runbook: bms1-ubuntu-2204-upgrade.md (#1683, pending human execution).
  • GitLab dependency — auto-deploy depends on GitLab CI and git-deploy-v42-prod container.
  • private-registry DOWN — do not prune images; all are mirrored to ECR but local cache is the working copy.

LogDNA / Mezmo Agent — Key Rotation Runbook

The LogDNA agent runs on bms-1 as a host-native service and ships Pinbox24 application logs (v42-prod, v42-notify-prod, s3-v2-v42-prod) to the Mezmo dashboard. The ingestion API key is stored in /etc/logdna.conf.

Secret name: LOGDNA_API_KEY Secret store: secrets/bms-servers.env.sops (SOPS+age) Dashboard: https://app.logdna.com — Organisation / Account Settings / API Keys

WARNING: Do NOT revoke the old key before confirming logs are flowing with the new key. A wrong key causes a silent log shipping gap — Pinbox24 logs will stop appearing in Mezmo with no application-level error. Confirm in the dashboard before revoking.

Rotation procedure (human-executed)

  1. Generate new key — Mezmo dashboard → Account Settings → API Keys → “Add Key”. Copy the new ingestion key value.

  2. Store in SOPS (dev workstation, PowerShell):

    $env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
sops --decrypt --input-type dotenv --output-type dotenv secrets\bms-servers.env.sops |
  Out-File secrets\bms-servers-edit.env -Encoding utf8
# Edit secrets\bms-servers-edit.env — update LOGDNA_API_KEY=<new-value>
$enc = sops --encrypt --input-type dotenv --output-type dotenv secrets\bms-servers-edit.env
[System.IO.File]::WriteAllText("$PWD\secrets\bms-servers.env.sops", ($enc -join "`n") + "`n",
  [System.Text.UTF8Encoding]::new($false))
Remove-Item secrets\bms-servers-edit.env -Force

  3. Update bms-1 — SSH to root@94.23.26.113:

    # Edit /etc/logdna.conf — replace the key= line with the new value
nano /etc/logdna.conf
# Restart agent
systemctl restart logdna-agent
systemctl status logdna-agent

  4. Verify — wait 2–5 minutes, then check the Mezmo dashboard for fresh log entries from bms-1. Confirm that v42-prod / v42-notify-prod / s3-v2-v42-prod logs are flowing.

  5. Revoke old key — only after confirming logs flow: Mezmo dashboard → delete the old key.

  6. Update compliance — run in Supabase SQL editor (mwkqmgadqnkkihjdeqsi):

    UPDATE dev_r_services
SET compliance_password_rotation = 'yes',
    rotation_freq = '180d',
    compliance_notes = 'LOGDNA_API_KEY stored in secrets/bms-servers.env.sops; rotated manually via Mezmo dashboard + bms-1 /etc/logdna.conf'
WHERE id = '5b4bbcfd-79fe-439e-a222-ead6913f3bad';

  7. Append to rotation log — add a row to docs/secrets-rotation-log.md:

    | <YYYY-MM-DD> | LOGDNA_API_KEY | scheduled (#728) | radieu | yes |

Rotation frequency

180 days (every 6 months). Set a calendar reminder after completing.

Verification commands

# On bms-1: check agent is running and not stuck
systemctl status logdna-agent
journalctl -u logdna-agent --since "5 minutes ago"
 
# Confirm key line is present (do not display value)
grep -c 'key=' /etc/logdna.conf   # should print 1

Netdata Health Alarm Tuning (2026-07-17, #4242)

Problem: the stock netdata.dbengine_global_errors health config ships three alarms, all with crit: $this > 0. One of them — 10min_dbengine_global_flushing_errors — counts transient in-flight page flushes that miss their window during Netdata’s own dbengine datafile rotation (~every 17 min on bms-1). Those spikes (~410–823 errors/10min observed) tripped CRITICAL and paged a human on every rotation cycle, even though I/O errors and FS errors stayed 0 and the disk/RAID were healthy — pure noise, self-recovering, no data loss.

Fix (issue #4242, option 2): a file-level override at /etc/netdata/health.d/dbengine.conf (user health.d overrides the stock /usr/lib/netdata/conf.d/health.d/dbengine.conf — so the override copies all three alarms, not just the tuned one). The two dimensions that mean real trouble keep CRITICAL paging unchanged:

AlarmBeforeAfter
10min_dbengine_global_fs_errorscrit: $this > 0 → sysadminunchanged (still pages CRITICAL)
10min_dbengine_global_io_errorscrit: $this > 0 → sysadminunchanged (still pages CRITICAL)
10min_dbengine_global_flushing_errorscrit: $this > 0 → sysadmin, every: 3swarn: $this > 3000 → to: silent, every: 10s — no crit, non-paging/informational

A genuinely failing disk surfaces first as I/O errors, which still page CRITICAL — so the safety net is intact while the rotation noise is gone.

Tracked copy of the deployed file: config-snapshots/bms-1-netdata-health.d-dbengine.conf (Netdata on bms-1 is not under Ansible/SOPS/CI — this repo copy is the only versioned record; re-apply it on rebuild. Ties into #2407 — bms-1 config outside the pipeline.)

Apply / re-apply:

scp bms-1-netdata-health.d-dbengine.conf root@94.23.26.113:/tmp/dbengine-override.conf
ssh root@94.23.26.113 'cp /tmp/dbengine-override.conf /etc/netdata/health.d/dbengine.conf && killall -USR2 netdata'
# Netdata v1.19.0 has NO netdatacli — USR2 is the reload-health signal on this build.

Verify:

ssh root@94.23.26.113 'curl -s "http://127.0.0.1:19999/api/v1/alarms?all" \
  | python3 -c "import json,sys; a=json.load(sys.stdin)[\"alarms\"]; [print(v[\"name\"],v[\"status\"],\"crit=\",v[\"crit\"],\"warn=\",v[\"warn\"],\"to=\",v[\"recipient\"]) for k,v in a.items() if \"dbengine\" in k]"'
# Expect: flushing_errors → crit=None, warn=$this > 3000, to=silent; fs/io → crit=$this > 0, to=sysadmin

Bundled follow-up applied (same session): /etc/netdata/apps_groups.conf was missing → apps.plugin logged repeated reconnect warnings. Symlinked to the stock default (zero behaviour change, silences the spam): ln -s /usr/lib/netdata/conf.d/apps_groups.conf /etc/netdata/apps_groups.conf.

Pre-existing unrelated noise (NOT fixed here): /usr/lib/netdata/conf.d/health.d/softnet.conf line 13 has an unparse-able warn expression (missing closing paren) — a stock v1.19.0 bug logged at every health reload. Out of scope for #4242; candidate for the eventual Netdata upgrade/replacement (§Open Tasks → “Netdata integration”).

Full runbook: netdata-health-alarm-tuning-bms1.md.

Graph View

Backlinks