p24-infra Docs

bms1-dr-plan

20 min read

Playbook: Pinbox24 (W3 + W4) Disaster Recovery — Total Loss of bms-1

Disaster-recovery runbook for bms-1 (p4-ovh-bms-1-ns367522 · 94.23.26.113 · ns367522 · OVH Kimsufi), the sole Pinbox24 production application host serving w3.pinbox24.com and w4.pinbox24.com.

Human-execution required. Provisioning a replacement server (OVH order), writing AWS/ECR credentials, and the DNS cutover are operator actions. A Claude agent prepares and verifies this runbook but must not order servers or repoint production DNS autonomously. Execution is tracked by the human-action issue spawned from #1851.

Source issue: #1851. Related: #1683 (Ubuntu 20.04 → 22.04 upgrade), #742 (automated MongoDB backup), #3611 (W3 mongoose no-reconnect — root cause of the 2026-07-09 cascading outage), #3660 (this update). Server workbook: p4-ovh-bms-1-ns367522-operations.md.

Last reviewed: 2026-07-09 — folded in findings from the 2026-07-09 W3 cascading outage: the real secrets/config recovery path (SOPS + secrets-sync.yml, not “GitLab CI variables”), the persistent-patches crash-fix mechanism, and the set of containers that run from separate compose files outside secrets-sync.yml management (v32-prod-reso, v32-prod-socket, and several W4 microservices). Items that could not be confirmed from repo files are marked ⚠ VERIFY LIVE — resolve them by read-only SSH inspection before the weekend DR drill; do not treat them as confirmed.

TL;DR — the single most important fact

bms-1 is a stateless application tier. The W3/W4 production database is MongoDB rs0, which runs on bms-2 (PRIMARY) / bms-3 (SECONDARY) / bms-4 (arbiter) — not on bms-1. The Pinbox24 containers connect out to rs0 via MONGODB_URL in their per-version .env files.

Consequence: losing bms-1 does not lose the database. Recovery is “redeploy stateless containers from AWS ECR, point them at the surviving rs0, cut DNS over” — not “restore a database.” The Wasabi MongoDB backup is only needed for the separate, worse scenario where rs0 itself is also lost (see §Scenario B).

Second most important fact (new, 2026-07-09): a restored container is only correct if it is recreated with its persistent-patches host-mounts and current SOPS-sourced secrets. A naïve docker pull + docker run of the ECR image alone would redeploy the exact crash-looping bugs fixed during the 2026-07-09 incident (Twilio/OneSignal require-time crashes, node-redis v5 API incompatibility, mongoose no-reconnect #3611). The recovery driver is therefore secrets-sync.yml, not the old GitLab start-v3.sh/start-v4.sh scripts. See §Secrets & config recovery.

What triggers this runbook

Any event that makes bms-1 unrecoverable or unacceptably degraded:

  • Hardware failure / RAID (/dev/md127) loss / OVH host death
  • OS corruption beyond repair (compounded by Ubuntu 20.04 EOL — no security patches, #1683)
  • Compromise requiring a clean rebuild on a new host
  • OVH account/billing loss of the server (paid until 2026-10)

Confirmation before declaring a disaster:

# From the operator workstation or any peer server
ssh -o ConnectTimeout=10 root@94.23.26.113 'echo alive; uptime' || echo "SSH DOWN"
curl -sS -o /dev/null -w "w4 HTTP %{http_code}\n" https://w4.pinbox24.com/ || echo "w4 DOWN"
curl -sS -o /dev/null -w "w3 HTTP %{http_code}\n" https://w3.pinbox24.com/ || echo "w3 DOWN"
# Grafana "Servers Overview" — node_exporter :9100 scrape gap from vps-i1 confirms host down

If SSH is up but a single container is down → this is not a disaster; use the per-container restart procedure in the server workbook instead. This runbook is for total host loss.

RTO / RPO targets

MetricTargetRealistic (this runbook)Notes
RTO< 4 h2–4 h for Scenario ADominated by OVH provisioning (0–120 min) + DNS TTL. Add ~30–60 min for the containers not covered by secrets-sync.yml (reso/socket, W4 microservices) which are recreated by hand — see §Containers NOT covered
RPO (Scenario A — bms-1 only)~0~0DB is rs0 on bms-2/3, untouched by bms-1 loss
RPO (Scenario B — bms-1 + rs0)< 24 h (goal)up to 7 daysLimited by weekly Wasabi backup — see §Backup gap

RPO caveat (Scenario B). The authoritative off-site MongoDB backup (scripts/mongodb-backup.sh on bms-3) runs weekly (0 1 * * 0), so the worst-case RPO for a full rs0 loss is up to 7 days. Issue #1851 requests moving to daily. Track under §Backup gap. A daily bms-1-local dump (#742) may also exist but is not the cross-host authoritative copy and must not be relied on for bms-1 loss.

Secrets & config recovery — how it ACTUALLY works (READ THIS FIRST)

The single biggest lesson of the 2026-07-09 incident: the DR plan previously said application secrets come from “GitLab CI variables.” That is not how recovery works. The current, verified mechanism is SOPS + secrets-sync.yml, and it also redeploys the container config and the crash-fix patches. Understand this before Scenario A.

The three moving parts, and where each lives

PartSource of truth (in git)Deployed to bms-1 by
Secrets (Mongo URLs, JWT, S3/Wasabi keys, Mailgun, Twilio, OneSignal, PayU…)secrets/pinbox24-w3.env.sops (V32_*), secrets/pinbox24-w4.env.sops (V42_*), secrets/pinbox24-backends.env.sops (mailgun)secrets-sync.yml → /opt/p24-infra/bms-1/pinbox24-{w3,w4,backends}.env, then rendered into the build-dir backend-environment.env / s3-environment.env
Container config (image, ports, networks, mounts, restart policy)infra-src/pinbox24/w3/docker-compose.yml, infra-src/pinbox24/w4/docker-compose.ymlsecrets-sync.yml → build-dir docker-compose.yml
Crash-fix code (persistent-patches/)infra-src/pinbox24/w3/persistent-patches/*, infra-src/pinbox24/w4/persistent-patches/*secrets-sync.yml → build-dir persistent-patches/ (host-mounted :ro into the container)

Because all three are now committed to radieu/p24-infra, the git repo IS the off-host backup for the secrets-sync-managed containers (v32-prod, s3-v32-prod, v42-prod, s3-v42-prod, mailgun-v42-prod). This closes most of the old §Secrets gap for those five containers.

Canonical build directories on bms-1 (⚠ VERIFY LIVE — see path-drift note)

StackBuild dir the compose + patches live inenv files
W3 (v32-prod, s3-v32-prod)/home/gitlab-runner/builds/eZQeLfuJe/0/pinbox24/p24-v-3.2/backend-environment.env, s3-environment.env
W4 (v42-prod, s3-v42-prod, wkhtml)/root/builds/7N4sbbrB/0/pinbox24/p24-back-ts/backend-environment.env, s3-environment.env
mailgun-prod/root/mailgun-prod/mailgun-environment.env

⚠ VERIFY LIVE — build-dir path drift. secrets-sync.yml (sync-pinbox24-w3) and the committed compose files use the GitLab-runner build paths above (.../eZQeLfuJe/..., .../7N4sbbrB/...) — these are authoritative. Older notes also reference a root-owned /root/builds/pn3C9eHo/0/... W3 copy (not the prod serving dir, but still an active W3-staging mount source — do not delete; #4985), and the issue mentions /home/p24-server-scripts/v3/{v32,v3reso,v3socket,v31}/. These runner-hash paths change if the GitLab runner is re-registered (see gitlab-runner-bms1-reregister.md). On a fresh DR host the path will be whatever the new runner registration produces — confirm the live path with docker inspect <container> --format '{{json .Mounts}}' on the current host now (pre-drill) and pin it in this table before the drill.

DR recovery via secrets-sync (the fast path for the 5 managed containers)

Once the replacement host is provisioned, reachable by SSH, has Docker + a registered GitLab runner, and is ECR-logged-in (Scenario A Steps 0–1), the managed containers are recreated by triggering the workflow — no manual env editing:

# From the operator workstation (needs gh auth + repo access):
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-w3
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-w4
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-backends
# Or target=bms1-all for file-only delivery WITHOUT container restart (use when you want to
# stage env/compose/patches first, then recreate containers by hand in the correct order).

Each sync-pinbox24-* job: scps the decrypted .env to /opt/p24-infra/bms-1/, scps the committed docker-compose.yml + persistent-patches/ into the build dir, renders backend-environment.env / s3-environment.env from the canonical env, then docker-compose up -d --force-recreate.

⚠ VERIFY LIVE — the DR host must look like bms-1 to secrets-sync.yml. The workflow SSHes to the hard-coded IP 94.23.26.113 with key id_bms1 and writes to the hard-coded build paths. For a DR host on a new IP, either (a) repoint the workflow’s target IP/paths first, or (b) skip the workflow and reproduce its steps by hand from the committed files (scp compose + patches, render env from /opt/p24-infra/bms-1/*.env, docker-compose up -d --force-recreate). Decide which before the drill and document the chosen path here.

persistent-patches — the crash-fixes a rebuild MUST carry

These host-mounted :ro JS files override buggy code inside the ECR images. They were added/updated during the 2026-07-08 and 2026-07-09 incidents. A container recreated without them redeploys the bugs. All are committed in this repo and deployed by secrets-sync.yml.

W3 — infra-src/pinbox24/w3/persistent-patches/ → mounted into v32-prod / s3-v32-prod

Patch fileMount target (in container)Fixes
mongoose.js/app/app-backend/config/mongoose.js#3611 — driver has no built-in reconnect; on a transient replica-set connect failure at boot, every buffered mongoose query hangs bufferTimeoutMS (10s) then 422s forever until recreate. Patch adds a reconnect-on-error path. This was the 2026-07-09 outage.
twilioService.js/app/app-backend/services/twilioService.jsTwilio client construction guarded — no longer crashes at require-time when twilioAccountSid is unset
pushNotification.helper.js/app/app-backend/helperFunction/pushNotification.helper.jsOneSignal client guarded — no require-time crash when OneSignal keys unset
crudService.js/app/app-backend/helperFunction/crudService.jscallback() on files without bucket in async.forEach (2026-07-06, B2)
filesUpload.js/app/app-backend/controllers/filesUpload.jsregId/recId fallback from query params (2026-07-06, B1)
local.js/app/config/local.js (s3-v32-prod)MongoDB URL from DB_URI, Wasabi S3 from s3Bucket_* env
controller.js/app/build/src/api/storage/controller.js (s3-v32-prod)upload handler saves file metadata to MongoDB after Wasabi upload

(Also mounted from the app build, not persistent-patches/: excel_import.js — a stock, git-tracked file in p24-v-3.2 (not a p24-infra patch), byte-identical to the image copy so the mount is a redundant no-op. Restored by the GitLab checkout; needs no p24-infra restore. #4708.)

W4 — infra-src/pinbox24/w4/persistent-patches/ → mounted into v42-prod

Patch fileMount targetFixes
uploadAwsS3.helper.js/app/dist/globalHelpers/uploadAwsS3.helper.jsS3 upload helper hotfix
postbookReport.helper.js/app/dist/apps/postbook/postbookReport.helper.jspostbook report helper hotfix
gus-api-regon-wsdl (dir)/app/dist/wsdlGUS/REGON WSDL crash-loop (v42-prod-gus-wsdl-crash-loop.md) — missing WSDL made v42-prod crash-loop
ecosystem.config.js/app/ecosystem.config.jsPM2 process config (pm2-runtime start)

DR rule: after any recreate, confirm the mounts are present: docker inspect <name> --format '{{range .Mounts}}{{.Source}} -> {{.Destination}}{{"\n"}}{{end}}' and confirm the container is NOT crash-looping (docker ps uptime climbing, logs clean of the #3611 mongoose hang / Twilio / OneSignal require errors).

Inventory — what must come back

Pinbox24 container stack (all images in AWS ECR)

Primary registry: 563740926945.dkr.ecr.eu-central-1.amazonaws.com (region eu-central-1). All active images confirmed mirrored to ECR (verified 2026-06-18). The secondary private-registry.dev.pinbox24.com is DOWN — do not rely on it; pull everything from ECR.

Legend: ✅ = recreated by secrets-sync.yml (config+secrets+patches from git); ⚠ = NOT covered by secrets-sync.yml, config/env lives only on bms-1 → see §Containers NOT covered.

W3 — Pinbox24 v3.x (legacy, real traffic — do not skip):

ContainerDomainECR imagesecrets-syncNote
v31-prodw3.pinbox24.comv31-prod:latest⚠Frontend, :80
v32-prodapi.w3.pinbox24.comv32-prod:latest✅Backend API — sync-pinbox24-w3
s3-v32-prod—old-s3:latest✅S3 microservice — sync-pinbox24-w3
v32-prod-socketsocket.w3.pinbox24.comv32-prod:socket-backup-2026-06-18⚠Live WebSocket users. Separate compose (/home/p24-server-scripts/v3/v3socket/ ⚠ VERIFY LIVE)
v32-prod-resow3.reso-integration-addrecords.pinbox24.comv32-prod:socket-backup-2026-06-18⚠RESO integration. Separate compose (/home/p24-server-scripts/v3/v3reso/ ⚠ VERIFY LIVE)
cron-v32-prod (+ -socket, -reso)—v32-cron:latest⚠Cron jobs

W4 — Pinbox24 v4.x (current production):

ContainerDomainECR imagesecrets-syncNote
v41-prodw4.pinbox24.comv41-prod:latest⚠Frontend, :80. CI/ECR build path restored 2026-07-11 — GitLab pinbox24/pinbox24-version-4, existing .gitlab-ci.yml re-enabled on the live bms-1-autodeploy runner (see docs/playbooks/v41-prod-ecr-rebuild.md); secrets-sync coverage still ⚠ (config/env not yet under secrets-sync.yml)
v42-prodapi.w4.pinbox24.comv42-prod:latest✅Backend API — sync-pinbox24-w4 (PM2, patches + WSDL)
s3-v42-prods3-api.w4.pinbox24.comv4-s3:latest✅S3 microservice — sync-pinbox24-w4
mailgun-v42-prodmailgun-api.w4.pinbox24.commailgun-v42-prod:latest✅Email — sync-pinbox24-backends
v42-notify-prodapi-notify.w4.pinbox24.comv42-notify-prod:latest⚠Push notifications
s3-v2-v42-prods3-v2-api.w4.pinbox24.coms3-v2-v42-prod:latest⚠S3 v2
pdf-gen-v42-prodpdf-gen-api.w4.pinbox24.compdf-gen-v42-prod:latest⚠PDF generation
git-deploy-v42-prodgit-deploy-api.w4.pinbox24.comgit-deploy-v42-prod:latest⚠GitLab CI webhook
wkhtml-v42-prod—openlabs/docker-wkhtmltopdf-aas (Docker Hub)✅wkhtmltopdf (in W4 compose)

Infrastructure: nginx-proxy (jwilder/nginx-proxy, :80/:443), nginx-proxy-letsencrypt (jrcs/letsencrypt-nginx-proxy-companion), portainer-pinbox24 (optional, not required for service).

Skip on rebuild (do not recreate): s3-v42-prod-02-25-old (zombie since 2021, different tenant DB), host-native PM2 NodeChat (stopped, zero traffic), host-native Redis (disabled), PostgreSQL 12 (empty system DBs only).

⚠ VERIFY LIVE — inventory truth. This table is reconstructed from committed compose files, secrets-sync.yml, and prior docs. Before the drill, run docker ps -a on bms-1 and reconcile: confirm the exact running set, the reso/socket/cron compose file locations, and each container’s current image tag. Record the reconciled list in the server workbook.

Stateful data inventory (what is / isn’t backed up)

Data on bms-1Stateful?BackupDR action
MongoDB W3/W4 dataNo — lives in external rs0 (bms-2/3/4)rs0 replication + weekly Wasabi dumpNothing — survives bms-1 loss (Scenario A)
Container imagesNo — in ECRECR (verified)docker pull from ECR
secrets-sync-managed secrets (v32/s3, v42/s3, mailgun)YesSOPS in git (secrets/pinbox24-{w3,w4,backends}.env.sops)Re-run secrets-sync.yml — gap closed
secrets-sync-managed config + patchesYesgit (infra-src/pinbox24/w{3,4}/)Deployed by secrets-sync.yml — gap closed
⚠ Separate-compose container env/config (reso, socket, cron, notify, s3-v2, pdf-gen, git-deploy, frontends)Yes — secrets/configNone off-host (bms-1-local only)CRITICAL — see §Containers NOT covered
nginx-proxy / Let’s Encrypt certsVolumeNoneAcceptable loss — re-issued automatically on new host (HTTP-01)
git-deploy-v42-prod git cacheVolumeNone (re-clones from GitLab)Acceptable loss — regenerated on first deploy
S3/Wasabi object dataNo — externalManaged by appNothing — proxy containers are stateless
PostgreSQL 12Empty system DBs onlyNone neededAcceptable loss
docker inspect snapshots of long-lived containersConfigOff-host copy neededSource of truth for never-recreated containers — see §Pre-disaster prep

Containers NOT covered by secrets-sync.yml (the 2026-07-09 blind spot)

secrets-sync.yml force-recreates only v32-prod+s3-v32-prod, v42-prod+s3-v42-prod, and mailgun-v42-prod. Every other production container’s env and compose config lives only on bms-1 and would be lost with the host:

  • W3: v32-prod-reso, v32-prod-socket, cron-v32-prod (+-socket, -reso), v31-prod
  • W4: v42-notify-prod, s3-v2-v42-prod, pdf-gen-v42-prod, git-deploy-v42-prod, v41-prod

Two consequences from the incident:

  1. Credential currency drift (⚠ VERIFY LIVE). During 2026-07-09 the w3_app MongoDB password was rotated several times. v32-prod-reso / v32-prod-socket were not touched by the incident fixes (they run from separate compose files). They may therefore still be running on a stale w3_app password — currently healthy only because their existing connections are alive. On a DR restore they would be recreated from whatever .env was captured, which must carry the current credential, not a stale one. Confirm live before the drill.
  2. No off-host config. There is no git or Wasabi copy of these containers’ .env/compose, so a DR restore has nothing to recreate them from except a pre-disaster backup that does not yet exist.

Decision required (pre-drill, human): for each ⚠ container, choose one and document it here:

  • (A) Bring under secrets-sync.yml — add its keys to the relevant SOPS file and a sync-* job/step (preferred for anything credential-bearing, esp. reso/socket which share the w3_app credential and thus MUST track its rotations).
  • (B) Keep separate, but back up off-host — add its compose + .env to the new Wasabi config-backup job (§Pre-disaster prep 2) and document exactly how its credentials are updated today and how a DR restore sources the current (not stale) values.

Until this decision is executed, these containers are a known RTO risk — budget extra manual recreate time in Scenario A Step 4.

Scenario A — bms-1 lost, rs0 intact (the expected case)

This is the fast path: redeploy the stateless tier, point it at the live rs0, cut DNS.

Step 0 — Order + provision replacement server (0–140 min)

# 1. Order OVH Kimsufi / SoYouStart / ECO-4 (min 8 GB RAM; bms-1 spec: 8 vCPU / 32 GB / RAID).
#    Account: radieu@gmail.com. OVH order is the long pole (0–120 min provisioning).
# 2. Base OS: Ubuntu 22.04 LTS (NOT 20.04 — bms-1's EOL OS is being retired, #1683).
# 3. Add SSH key + base hardening (UFW deny-in, allow 22/80/443; fail2ban).
apt update && apt install -y git awscli docker.io docker-compose ufw fail2ban
ufw default deny incoming && ufw allow 22 && ufw allow 80 && ufw allow 443 && ufw --force enable
# 4. Register a GitLab runner if you intend to drive recovery via secrets-sync build paths
#    (see gitlab-runner-bms1-reregister.md). NOTE: the runner hash becomes the build-dir path —
#    it will NOT match the committed compose paths; see the ⚠ path-drift note above.

Step 1 — AWS / ECR credentials + login

mkdir -p /root/.aws
# Write /root/.aws/credentials from SOPS — NEVER hardcode or echo the values:
#   AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY  in  secrets/bms-servers.env.sops
# Safe extraction pattern (run on a host with the age key / from CI):
#   K=$(sops -d --input-type dotenv --output-type dotenv secrets/bms-servers.env.sops \
#        | grep '^AWS_ACCESS_KEY_ID=' | cut -d= -f2-); printf '[default]\naws_access_key_id=%s\n' "$K" >> /root/.aws/credentials; unset K
#   (repeat for AWS_SECRET_ACCESS_KEY; add 'region = eu-central-1')
 
aws ecr get-login-password --region eu-central-1 | \
  docker login --username AWS --password-stdin \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com

Step 2 — Restore secrets + config + patches for the 5 managed containers (via secrets-sync)

This replaces the old “recover .env from an off-host copy” step for the managed containers — the source of truth is SOPS in git, deployed by secrets-sync.yml. See §Secrets & config recovery for the full mechanism and the ⚠ VERIFY LIVE caveats (target IP / build paths).

# Recreate the Docker networks the stack expects FIRST (compose files declare them external):
docker network create test-net      2>/dev/null || true   # 172.20.x — v4.x active prod + v32 stack
docker network create prod-v-3-net  2>/dev/null || true   # 172.19.x — v3.x (socket, cron)
docker network create prod-v-4-net  2>/dev/null || true   # 172.18.x — v4.x
 
# Then trigger the managed-container recovery from the operator workstation:
#   gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-w3
#   gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-w4
#   gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=pinbox24-backends
# If the DR host is on a new IP, DO NOT rely on the workflow (it targets 94.23.26.113) — instead
# reproduce its steps by hand from the committed files:
#   - scp infra-src/pinbox24/w{3,4}/docker-compose.yml + persistent-patches/ to the build dir
#   - render backend-environment.env / s3-environment.env from /opt/p24-infra/bms-1/*.env
#     (which itself comes from `sops -d secrets/pinbox24-{w3,w4,backends}.env.sops`)
#   - docker-compose up -d --force-recreate backend s3   (W4 also needs CONTAINER_NAME/IMAGE_NAME
#     from V42_CONTAINER_NAME / V42_IMAGE_NAME in the env; mailgun uses mailgun-environment.env)
# Confirm every MONGODB_URL / DB_URI points at the surviving rs0 (bms-2 145.239.133.104 / bms-3 51.68.155.224).

Step 3 — Restore the ⚠ NOT-covered containers (manual, from off-host backup)

# reso, socket, cron (W3) + notify, s3-v2, pdf-gen, git-deploy (W4) + frontends v31/v41 are NOT
# recreated by secrets-sync. Recover them from the Wasabi config-backup (§Pre-disaster prep #1/#2):
#   - restore each container's compose file + .env to its build/compose dir
#   - recreate from the saved `docker inspect` config for the long-lived never-recreated ones
#     (v32-prod-socket, v32-prod-reso, v31-prod, v41-prod)
# ⚠ Use CURRENT credentials — reso/socket share w3_app which was rotated on 2026-07-09; a stale
#   .env backup would deploy the old password. Cross-check against secrets/pinbox24-w3.env.sops.
# Fallback if no backup exists (worst case): escalate to the Pinbox24 owner (human-gated) and
# reconstruct from the GitLab p24-server-scripts repo + current SOPS credentials.
cd /home && git clone https://gitlab.com/pinbox24/p24-server-scripts.git   # legacy fallback only

Step 4 — Start order (dependencies first)

  1. nginx-proxy + nginx-proxy-letsencrypt (routing + TLS; certs re-issue via Let’s Encrypt)
  2. W3 backends: v32-prod, v32-prod-socket, v32-prod-reso + their s3-* / cron-*
  3. W4 backends + microservices: v42-prod, v42-notify-prod, s3-v42-prod, s3-v2-v42-prod, mailgun-v42-prod, pdf-gen-v42-prod, wkhtml-v42-prod, git-deploy-v42-prod
  4. Frontends last: v31-prod, v41-prod

After each recreate, run the persistent-patches mount check (see §persistent-patches DR rule) and confirm the container is not crash-looping.

Step 5 — Smoke test every endpoint (against the NEW IP, before DNS cutover)

NEW_IP=<new-server-ip>
for h in w3.pinbox24.com api.w3.pinbox24.com socket.w3.pinbox24.com \
         w4.pinbox24.com api.w4.pinbox24.com api-notify.w4.pinbox24.com \
         s3-api.w4.pinbox24.com s3-v2-api.w4.pinbox24.com mailgun-api.w4.pinbox24.com \
         pdf-gen-api.w4.pinbox24.com git-deploy-api.w4.pinbox24.com; do
  code=$(curl -sS -o /dev/null -w '%{http_code}' --resolve "$h:443:$NEW_IP" "https://$h/" || echo ERR)
  echo "$h -> $code"
done
# Confirm app→DB: hit an api.w4 endpoint that reads data; verify a real response (DB reachable).
# Confirm #3611 fix present: an api.w3 route that goes through mongoose (e.g. /api/i18n/langs)
# returns promptly — NOT a 10s hang → 422 (that would mean the mongoose.js patch is missing).

Step 6 — DNS cutover (Cloudflare, TTL 60s)

# pinbox24.com is on Cloudflare (separate from the p24-infra zintegrowana.online account —
# CONFIRM which Cloudflare account/zone holds pinbox24.com before a disaster; see §Pre-disaster prep
# and docs/playbooks/cloudflare-pinbox24-onboarding.md).
# Lower TTL to 60s in advance if a planned migration; in a real disaster repoint all A records:
#   w3, api.w3, socket.w3, w3.reso-integration-addrecords,
#   w4, api.w4, api-notify.w4, s3-api.w4, s3-v2-api.w4, mailgun-api.w4, pdf-gen-api.w4,
#   git-deploy-api.w4, portainer  ->  <new-server-ip>
# After propagation, Let's Encrypt companion issues fresh certs (HTTP-01) automatically.

Step 7 — Monitor for 1 h

  • Watch Grafana “Servers Overview” (node_exporter on the new host — install + open :9100 to vps-i1).
  • Tail container logs; confirm socket.w3.pinbox24.com WebSocket reconnects.
  • Confirm no crash-loops on the patched containers (mongoose/Twilio/OneSignal errors absent).
  • Confirm LogDNA/Mezmo shipping resumes (re-install agent + key from secrets/bms-servers.env.sops).

Scenario B — bms-1 AND rs0 both lost (worst case)

Add database restore from Wasabi before Step 4 above. RPO = up to 7 days (weekly backup).

# Stand up a MongoDB 7.0 instance (or recover rs0) first, then restore the latest off-site dump.
# Backup path (produced weekly by scripts/mongodb-backup.sh on bms-3), retention 28 days:
#   s3://p24-infra/mongodb/full/mongodb-full-YYYY-MM-DD.tar.gz
export AWS_ACCESS_KEY_ID=...        # P24_INFRA_WASABI_ACCESS_KEY  (secrets/monitoring.env.sops)
export AWS_SECRET_ACCESS_KEY=...    # P24_INFRA_WASABI_SECRET_KEY
export AWS_DEFAULT_REGION=eu-central-2
aws s3 ls s3://p24-infra/mongodb/full/ --endpoint-url https://s3.eu-central-2.wasabisys.com
aws s3 cp s3://p24-infra/mongodb/full/mongodb-full-<DATE>.tar.gz /root/ \
  --endpoint-url https://s3.eu-central-2.wasabisys.com
tar -xzf /root/mongodb-full-<DATE>.tar.gz -C /root/
mongorestore --host <new-mongo-primary>:27017 /root/mongodb-full-<DATE>/
# Then point each app .env MONGODB_URL at the restored host and continue from Scenario A Step 4.

The restore timing (w3_db + w4_db) is measured by the quarterly drill scripts/mongodb-restore-drill.sh — record the measured restore duration in docs/playbooks/mongodb-restore-drill.md and fold it into the RTO above once available.

Pre-disaster prep (do these NOW — they make the runbook executable)

These are the gaps that would block or slow a real recovery. Track each as its own task.

  1. Off-host copy of the ⚠ NOT-covered containers’ compose + .env — the biggest remaining blocker after 2026-07-09. Schedule a job (mirroring scripts/mongodb-backup.sh → Wasabi) that copies, for v32-prod-reso, v32-prod-socket, cron-v32-prod*, v31-prod, v42-notify-prod, s3-v2-v42-prod, pdf-gen-v42-prod, git-deploy-v42-prod, v41-prod: their compose file and .env to s3://p24-infra/bms1-config/. Redact nothing at rest but ensure the Wasabi bucket is private. (Prefer instead: bring the credential-bearing ones under secrets-sync.yml — see §Containers NOT covered decision.)
  2. Off-host copy of container configs — schedule a job that copies each long-lived container’s docker inspect JSON (/root/container-configs/*.json) off bms-1 to s3://p24-infra/bms1-config/inspect/. Without them the 4–5-year-old never-recreated containers (v32-prod-socket, v32-prod-reso, v31-prod, v41-prod) lose their exact env/port/network params.
  3. Confirm pinbox24.com Cloudflare account/zone + API access — the DNS cutover is on the critical path; do not discover the access gap during an outage. Pre-lower TTL is only possible if access is confirmed. See docs/playbooks/cloudflare-pinbox24-onboarding.md.
  4. Resolve the §Containers NOT covered decision for reso/socket + W4 uncovered set (bring under secrets-sync.yml vs. keep-separate-plus-backup). Reso/socket share the w3_app credential and MUST track its rotations — prioritise these.
  5. Move MongoDB backup weekly → daily (#1851 / #742) to cut Scenario-B RPO from 7 days to 24 h.
  6. Run the restore drill (scripts/mongodb-restore-drill.sh) and record real timings.
  7. Pin the live build-dir paths (⚠ path-drift note) and reconcile the live docker ps -a inventory into the server workbook before the drill.
  8. Document GitLab-script drift in docs/playbooks/pinbox24-deploy-delta.md (the GitLab start-v3.sh/start-v4.sh predate the current container config — the committed compose + docker inspect configs are authoritative; the scripts are a fallback).

Known gaps / risks

Secrets gap — PARTIALLY CLOSED (2026-07-09)

The secrets-sync-managed containers (v32-prod, s3-v32-prod, v42-prod, s3-v42-prod, mailgun-v42-prod) now recover their secrets from SOPS in git (secrets/pinbox24-{w3,w4,backends}.env.sops) and their config + crash-patches from infra-src/pinbox24/w{3,4}/. The remaining gap is the ⚠ NOT-covered containers (reso, socket, cron, notify, s3-v2, pdf-gen, git-deploy, frontends), whose env/config is still bms-1-local with no off-host copy — see §Containers NOT covered and Pre-disaster prep 4. Until those are backed up or brought under secrets-sync, a DR restore of those specific containers is still human-gated.

Patch-loss risk (NEW)

A container recreated without its persistent-patches mounts redeploys the crash bugs fixed on 2026-07-08/09. The patches are committed and secrets-sync-deployed for the managed containers; verify the mounts post-recreate (§persistent-patches DR rule). This is now the top functional (as opposed to availability) DR risk.

Backup gap (RPO)

Authoritative off-site MongoDB dump is weekly → Scenario-B RPO up to 7 days. Move to daily.

Image / registry risk

private-registry.dev.pinbox24.com is DOWN; everything is mirrored to ECR (verified 2026-06-18), but the verification is a point-in-time check — re-confirm ECR completeness periodically (a new image pushed only to the private registry would be unrecoverable).

No IaC

bms-1 was provisioned manually; there is no Ansible playbook to reproduce it. If 4 h RTO proves unachievable in a drill, evaluate writing an Ansible role for bms-1 DR (issue #1851 decision item).

Human-action items (do NOT attempt autonomously)

ActionOwnerWhenNotes
OVH replacement server orderOperator (radieu@gmail.com)On disaster declarationLong pole (0–120 min)
pinbox24.com Cloudflare DNS access confirm + cutoverOperatorPre-drill (confirm) / on disaster (cutover)Critical path
Pinbox24 .env for ⚠ NOT-covered containers if no off-host backupPinbox24 ownerOn disaster if backup missingBlocks reso/socket/notify/etc. recovery
Confirm reso/socket credential currency vs. w3_app rotationsOperator + ownerPre-drill⚠ VERIFY LIVE — may run on stale password
§Containers NOT covered decision (bring under secrets-sync vs. backup)OperatorPre-drillPrioritise reso/socket

Escalation path

  1. Declare disaster — post to Discord P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (red) + open a GH issue in radieu/p24-infra (label bug, P1) with the confirmation-command output.
  2. Human-gated items (open immediately, in parallel — do not block the rest): see the Human-action items table above.
  3. If rs0 also lost → escalate to a db-admin (bms-2) session for MongoDB restore from Wasabi.
  4. Sign-off — close the disaster issue only after Step 7 monitoring is clean and RTO/RPO are recorded for the post-mortem.

Graph View

Backlinks