p24-infra Docs

bms-1-v3x-sunset-migration-plan

10 min read

bms-1 — Pinbox24 v3.x Sunset & Migration Plan

Issue: #745 Status: In Progress — awaiting client identification and business decision Last updated: 2026-06-23 Owner: radieu Prerequisite reading: p4-ovh-bms-1-ns367522-operations.md

1. Executive Summary

Pinbox24 v3.x (w3.pinbox24.com) runs on bms-1 alongside the active v4.x stack. Traffic analysis (2026-06-19) confirmed real users are still connecting to v3.x — WebSocket sessions are live, client-specific subdomain abc3.w3.pinbox24.com is active, and API calls with real data responses are occurring. Sunset cannot proceed until every remaining client is identified, migrated to v4.x, and confirmed to no longer be using v3.x.

This document is the single authoritative reference for:

  • The v3.x container inventory and their recoverability status
  • The investigation steps needed to identify remaining clients
  • The per-client migration / cutover procedure
  • The decommission sequence and post-decommission cleanup
  • The timeline and blockers

2. v3.x Container Inventory

All containers listed below serve w3.pinbox24.com traffic. Do not stop any of these without first confirming zero active clients.

2.1 Web-facing containers

ContainerDomainImageRegistryRecoverability
v31-prodw3.pinbox24.com (frontend, port 80)v31-prod:latestECR 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v31-prod:latest (pushed 2022-05-31)Safe — pull from ECR
v32-prodapi.w3.pinbox24.com (backend API, port 3000)v32-prod:latestECR (pushed 2026-02-04)Safe — pull from ECR
v32-prod-socketsocket.w3.pinbox24.com (WebSocket backend)v32-prod-old:old (SHA 63225032f8f8)ECR backup: v32-prod:socket-backup-2026-06-18Recoverable via ECR backup
v32-prod-resow3.reso-integration-addrecords.pinbox24.com (RESO integration)v32-prod-old:old (SHA 63225032f8f8)ECR backup: v32-prod:socket-backup-2026-06-18Recoverable via ECR backup

2.2 Supporting microservices (no public domain — internal to v3.x stack)

ContainerImageRegistryPurpose
s3-v32-prodold-s3:latestECRS3 proxy for v32-prod
s3-v32-prod-renamedold-s3:latestECRS3 proxy alias
s3-v32-prod-socketold-s3:latestECRS3 proxy for v32-prod-socket
s3-v32-prod-resoold-s3:latestECRS3 proxy for v32-prod-reso
cron-v32-prodv32-cron:latestECRScheduled jobs for v32-prod
cron-v32-prod-socketv32-cron:latestECRCron for socket instance
cron-v32-prod-resov32-cron:latestECRCron for RESO instance

2.3 Docker network

All v3.x containers share the prod-v-3-net network (172.19.x.x). The web-facing containers also reach nginx-proxy via the default bridge network (172.17.x.x).

2.4 Image risk register

RiskContainers affectedMitigation status
Local-only image v32-prod-old:old (SHA 63225032f8f8) — not in any registryv32-prod-socket, v32-prod-resoMitigated 2026-06-18 — image backed up to ECR as v32-prod:socket-backup-2026-06-18. Container configs saved at /root/container-configs/.
private-registry.dev.pinbox24.com DOWN (54.38.137.250)All containers that originally referenced this registryMitigated — all images confirmed present in ECR (2026-06-18). Local cache intact. Do NOT docker image prune.
v41-prod locally untaggedv41-prod (v4.x, not v3.x)Fixed 2026-06-18 via ECR pull.

3. Traffic Analysis (2026-06-19)

nginx-proxy log sample from bms-1 confirmed the following w3 activity:

SubdomainRequest countTraffic typeInterpretation
w3.pinbox24.com~668HEAD (monitoring) + GET (real browsers)Uptime checks + active users
socket.w3.pinbox24.com~146WebSocket upgrade + keepalivesLive WebSocket sessions
abc3.w3.pinbox24.com68Browser GET requestsSpecific client still active
api.w3.pinbox24.com~18POST/GET with real data responsesAPI-level usage

Conclusion: v3.x has live end users. At minimum one client (abc3) is confirmed active. The DB investigation below is required to enumerate all clients before any decommission action.

4. Investigation Steps (Required Before Any Decommission)

These steps require production access to bms-1 and the Pinbox24 v3.x database. They are human-gated — an operator with DB access must run these queries.

4.1 Identify all v3.x client accounts

# SSH to bms-1
ssh root@94.23.26.113
 
# Connect to the Pinbox24 v3.x database (MongoDB rs0 primary — bms-2 or bms-3)
# Credentials: SOPS secrets/bms-servers.env.sops (MONGO_RS0_USER + MONGO_RS0_PASSWORD)
mongo "mongodb://bms-2:27017,bms-3:27017,bms-4:27017/w3_db?replicaSet=rs0" \
  -u <user> -p <password> --authenticationDatabase admin
 
# Query: find all client accounts and their last-login timestamps
db.users.aggregate([
  { $group: {
      _id: "$company",
      lastLogin: { $max: "$updatedAt" },
      userCount: { $sum: 1 }
  }},
  { $sort: { lastLogin: -1 } }
])
 
# Query: active session tokens (indicates clients that have logged in recently)
db.sessions.find(
  { createdAt: { $gte: new Date(Date.now() - 90 * 24 * 60 * 60 * 1000) } },
  { _id: 0, userId: 1, createdAt: 1, userAgent: 1 }
).sort({ createdAt: -1 }).limit(50)
 
# Query: p24-usages — last activity per account
db["p24-usages"].aggregate([
  { $group: {
      _id: "$account",
      lastActivity: { $max: "$date" },
      totalOps: { $sum: 1 }
  }},
  { $match: { lastActivity: { $gte: new Date("2026-01-01") } } },
  { $sort: { lastActivity: -1 } }
])

4.2 Identify v3.x subdomains in use

Each v3.x client connects via a dedicated subdomain (<slug>.w3.pinbox24.com). Cross-reference DB accounts with nginx-proxy logs to find all active subdomains:

# On bms-1: extract unique subdomains from nginx-proxy access log (last 30 days)
docker logs nginx-proxy 2>&1 | \
  grep -oP 'Host: \K[^\s]+\.w3\.pinbox24\.com' | \
  sort | uniq -c | sort -rn
 
# Or from nginx-proxy log files (if volume-mounted)
grep -oP '"[A-Z]+ https?://\K[^/]+' /var/log/nginx/access.log | \
  grep '\.w3\.pinbox24\.com' | sort | uniq -c | sort -rn

4.3 Confirm traffic is human (not bot/monitoring)

Known non-human traffic patterns to exclude from the client list:

  • HEAD requests to w3.pinbox24.com — uptime monitoring (Uptime Robot / internal heartbeat)
  • Requests with no User-Agent or generic scanner agent strings
  • abc3.w3.pinbox24.com requests from known bot ASNs

Filter for genuine sessions:

docker logs nginx-proxy 2>&1 | \
  grep '\.w3\.pinbox24\.com' | \
  grep -v '"HEAD ' | \
  grep -v 'UptimeRobot\|StatusCake\|Pingdom\|bot\|spider\|crawler' | \
  awk '{print $1, $7, $12}' | sort | uniq -c | sort -rn | head -50

5. Migration Path — v3.x to v4.x

Once clients are identified, each must be migrated to w4.pinbox24.com. The migration involves:

5.1 Per-client migration checklist

For each active v3.x client (<slug>.w3.pinbox24.com):

  • Notify client — email/call with migration date (minimum 30 days notice)
  • Provision v4.x account — create client account on api.w4.pinbox24.com
  • Data migration — export client data from w3_db, import to w4_db
    • Script: verify with Pinbox24 dev team (gitlab.com/pinbox24/p24-server-scripts)
  • DNS cutover — update Cloudflare <slug>.w3.pinbox24.com CNAME to point to w4.pinbox24.com
  • Verify — confirm client can log in and access their data on v4.x
  • Monitor — watch nginx-proxy logs for continued w3 requests from this client (should drop to zero)
  • Decommission subdomain — remove VIRTUAL_HOST entry from v3.x nginx config for this client after 14-day silent period

5.2 Domain/subdomain strategy

Current routing in nginx-proxy uses the VIRTUAL_HOST environment variable on each container. The main routing entries to decommission are:

nginx-proxy VIRTUAL_HOSTContainerAction
w3.pinbox24.comv31-prodRemove after all clients migrated
api.w3.pinbox24.comv32-prodRemove after all clients migrated
socket.w3.pinbox24.comv32-prod-socketRemove after WebSocket clients confirmed off
abc3.w3.pinbox24.com(via v31-prod or v32-prod)Client-specific — decommission per client
w3.reso-integration-addrecords.pinbox24.comv32-prod-resoRemove after RESO clients migrated

To remove a specific subdomain from nginx-proxy routing without stopping the container:

# Option A: recreate the container without the subdomain in VIRTUAL_HOST
# Option B: remove the Cloudflare DNS record for the subdomain
# Preferred: Option B (remove DNS record first — zero downtime, instant rollback)

5.3 WebSocket migration consideration

socket.w3.pinbox24.com serves WebSocket connections. Migrating WebSocket clients requires:

  1. Ensure api-notify.w4.pinbox24.com (v42-notify-prod) covers the same functionality
  2. Update client-side app config to use the new WebSocket endpoint
  3. Coordinate a low-traffic window for the final cutover (existing sessions will drop on restart)

This is the highest-risk migration step — coordinate with Pinbox24 dev team before executing.

6. Decommission Sequence

Execute only after all identified clients have migrated and their w3 traffic has been zero for 14+ days.

Phase 1 — Remove public DNS records (Day 0)

# Via p24-infra dns-manager.py or Cloudflare API
# Remove or redirect to 404: w3.pinbox24.com, *.w3.pinbox24.com
python3 /opt/p24-infra/scripts/dns-manager.py delete w3.pinbox24.com
python3 /opt/p24-infra/scripts/dns-manager.py delete api.w3.pinbox24.com
python3 /opt/p24-infra/scripts/dns-manager.py delete socket.w3.pinbox24.com
# Repeat for all client subdomains identified in §4.2

Phase 2 — Stop and confirm zero traffic (Day 0 → Day +7)

Watch nginx-proxy logs for 7 days after DNS removal. Any requests arriving for w3 subdomains will 503 (nginx-proxy has no route) — confirm the count trends to zero.

Phase 3 — Stop v3.x containers (Day +7)

# Capture current configs before stopping (safety)
mkdir -p /root/container-configs-v3x-final-$(date +%Y%m%d)
for c in v31-prod v32-prod v32-prod-socket v32-prod-reso \
          s3-v32-prod s3-v32-prod-renamed s3-v32-prod-socket s3-v32-prod-reso \
          cron-v32-prod cron-v32-prod-socket cron-v32-prod-reso; do
  docker inspect $c > /root/container-configs-v3x-final-$(date +%Y%m%d)/$c.json 2>/dev/null || true
done
 
# Stop all v3.x containers
docker stop v31-prod v32-prod v32-prod-socket v32-prod-reso \
            s3-v32-prod s3-v32-prod-renamed s3-v32-prod-socket s3-v32-prod-reso \
            cron-v32-prod cron-v32-prod-socket cron-v32-prod-reso
 
# Monitor for 48 hours — if no incident, proceed to Phase 4

Phase 4 — Remove v3.x containers (Day +9)

docker rm v31-prod v32-prod v32-prod-socket v32-prod-reso \
          s3-v32-prod s3-v32-prod-renamed s3-v32-prod-socket s3-v32-prod-reso \
          cron-v32-prod cron-v32-prod-socket cron-v32-prod-reso
 
# Remove v3.x Docker network
docker network rm prod-v-3-net

Phase 5 — Image cleanup (Day +30)

Only after a 30-day incident-free period post-decommission:

# Remove local v3.x images (ECR remains as permanent archive)
docker rmi v32-prod-old:old
docker rmi $(docker images old-s3 -q)
docker rmi $(docker images v32-cron -q)
docker rmi $(docker images v31-prod -q)
 
# Verify images that remain are all v4.x
docker images

Phase 6 — Create sub-issues for remaining cleanup (same day as Phase 5)

After v3.x is confirmed decommissioned, open the following sub-issues:

IssueTitleWhy
featbms-1: v31-prod final removal + ECR tag cleanupRemove ECR tags for v31 (last push 2022)
featbms-1: migrate v32-prod-socket/reso to v42 equivalent or decommissionSocket/RESO may be unused post-sunset
featbms-1: bms-1 OS upgrade Ubuntu 20.04 → 22.04 LTSRequired after v3.x removal reduces restart risk

7. Dependencies

DependencyOwnerBlocking?
Pinbox24 DB access to run client queries (§4)radieu (Pinbox24 business owner)Yes — blocks client identification
Client communication and agreed migration dateradieu + clientsYes — blocks v3.x cutover
v4.x data migration toolingPinbox24 dev teamYes — needed for client data migration
AWS ECR credentialsSOPS secrets/bms-servers.env.sops (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY)No (already available)
Cloudflare DNS edit tokenSOPS secrets/monitoring.env.sops (CLOUDFLARE_TOKEN_ZINTEGROWANA)No (already available)

8. Risks

RiskProbabilityImpactMitigation
Cutting off an active client by stopping v3.x too earlyMediumHIGH — production outage for that clientMandatory: complete §4 investigation before any container stop
v32-prod-socket image lossLow (already backed up to ECR)HIGH — WebSocket service unrecoverable without hostAlready mitigated 2026-06-18 (see §2.4)
bms-1 disk pressure during migrationLow (disk at 17%)LowMonitor with df -h weekly
v4.x feature gap — a client relies on v3.x-only functionalityUnknownHIGHVerify with clients during migration scoping
MongoDB w3_db size — migration of 7.3 GB during business hoursLowMediumSchedule migration window at night (UTC 01:00–04:00)

9. Timeline

This is a human-gated plan. Dates are targets, not hard deadlines.

MilestoneTarget dateStatusBlocker
Complete §4 DB investigation2026-07-07PendingPinbox24 DB access
Client list compiled2026-07-10PendingDepends on §4
Client communication sent2026-07-14PendingClient list
Migration window agreed with each client2026-07-28PendingClient response
All clients migrated to v4.x2026-08-18PendingClient migration
v3.x DNS removed2026-08-18PendingClient sign-off
v3.x containers stopped2026-08-25Pending7-day zero-traffic window
v3.x containers removed2026-08-27Pending48-hour quiet period
v3.x images removed2026-09-27Pending30-day incident-free
Sub-issues opened for remaining cleanup2026-09-27PendingPhase 5 complete

Note: The PM2 NodeChat process (stopped 2026-06-18) is already scheduled for deletion on 2026-08-18 (Google Calendar reminder). This aligns with the v3.x sunset window.

10. Rollback Procedure

At any phase, v3.x can be fully restored:

# Restore containers from saved configs
for c in v31-prod v32-prod v32-prod-socket v32-prod-reso \
          s3-v32-prod s3-v32-prod-renamed s3-v32-prod-socket s3-v32-prod-reso \
          cron-v32-prod cron-v32-prod-socket cron-v32-prod-reso; do
  # Recreate from saved docker inspect JSON
  # Use: docker run with params extracted from /root/container-configs-v3x-final-<date>/$c.json
  docker start $c 2>/dev/null || echo "Need to recreate $c from config"
done
 
# For v32-prod-socket / v32-prod-reso (local-only image):
aws ecr get-login-password --region eu-central-1 | \
  docker login --username AWS --password-stdin \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com
docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod:socket-backup-2026-06-18
docker tag 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod:socket-backup-2026-06-18 \
  v32-prod-old:old
# Then recreate container from saved config
 
# Restore DNS (Cloudflare)
# Re-add A records: w3.pinbox24.com, *.w3.pinbox24.com → 94.23.26.113
# DNS propagation: 1–5 minutes (Cloudflare orange-cloud)
ReferenceLink
bms-1 operations workbookp4-ovh-bms-1-ns367522-operations.md
MongoDB rs0 operationsmongodb-rs0-operations.md
Cloudflare DNS management09-networking-dns
Issue #742 — automated MongoDB backupGitHub issue #742
Issue #745 — this issueGitHub issue #745

12. Open Questions (Human Decision Required)

The following questions must be answered by the business owner before implementation proceeds:

  1. Who are the 2–3 clients the owner believes are still using v3.x? (Owner mentioned this in §Low/Future of the bms-1 ops doc.) Can they be contacted directly?
  2. Is w3.reso-integration-addrecords.pinbox24.com serving a paying client or is it an abandoned integration? If the latter, it can be decommissioned independently ahead of the full v3.x sunset.
  3. Does v4.x fully cover v3.x functionality? Specifically: WebSocket notifications (v42-notify-prod covers push; does it replace socket.w3?) and RESO integration equivalents.
  4. What is the sunset notice period required in client contracts? This determines whether the 2026-08-18 target is achievable.

Plan created by Claude Code worker — issue #745. Owner: radieu.

Graph View

Backlinks