Playbook: bms-1 In-Place Upgrade — Ubuntu 20.04 EOL → 22.04 LTS

Runbook for the in-place do-release-upgrade of bms-1 (p4-ovh-bms-1-ns367522 · 94.23.26.113 · ns367522 · OVH Kimsufi) from Ubuntu 20.04.1 LTS (EOL April 2025) to Ubuntu 22.04 LTS (supported until April 2027).

Human-execution required. bms-1 is the Pinbox24 production app server (24 Docker containers serving w3.pinbox24.com / w4.pinbox24.com). The upgrade needs a maintenance window, OVH console/KVM access for interactive apt prompts, and live judgement on config-keep decisions. A Claude agent cannot and must not run do-release-upgrade autonomously. This document is the step-by-step runbook the operator follows; execution is tracked by the human-action issue created from #1683.

Tracking issues: #1683 (this upgrade) · #1756 (human execution of this runbook) · #3725 (Docker Engine 20.10.2 glibc/clone3 incompatibility — resolved by this same upgrade) · #1661 (OpenSSH post-quantum on bms-1 — unblocked by reaching 22.04). Server workbook: p4-ovh-bms-1-ns367522-operations.md.

What triggers this

Ubuntu 20.04 reached end-of-standard-support in April 2025: no more USN security advisories, CVEs accumulate unpatched on a public production host.
OpenSSH on 20.04 caps at 8.2p1 (no 9.x in repos/backports) — post-quantum KEX (#1661) is blocked until the OS reaches 22.04 (ships OpenSSH 8.9p1, 9.x available via the 22.04 chain).
Docker Engine 20.10.2 on this host cannot run current Debian-based images (glibc ≥ 2.34 uses the clone3 syscall, which the shipped seccomp profile rejects) — #3725. Every new bms-1 service has to re-apply the python:3.11-slim-bullseye pin as a workaround. The 22.04 upgrade carries a current Docker Engine and retires that workaround.

Committed target date & SLA

This is the canonical record of the committed date for the bms-1 EOL resolution. The D12 monthly check in compliance-audit-policy.md §Domain 12 (“bms-1 EOL resolution … has an active GH issue with target date”) reads this table. Set by compliance-gap issue #4540 (D12 audit run 2026-07-29, audit row 9440774a-c298-4eee-90f2-80f4c92d99cf).

Gate	Date	Owner	Why this date
Pre-condition — `v32-prod-socket` / `v32-prod-reso` local-only images secured to ECR (#2389)	2026-09-05	sys-admin	Step 2 below is CRITICAL and #2389 is not started: image `63225032f8f8` exists only on bms-1 and cannot be re-pulled if the upgrade fails
Pre-condition — OVH full-disk image captured (Step 1, ref `bms1-dr-plan.md`)	day-of, before Step 4	human operator	Only rollback path — Kimsufi has no instance snapshots
Upgrade executed (#1756)	2026-09-30	human operator — OVH KVM + maintenance window	~9 weeks from the audit: clears #2389 and allows a production window to be booked and announced for a 24-container host (30–90 min expected, 2 h worst case)
Review checkpoint	2026-08-28	D12 monthly audit (`next_audit_due`)	Confirms the pre-conditions are on track; aligns the check with the existing audit cadence rather than a separate reminder

Escalation rule. If 2026-09-30 passes without a confirmed-healthy upgrade, the next D12 run escalates the bms-1 EOL finding from partial (HIGH) to a P0 row in docs/priorities.md and files a fresh compliance-gap issue referencing this slip. The date does not silently roll forward.

Amending the date. The human owner may move it — edit this table and the Target date block on #1756 together. Do not record a date in only one of the two places; the whole point of #4540 was that an active tracking issue carried no date at all.

Pre-conditions before booking the window

Local-only images secured to ECR (#2389, due 2026-09-05 — see the gate table above). This is the one pre-condition that cannot be done inside the window; Step 2 below verifies it.
Maintenance window scheduled and announced (production — expect 30–90 min, plan for a 2 h worst case including rollback), on or before the committed target date 2026-09-30.
Operator has OVH Manager / IPMI-KVM access (server ID 1823494) — do NOT rely on SSH alone; a failed upgrade can drop the network and the only recovery path is the OVH console.
SSH session opened inside tmux or screen so a dropped connection does not kill the upgrade mid-flight.
A second SSH session / OVH KVM kept open as a lifeline.

Step 0 — Capture current state (rollback baseline)

Run on bms-1 as root and save the output off-server (paste into the human-action issue):

lsb_release -a                                   # confirm starting point: Ubuntu 20.04.1
uname -r                                         # current kernel
ssh -V                                           # baseline OpenSSH (expect 8.2p1)
python3 --version                                # baseline Python (expect 3.8.x)
docker ps --format '{{.Names}}' | sort | tee /root/pre-upgrade-containers.txt
docker ps --format '{{.Names}}' | wc -l          # baseline container count (expect ~24)
df -h /                                           # baseline disk (expect ~17% used)
ufw status verbose                                # baseline firewall
iptables -L DOCKER-USER -n --line-numbers         # baseline Docker-UFW bypass rules

Capture every container’s restart config before any restart (long-lived containers, 4–5 y uptime, cannot be re-derived):

mkdir -p /root/container-configs
for c in $(docker ps -a --format '{{.Names}}'); do
  docker inspect "$c" > "/root/container-configs/$c.json"
done
tar -czf /root/container-configs-pre-2204.tar.gz -C /root container-configs

SECURITY NOTE (issue #2970): Each $c.json file contains the full Env section — all container environment variables including secrets and credentials. Treat these files as highly sensitive: do not cat them in a terminal session, do not copy them to a non-root-only location, and delete them (rm -rf /root/container-configs) after the upgrade is complete. Store the .tar.gz backup in Wasabi with server-side encryption, not on public/shared storage.

Step 1 — OVH snapshot (mandatory rollback point)

The only reliable rollback for an in-place release upgrade is a full-disk image taken before it starts. Kimsufi/So-you-Start dedicated servers do not have OVH “instance snapshots” like Public Cloud VPS — use one of these, in order of preference:

OVH Manager → server → Backup / “Réinstaller depuis une sauvegarde” if a backup storage plan is attached to this server. Trigger a fresh full backup and wait for it to complete before proceeding.

Manual full-disk image to Wasabi if no OVH backup plan exists (most Kimsufi). Boot the server into OVH rescue mode (Manager → Netboot → Rescue → reboot), then from the rescue OS:

# In OVH rescue mode — md127 is the bms-1 RAID array (see workbook)
# Stream a compressed block image straight to Wasabi (p24-infra bucket, eu-central-2).
# Credentials: SOPS secrets/bms-servers.env.sops -> P24_INFRA_WASABI_ACCESS_KEY / SECRET.
export AWS_ACCESS_KEY_ID=<from SOPS>          # never echo the value
export AWS_SECRET_ACCESS_KEY=<from SOPS>
export AWS_DEFAULT_REGION=eu-central-2
dd if=/dev/md127 bs=64M status=progress | gzip -1 | \
  aws s3 cp - s3://p24-infra/backups/bms-1/disk-image-pre-2204-$(date +%F).img.gz \
    --endpoint-url https://s3.eu-central-2.wasabisys.com

Then reboot back to the normal disk (Manager → Netboot → Hard disk → reboot).

Do not skip this. Without a disk image there is no rollback — a failed do-release-upgrade on production with no image = full disaster-recovery rebuild from §Restore Procedure in the workbook, with the local-only Docker images at risk (Step 2).

App-data safety net (in addition to the disk image): confirm the MongoDB backup cron ran recently (/usr/local/bin/mongo-backup.sh, 0 2 * * * UTC → s3://p24-infra/backups/bms-1/):

tail -5 /var/log/mongo-backup.log
aws s3 ls s3://p24-infra/backups/bms-1/ --endpoint-url https://s3.eu-central-2.wasabisys.com | tail

Step 2 — Docker image audit (CRITICAL — local-only images)

Two production containers run from a local-only image with no registry copy. If the Docker daemon does not restart cleanly after the OS upgrade and these containers are lost, they cannot be re-pulled. Secure them to ECR before upgrading.

Container	Local image	Image ID	Registry status
`v32-prod-socket`	`v32-prod-old:old`	`63225032f8f8`	local only — back up to ECR first
`v32-prod-reso`	`v32-prod-old:old`	`63225032f8f8`	local only — back up to ECR first
`v41-prod`	(untagged) `466afeafe154`	—	ECR `v41-prod:latest` — re-pull recoverable

Per the workbook, an ECR backup v32-prod:socket-backup-2026-06-18 already exists. Verify it is still present and, if in any doubt, push a fresh backup tag before the window.

# 2a. Authenticate to ECR (creds: /root/.aws/credentials, from SOPS secrets/bms-servers.env.sops)
aws ecr get-login-password --region eu-central-1 | \
  docker login --username AWS --password-stdin \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com
 
# 2b. Confirm the existing local-only backup tag is in ECR (no re-push needed if present)
aws ecr describe-images --region eu-central-1 --repository-name v32-prod \
  --query "imageDetails[].imageTags" --output text | tr '\t' '\n' | grep socket-backup || echo "MISSING"
 
# 2c. If MISSING — tag the local image and push a dated backup before upgrading
docker tag 63225032f8f8 \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod:socket-backup-$(date +%F)
docker push \
  563740926945.dkr.ecr.eu-central-1.amazonaws.com/v32-prod:socket-backup-$(date +%F)
 
# 2d. Confirm v41-prod has its ECR tag present locally (re-pull path)
docker pull 563740926945.dkr.ecr.eu-central-1.amazonaws.com/v41-prod:latest

Do NOT docker image prune at any point — the locally cached private-registry.dev.pinbox24.com images are the only working copies (that registry is DOWN). See workbook §Image Registry Status.

Step 3 — Pin Docker Engine (prevent surprise daemon/container restarts)

22.04 repos may ship a newer docker-ce. An unplanned Engine upgrade can restart every container with changed daemon defaults mid-upgrade. Hold the Docker packages across the release upgrade:

# Hold whichever Docker packages are installed (docker-ce OR distro docker.io)
apt-mark hold docker-ce docker-ce-cli containerd.io 2>/dev/null || true
apt-mark hold docker.io containerd 2>/dev/null || true
apt-mark showhold                                  # confirm the holds are set

After the upgrade is confirmed healthy (Step 7) you can apt-mark unhold and upgrade Docker deliberately in a separate, controlled change.

Step 4 — Pre-upgrade compatibility audit

4a. Python 3.8 → 3.10

22.04 ships Python 3.10 (vs 3.8 on 20.04). Audit host-native scripts for 3.8-only syntax before upgrading — anything importing removed stdlib or using deprecated patterns breaks post-boot:

# Find host-native python scripts (Docker containers carry their own interpreters — unaffected)
grep -rEl '^#!.*python3' /usr/local/bin /opt/scripts /root 2>/dev/null
# Spot-check each for known 3.9/3.10 removals (e.g. removed `collections` ABC aliases,
# `asyncio.coroutine`, `cgi`/`distutils` deprecations). Note findings in the human-action issue.

Pinbox24 app code runs inside Docker containers, which bundle their own Python/Node — the host Python bump does not affect them. Only host-level helper scripts matter here.

4b. Refresh the current release fully (required by do-release-upgrade)

apt update && apt upgrade -y
apt --purge autoremove -y
apt install -y update-manager-core
grep -i '^Prompt' /etc/update-manager/release-upgrades   # should be 'lts'

Step 5 — Run the release upgrade

# MUST be inside tmux/screen with OVH KVM as a lifeline.
do-release-upgrade

Interactive prompt answers (recommended)

Prompt	Answer	Why
”Continue running under SSH?” / start a second sshd on :1022	Yes	Lets you reconnect on :1022 if the main sshd restarts mid-upgrade. Keep the KVM open regardless.
”Do you want to start the upgrade?” (package summary)	y	Proceed once the to-remove list has no production-critical packages.
`sshd_config` — “A new version of /etc/ssh/sshd_config is available”	keep the local version currently installed (`N` / keep)	Preserves `PasswordAuthentication no` + key-only hardening. Manually merge any new defaults afterwards.
nginx config — “replace /etc/nginx/…“	keep the local version currently installed (`N` / keep)	nginx-proxy routing depends on `VIRTUAL_HOST`/custom site configs — a replacement would break w3/w4 routing.
`/etc/docker/daemon.json` or any Docker conf prompt	keep the local version	Preserves the log-rotation caps (`max-size 10m / max-file 5`) and bridge settings.
Any other `*.conf` / service config prompt	keep the local version unless you know the change is wanted	Default-safe: never let the upgrader silently overwrite tuned configs. Note every kept/replaced file in the issue.
`grub` — install to which disk?	the existing boot device (RAID member shown)	Keep boot config consistent with the `/dev/md127` array.
”Remove obsolete packages?“	y	Safe — removes only superseded 20.04 packages.
”Restart the system to finish?“	y	Reboot completes the upgrade.

If the connection drops, reconnect via the :1022 sshd (ssh -p 1022 root@94.23.26.113) or the OVH KVM console and re-attach the tmux/screen session (tmux attach).

Step 6 — First boot

ssh root@94.23.26.113          # main sshd should be back on :22
tmux attach 2>/dev/null || true

If SSH does not come back within a few minutes, use the OVH KVM console to log in locally and check systemctl status ssh / journalctl -xe.

Step 7 — Post-upgrade verification checklist

Run each; record results in the human-action issue. All must pass before declaring success.

# OS + kernel
lsb_release -a                       # EXPECT: Ubuntu 22.04 LTS  (Description: Ubuntu 22.04.x LTS)
uname -r                             # EXPECT: 5.15.x (22.04 GA kernel) or newer
 
# OpenSSH (unblocks #1661)
ssh -V                               # EXPECT: OpenSSH_8.9p1 or newer
systemctl is-active ssh              # EXPECT: active
 
# Docker daemon + container count
systemctl is-active docker           # EXPECT: active
docker ps --format '{{.Names}}' | wc -l   # EXPECT: ~24 (compare to /root/pre-upgrade-containers.txt)
diff <(docker ps --format '{{.Names}}' | sort) /root/pre-upgrade-containers.txt && echo "ALL CONTAINERS UP"
 
# Local-only images survived
docker images | grep -E 'v32-prod-old|466afeafe154' || echo "CHECK: local-only image missing — restore from ECR (Step 2)"
 
# Application HTTP health (production endpoints)
curl -s -o /dev/null -w '%{http_code}\n' https://w3.pinbox24.com     # EXPECT: 200
curl -s -o /dev/null -w '%{http_code}\n' https://w4.pinbox24.com     # EXPECT: 200
 
# Firewall + Docker-UFW bypass rules persisted
ufw status verbose                                  # EXPECT: active, default deny incoming
iptables -L DOCKER-USER -n --line-numbers           # EXPECT: ESTABLISHED/RELATED accept + 49100:65535 drop
# (if DOCKER-USER rules are gone, re-apply from workbook §Docker-UFW bypass fix, then netfilter-persistent save)
 
# Python bump
python3 --version                    # EXPECT: 3.10.x — re-test any host scripts flagged in Step 4a
 
# fail2ban / node_exporter still up
systemctl is-active fail2ban         # EXPECT: active
curl -s -o /dev/null -w '%{http_code}\n' http://localhost:9100/metrics   # EXPECT: 200

If any container did not come back: restore it from its saved config — /root/container-configs/<name>.json — pulling the image from ECR if needed (workbook §Restore Procedure). For the local-only v32-prod-socket / v32-prod-reso, use the ECR backup tag from Step 2.

Step 8 — Post-upgrade follow-ups

# Release the Docker hold and plan a separate, deliberate Engine upgrade if desired
apt-mark unhold docker-ce docker-ce-cli containerd.io docker.io containerd 2>/dev/null || true
 
# OpenSSH 9.x / post-quantum KEX (#1661): 22.04 ships 8.9p1; for 9.x evaluate jammy backports /
# hwe, then re-run the PQ KEX test from #1661 and close it.

Documentation updates after a confirmed healthy upgrade (do these only once verification passes — they record real state for every agent that reads them):

docs/servers/p4-ovh-bms-1-ns367522-operations.md — set the OS header to the new 22.04.x version and flip the Ubuntu 20.04 EOL risk row to Resolved (YYYY-MM-DD).
CLAUDE.md — bms-1 OS field → Ubuntu 22.04.x LTS (drop the (EOL — upgrade required) note).
Re-test and close #1661 (OpenSSH post-quantum on bms-1).
Append the maintenance to the bms-1 workbook §Open Tasks and note the disk-image backup location in Wasabi so it can be cleaned up after a safe period.

Rollback procedure

If the upgrade fails (boot loop, mass container failure, or production endpoints down and not quickly recoverable):

Stop digging. Do not attempt further in-place fixes on a half-upgraded production host.
Boot OVH rescue mode (Manager → Netboot → Rescue → reboot).

Restore the pre-upgrade disk image captured in Step 1:

# In rescue mode — restore the Wasabi disk image back onto the RAID array
export AWS_ACCESS_KEY_ID=<from SOPS secrets/bms-servers.env.sops>
export AWS_SECRET_ACCESS_KEY=<from SOPS>
export AWS_DEFAULT_REGION=eu-central-2
aws s3 cp s3://p24-infra/backups/bms-1/disk-image-pre-2204-<DATE>.img.gz - \
  --endpoint-url https://s3.eu-central-2.wasabisys.com | gunzip | \
  dd of=/dev/md127 bs=64M status=progress
sync

(If an OVH backup-plan image was used instead, restore via Manager → “Réinstaller depuis une sauvegarde”.)

Reboot to disk (Netboot → Hard disk → reboot). Server returns to 20.04 with all containers.
Re-run Step 7 verification against the restored 20.04 system.
Escalate: post the failure detail (which step, console output) to the human-action issue and re-plan. Do not retry the upgrade without addressing the root cause.

Worst case — image restore itself fails: rebuild from scratch per the workbook §Restore Procedure (fresh 22.04 server + p24-server-scripts + MongoDB restore from Wasabi).

Escalation / error reporting

Per the p24-infra Error Notification Standard, on any failure during this procedure that leaves a service down, the operator (or the agent assisting) MUST:

Post a Discord alert to P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (color red, 🔴 ERROR — bms-1 Ubuntu 22.04 upgrade).
Update the human-action issue (and #1683) with the exact failing step, console output, and current state (rolled back / partially upgraded / containers down).

Prevention

Add bms-1 to the Ansible inventory (ansible/inventory/hosts.yml + ansible/inventory/host_vars/bms-1.yml, done in #1683) so future OS facts and provisioning are tracked as IaC rather than manual.
Watch the Ubuntu LTS support clock: schedule the next OS upgrade before EOL (22.04 EOL is April 2027), not after — EOL-driven upgrades carry unpatched-CVE exposure during the gap.

p24-infra Docs

Explorer

bms1-ubuntu-2204-upgrade