p24-infra Docs

plan-3725-bms1-docker-engine-upgrade

10 min read

Plan: Upgrade bms-1 Docker Engine (20.10.2 → modern release)

Issue: #3725 Type: Infrastructure upgrade plan (PLANNING ONLY — no execution performed) Status: Draft — ready for human go/no-go Last updated: 2026-07-11 Author: Claude Sonnet 5 (p24-infra sys-admin) via read-only research

0. Why This, Why Now

bms-1’s Docker Engine is 20.10.2 (built Dec 2020). Its default seccomp profile predates the clone3 syscall that glibc >= 2.34 (Debian bookworm/trixie) uses internally. Any new container build on bms-1 using a current base image fails during apt-get update’s Post-Invoke cleanup script with:

E: Problem executing scripts APT::Update::Post-Invoke 'rm -f /var/cache/apt/archives/*.deb ...'
E: Sub-process returned an error code

This was hit for real on 2026-07-11 while building mailgun-pipeline-exporter (issue #3688) and worked around narrowly by pinning that one Dockerfile to python:3.11-slim-bullseye (PR #3724). The underlying host-level problem is still there and will resurface on the next image build that uses a modern base.

This document is the upgrade plan requested by issue #3725. No changes have been made to bms-1. Everything below was gathered read-only via SSH.

1. Confirmed Current State (read-only, 2026-07-11)

ItemValue
OSUbuntu 20.04.1 LTS (Focal Fossa), VERSION_CODENAME=focal
Kernel5.4.0-216-generic (supports clone3 — added in Linux 5.3; kernel is not the blocker)
Docker Server version20.10.2 (API 1.41, built 2020-12-28)
containerd1.4.3
runc1.0.0-rc92
Storage driveroverlay2
Cgroup driver / versioncgroupfs / cgroup v1
Logging driverjson-file
Docker package sourceDocker’s official apt repo — deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable (NOT Ubuntu’s own repo, NOT a manual/binary install)
apt policy docker-ce candidate5:28.1.1-1~ubuntu.20.04~focal
Disk (/)440G total, 94G used (23%), 324G free — healthy headroom
Uptime5 days 20h at time of check (last reboot ≈ 2026-07-06)

Key finding: Docker’s focal apt channel is still fully populated, all the way from 19.03.9 through 28.1.1. apt-cache policy lists every point release in between as an installable candidate. This means:

  • No manual .deb download or binary swap is needed — a straight apt-get install docker-ce=<version> docker-ce-cli=<version> containerd.io=<version> works for any target version in that range.
  • Downgrade is equally simple via the same mechanism — this is the basis of the rollback plan in §4.
  • bms-1 does not need an OS upgrade to get a modern Docker Engine. (Ubuntu 20.04 itself is past standard EOL and only on ESM — that is a separate, larger, unrelated modernization effort and is explicitly out of scope here.)

2. Root Cause Confirmation & Target Version Research

The clone3-in-default-seccomp-profile fix landed in Moby’s default seccomp profile update around Docker 20.10.9–20.10.10 (2021-10). Any version at or above that point release resolves the specific apt-get/glibc 2.34 failure described in 3724. Kernel-side, clone3 requires Linux ≥ 5.3 — bms-1’s 5.4.0-216-generic already satisfies that; the kernel was never the blocker, only the Docker daemon’s seccomp default.

That gives two structurally different options, both valid per the issue’s own framing (“even a moderate point-upgrade may suffice”):

Option A — Minimal-risk point release: Docker CE 20.10.24

  • Latest published release on the 20.10.x branch (still on the focal channel).
  • Same major branch as what’s running now — smallest behavioral delta, smallest risk of surprising an old container built years ago.
  • Fixes the clone3 issue (well past 20.10.9/20.10.10).
  • Downside: Docker 20.10.x itself reached end of upstream support in 2023. No further security patches will land on this branch. This only defers the underlying “old Engine” problem, it doesn’t retire it.

Option B — Modern LTS-equivalent: Docker CE 24.0.9

  • Latest point release on the 24.0.x line — the most widely deployed, longest bugfix tail of the currently-supported branches, still compatible with cgroupfs + cgroup v1 (bms-1’s current setup) and overlay2.
  • Actually resolves the “any future build breaks” risk class, not just this one instance of it.
  • Downside: Bigger jump (20.10 → 24.0), more surface area for surprises on a host with several multi-year-old containers.

Option C — Latest available: Docker CE 28.1.1

  • What apt policy currently offers as candidate.
  • Gets everything current, but is the largest jump from a 5-year-old daemon, on a host that has never been reliably restart-tested recently (several containers here haven’t been recreated in 4+ years — see §3). Higher uncertainty for the amount of risk-reduction gained over Option B.

Recommendation (for human decision, not executed by this task): Option B (24.0.9) — it durably fixes the underlying problem (not just this one symptom) while stopping short of the largest, riskiest jump. If the team wants to minimize this window’s blast radius as much as possible and revisit again later, Option A (20.10.24) is an acceptable interim step — but note it will need a second upgrade later since the 20.10.x branch has no further security support.

This choice (A vs B vs C) is exactly the kind of go/no-go call this issue asks to be flagged to a human rather than decided by the agent.

3. What a Docker Daemon Restart Affects on bms-1

docker ps -a (2026-07-11) shows 34 containers. A Docker Engine package upgrade requires systemctl restart docker (or equivalent), which briefly stops every running container on the host — this is not a rolling/zero-downtime operation for the daemon itself. Containers with a restart policy of unless-stopped or always come back automatically once the daemon is back up; containers with no restart policy do not.

Production containers (Pinbox24) — restart policy confirmed via docker inspect

ContainerRestart policyAuto-recovers?
v42-prod, v42-notify-prod, pdf-gen-v42-prod, s3-v42-prod, s3-v2-v42-prod, redis-v42, git-deploy-v42-prod, wkhtml-v42-produnless-stoppedYes
v32-prod, v32-prod-reso, v32-prod-socket, s3-v32-prod, redis-v32unless-stoppedYes
v41-prod, v31-prodalwaysYes
mailgun-v42-prod(none)No — needs manual docker start
cron-v32-prod, cron-v32-prod-reso, cron-v32-prod-socket(none)No — needs manual docker start
s3-v32-prod-reso, s3-v32-prod-socket, s3-v32-prod-renamed(none)No — needs manual docker start

Monitoring / infra containers

ContainerRestart policyAuto-recovers?
mezmo-agent_mezmo-agent_1, mezmo-agent_mailgun-pipeline-exporter_1 (PR #3724 exporter)unless-stoppedYes
cadvisor, portainer-pinbox24alwaysYes
nginx-proxy, nginx-proxy-letsencrypt(none — explicitly no)No — needs manual restart, but this is the public HTTP/HTTPS entrypoint for the whole host

Staging / misc (lower priority but still present)

s3-v42-stage, s3-v32-stage, v31-stage-front, v41-stage-front, s3-v42-stage-old-admin-cred, wkhtml-v42-stage — all unless-stopped, will self-recover.

Also observed: distracted_lamarr and inspiring_solomon — two Exited (100) containers running apt-get..., 6 hours old. These are almost certainly leftover repro containers from the #3688 investigation itself (exit code 100 is apt’s own error code, consistent with the bug). Not part of the live stack; worth docker rm as housekeeping but out of scope for this plan.

Action item for the maintenance-window runbook: before restarting Docker, re-run docker inspect --format '{{.Name}}: {{.HostConfig.RestartPolicy.Name}}' $(docker ps -aq) to get the current, authoritative “will not auto-recover” list (the set above was captured 2026-07-11 and containers/policies may have changed by the time the window is scheduled) — then explicitly docker start <name> each of them immediately after the daemon comes back, especially nginx-proxy, since until it restarts, no service on the host is reachable from outside even if every backend container is healthy.

PM2

pm2 list on bms-1 shows zero managed processes — all Pinbox24 workloads here run inside Docker (pm2-runtime is used inside some containers, e.g. v42-prod, v32-prod, but there is no host-level PM2 process to account for separately).

4. Upgrade Plan (commands — NOT executed, for the scheduled maintenance window)

4.1 Pre-flight (safe to run any time, read-only, no service impact)

# PLAYBOOK: plan-3725-bms1-docker-engine-upgrade.md
mkdir -p /root/docker-upgrade-2725
docker ps -a > /root/docker-upgrade-2725/pre-ps-$(date +%F).txt
docker inspect --format '{{.Name}}: restart={{.HostConfig.RestartPolicy.Name}}' $(docker ps -aq) \
  > /root/docker-upgrade-2725/pre-restart-policies-$(date +%F).txt
dpkg -l | grep -i docker > /root/docker-upgrade-2725/pre-versions-$(date +%F).txt
apt-cache policy docker-ce docker-ce-cli containerd.io \
  > /root/docker-upgrade-2725/pre-apt-policy-$(date +%F).txt

Optional extra safety net (human decision — costs may apply): take an OVH Kimsufi snapshot/backup of bms-1 immediately before the window if the provider’s plan supports it.

4.2 Upgrade (example commands for recommended target 24.0.9 — Option B)

# PLAYBOOK: plan-3725-bms1-docker-engine-upgrade.md
apt-get update
 
# Verify exact containerd.io version paired with 24.0.9 at execution time —
# `apt-cache policy containerd.io` candidate list shifts; pick the version
# that was current when 24.0.9 was published (do not blindly take "latest").
apt-get install -y --allow-downgrades \
  docker-ce=5:24.0.9-1~ubuntu.20.04~focal \
  docker-ce-cli=5:24.0.9-1~ubuntu.20.04~focal \
  containerd.io=<matching-version>
 
systemctl restart docker

For Option A (20.10.24, smaller step) substitute: docker-ce=5:20.10.24~3-0~ubuntu-focal docker-ce-cli=5:20.10.24~3-0~ubuntu-focal.

4.3 Post-upgrade verification

docker version --format '{{.Server.Version}}'
docker ps -a --format 'table {{.Names}}\t{{.Status}}'
 
# Explicitly bring up everything without an auto-restart policy
# (re-derive this list from the pre-flight file, do not trust this doc's
# 2026-07-11 snapshot if time has passed):
docker start nginx-proxy nginx-proxy-letsencrypt mailgun-v42-prod \
  cron-v32-prod cron-v32-prod-reso cron-v32-prod-socket \
  s3-v32-prod-reso s3-v32-prod-socket s3-v32-prod-renamed
 
# Smoke test: confirm the actual bug (#3688) is fixed
docker run --rm python:3.13-slim apt-get update
# Expected: completes without "Sub-process returned an error code" from
# APT::Update::Post-Invoke — this is the actual acceptance test for this issue.
 
# Confirm public HTTP/HTTPS still resolves through nginx-proxy, and spot-check
# a couple of Pinbox24 endpoints (v42-prod, v32-prod) respond.

4.4 Rollback plan

Because Docker’s focal apt channel still serves every historical version back to 19.03.9 (confirmed in §1), rollback is a same-mechanism downgrade — no need to source old .deb files or rebuild anything:

# PLAYBOOK: plan-3725-bms1-docker-engine-upgrade.md
apt-get install -y --allow-downgrades \
  docker-ce=5:20.10.2~3-0~ubuntu-focal \
  docker-ce-cli=5:20.10.2~3-0~ubuntu-focal \
  containerd.io=1.4.3-1
systemctl restart docker
# Repeat the manual `docker start` list from §4.3 — the daemon restart happens
# again on the way down, so containers without an auto-restart policy need
# the same explicit recovery step.

Rollback restores exactly the pre-upgrade state (20.10.2 / containerd 1.4.3-1, confirmed as the exact installed versions before any change).

5. Expected Downtime

  • Docker daemon restart itself: ~5–15 seconds.
  • Docker restarting all unless-stopped/always containers (25 of the 34): typically 1–3 minutes for the container-level restarts to all report “Up”, plus normal application boot time (node/pm2 warm-up, Redis ready, nginx-proxy re-discovering backends via its Docker socket watch) — call it 2–5 minutes until the stack is fully warm.
  • Containers without an auto-restart policy (nginx-proxy chief among them) are down until the explicit docker start step in §4.3 runs — this should be scripted/ready to fire immediately after systemctl restart docker returns, not discovered ad hoc, or the effective outage extends by however long it takes a human to notice and react.
  • Estimated total service interruption: 5–10 minutes, assuming the post-upgrade docker start list is run immediately and correctly.
  • Recommended maintenance-window budget: 30–60 minutes total (execution + verification + buffer for troubleshooting or triggering the rollback in §4.4), even though actual customer-facing downtime should be under 10 minutes if everything goes to plan.

Per the existing nightly-restart convention (20:00–06:00 UTC) and given bms-1 is live Pinbox24 production:

  • Start of window (20:00–21:00 UTC) rather than later in the night — this leaves the rest of the overnight window available to troubleshoot or execute the rollback plan before business hours resume, instead of running the upgrade at 05:00 UTC with no buffer left.
  • Avoid Friday/weekend-adjacent nights — if something goes wrong, weekday follow-up coverage is better than a weekend page.
  • Coordinate with anyone actively working on Pinbox24 W3/W4 deploys that night (the host also carries git-deploy-v42-prod and active stage containers — a mid-deploy Docker restart is avoidable friction).

This is exactly the go/no-go and scheduling decision flagged back to a human (per the issue’s explicit request) — this plan does not pick a date, does not pick Option A vs B vs C, and has not touched bms-1 beyond read-only docker/apt/ssh checks used to gather the facts above.

7. Explicitly Out of Scope / Deferred

  • Executing any part of §4 — this issue is planning-only.
  • Restarting the Docker daemon, or any container, on bms-1.
  • Upgrading bms-1’s underlying Ubuntu 20.04 install (separate, larger effort; 20.10.x/24.x/28.x are all installable on the current OS without this).
  • Reverting PR #3724’s python:3.11-slim-bullseye pin for mailgun-pipeline-exporter — no need to touch it once the Engine is fixed; it continues to work fine either way.
  • Cleaning up the two stale Exited (100) repro containers (distracted_lamarr, inspiring_solomon) — harmless, noted for housekeeping only.

8. Human Decisions Required Before Execution

  1. Target version: Option A (20.10.24, minimal risk, defers the underlying-EOL problem) vs Option B (24.0.9, recommended) vs Option C (28.1.1, largest jump).
  2. Maintenance window date/time (within the 20:00–21:00 UTC start recommendation).
  3. Go/no-go on proceeding at all vs. continuing to pin individual Dockerfiles to bullseye-era bases as a stopgap (current state — works, but leaves the underlying problem for the next engineer to rediscover).
  4. Optional: whether to take an OVH snapshot before the window.

Graph View