p24-infra Docs

mongodb-backup-bom-recovery

4 min read

Playbook: MongoDB backup script dies silently — UTF-8 BOM in the deployed shell script

Status: ACTIVE (first written 2026-10-10, #7717 / partially implements #7609) Host: bms-3 (51.68.155.224, rs0 SECONDARY) — canonical SSH user root (per dev_r_server_capacity.ssh_user, the single source of truth; the ubuntu entry in some older tables/worker-prompt copies is stale). Script: /root/mongodb-backup.sh (repo source of truth: scripts/mongodb-backup.sh).

Symptom

s3://p24-infra/mongodb/full/ stays empty and /var/log/mongodb-backup.log repeats exactly:

/root/mongodb-backup.sh: 1: <BOM>#!/usr/bin/env: not found
/root/mongodb-backup.sh: 17: set: Illegal option -o pipefail

The cron fires daily but the script dies on line 1, so its own Discord/GH error-notification never runs → silent failure (117 days undetected in #7717). RPO silently frozen at the last good backup.

Root cause

The deployed file begins with a UTF-8 BOM (EF BB BF) before the shebang. The kernel does not honour a shebang that isn’t at byte 0, so the script runs under /bin/sh (dash), which fails on set -o pipefail (and prints the BOM as a mojibake prefix). Cause of the BOM: the file was edited/ saved on a Windows editor. Confirm with head -c 16 /root/mongodb-backup.sh | xxd → leading efbbbf.

Fix (live, surgical — do ONLY this on the live file)

F=/root/mongodb-backup.sh
cp -p "$F" "${F}.bak-$(date -u +%F)"        # back up first
sed -i '1s/^\xEF\xBB\xBF//' "$F"            # strip ONLY the leading BOM on line 1
# verify:
head -c 3 "$F" | xxd                         # expect 23 21 2f  (#!/)
head -1 "$F"                                 # expect #!/usr/bin/env bash
bash -n "$F" && echo ok                       # syntax clean
cmp <(tail -c +4 "${F}.bak-"*) "$F"           # expect identical — nothing but 3 BOM bytes changed

Before RE-RUNNING the backup — pre-flight gates (bms-3)

  • bms-3 must be SECONDARY — mongosh --quiet --eval 'db.hello()' (unauthenticated handshake): isWritablePrimary=false. If it is PRIMARY, abort.
  • Other rs0 members health=1 and secondary replication lag < 10 s — rs.status() (needs auth, see credential note below).
  • /tmp filesystem ≥ 80 GB free; no mongodump already running; load1 < 6.
  • Run low-priority so it can’t starve the replica: nice -n 19 ionice -c3 /root/mongodb-backup.sh full (the full arg is ignored by the script). Monitor every ~2 min: lag < 60 s, load < 12, /tmp > 40 GB; on breach SIGTERM the dump, clean /tmp/mongodb-full-* + /tmp/full-*.tar.gz, report, do not blind-retry.

⚠️ Gotcha: /root/.backup-env can be STALE (separate failure from the BOM)

/root/.backup-env is a server-local plaintext file (NOT managed by secrets-sync). The script authenticates mongodump with its MONGODB_ADMIN_USER/MONGODB_ADMIN_PASSWORD and uploads with its WASABI_ACCESS_KEY/WASABI_SECRET_KEY. When the rs0 admin password (or a Wasabi key) is rotated, .backup-env is not updated automatically → mongodump fails Authentication failed even after the BOM is fixed. This is exactly what blocked #7717 (.backup-env admin pass 18 chars vs current SOPS MONGODB_RS0_ADMIN_PASSWORD 32 chars).

Diagnose WITHOUT printing values (lengths/fingerprints only):

# length in .backup-env (run on bms-3):
ssh root@51.68.155.224 '( set -a; . /root/.backup-env; set +a; echo "pass_len=${#MONGODB_ADMIN_PASSWORD}" )'
# current length in SOPS (on a host with the age key):
sops -d --input-type dotenv --output-type dotenv secrets/mongodb-bms.env.sops \
  | grep '^MONGODB_RS0_ADMIN_PASSWORD=' | cut -d= -f2- | wc -c

Refreshing .backup-env is a credential-handling action — route it through secret-manager / explicit human approval (it is NOT a SOPS write and NOT a new rotation; the current value already lives in secrets/mongodb-bms.env.sops → it is a one-line sync of an existing secret). An infra-task worker whose issue says “do not touch /root/.backup-env” must escalate (human-action), not override.

Permanent prevention (tracked on #7609)

  • Deploy the backup script from the repo copy with a BOM check at deploy time (or Ansible-managed), not hand-copied. scripts/tests/test_no_bom_in_shell_scripts.py already guards the repo copy — the drift is purely deploy-side.
  • Reconcile cron vs repo: live cron is daily 01:30 with an ignored full arg; the repo header says weekly Sunday 01:00. Make repo + cron + Ansible + retention agree.
  • Add a Prometheus alert on the age of the newest s3://p24-infra/mongodb/full/ object so a silent failure can’t exceed ~25 h, and fix why backup-exporter missed 117 days.
  • Add P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL to .backup-env (currently absent → the script’s own notify_discord_error is a no-op even when the script runs).
  • docs/playbooks/sops-windows-crlf.md — BOM/CRLF corruption in SOPS files (same byte-level class).
  • docs/playbooks/mongodb-restore-drill-troubleshooting.md, docs/playbooks/mongodb-rs0-full-restore.md.
  • docs/playbooks/server-ssh-connectivity.md — bms-3 canonical user = root (dev_r_server_capacity).

Graph View