p24-infra Docs

mongodb-restore-drill-troubleshooting

4 min read

Playbook: MongoDB Restore Drill — Troubleshooting

Reference for diagnosing failures in scripts/mongodb-restore-drill-cron.sh and scripts/mongodb-restore-drill.sh.

First successful drill run: 2026-07-01 (w3_db + w4_db, backup 2026-06-15). All issues below were discovered and fixed before that run.

Issue 1 — sops exec-env fails on *.env.sops extension

Symptom: Drill exits with:

Error unmarshalling input json: invalid character 'C' looking for beginning of value

Root cause: sops exec-env detects file format from extension only. The extension *.env.sops is NOT recognized as dotenv (only *.env is recognized). SOPS falls back to JSON parsing; the file starts with CF_API_TOKEN=C... → C is invalid JSON.

The --input-type flag is not available on the exec-env subcommand.

Fix (in mongodb-restore-drill-cron.sh): Replace sops exec-env with explicit decrypt:

_tmpenv=$(mktemp)
chmod 600 "${_tmpenv}"
trap 'rm -f "${_tmpenv}"' EXIT
sops --decrypt --input-type dotenv --output-type dotenv "${MONITORING_SOPS}" >> "${_tmpenv}"
sops --decrypt --input-type dotenv --output-type dotenv "${BMS_SERVERS_SOPS}" >> "${_tmpenv}"
# ... then inject into env (see Issue 2 and 3)

Issue 2 — source fails on values with special characters

Symptom: After decrypting SOPS, env injection silently fails. Some variables not exported. The drill script exits with “SKIPPED (unconfigured)” even though SOPS decrypt succeeded.

Root cause: set -a; source "$tmpenv"; set +a calls source which parses each line as a bash statement. Values like SUPABASE_GRAFANA_PASSWORD=.nSw^Jkn)Y?-(npgCwSW... contain ) which bash interprets as syntax (end of subshell).

Fix: Use a safe line-by-line while loop:

while IFS= read -r _line; do
    [[ -z "${_line}" || "${_line}" == \#* ]] && continue
    _k="${_line%%=*}"
    _v="${_line#*=}"
    [[ "${_k}" =~ ^[a-zA-Z_][a-zA-Z_0-9]*$ ]] || continue  # Issue 3
    export "${_k}"="${_v}"
done < "${_tmpenv}"

Issue 3 — Invalid bash identifiers in dotenv file

Symptom:

export: 'p24-infra-1h-check-claude-discord=...': not a valid identifier

Root cause: Some keys in monitoring.env.sops have hyphens (e.g. p24-infra-1h-check-claude-discord) which are valid in dotenv but not as bash variable names.

Fix: Guard with regex before export (included in Issue 2 fix):

[[ "${_k}" =~ ^[a-zA-Z_][a-zA-Z_0-9]*$ ]] || continue

Issue 4 — MONGODB_ADMIN_PASSWORD not set for --compare-prod

Symptom: Drill logs “WARNING: MONGODB_ADMIN_PASSWORD not set — skipping production comparison”. The --compare-prod step is silently skipped.

Root cause: bms-servers.env.sops uses key mongodb_rs0_admin_password (lowercase) but the drill script expects MONGODB_ADMIN_PASSWORD. After the line-by-line injection, only mongodb_rs0_admin_password is exported.

Fix (in mongodb-restore-drill-cron.sh): Add mapping after env injection:

export MONGODB_ADMIN_PASSWORD="${mongodb_rs0_admin_password:-}"
export MONGODB_ADMIN_USER="${mongodb_rs0_admin_user:-admin}"

Issue 5 — _aws_s3 helper: redundant s3 prefix

Symptom:

aws: [ERROR]: An error occurred (ParamValidation): argument subcommand: Found invalid choice 's3'

Root cause: _aws_s3() already calls aws s3 .... All call sites were passing s3 as the first argument (e.g. _aws_s3 s3 ls), resulting in aws s3 s3 ls which fails because s3 is not a valid subcommand of aws s3.

Fix: Change all call sites to pass only the subcommand:

_aws_s3 ls "s3://bucket/prefix/"   # not: _aws_s3 s3 ls ...
_aws_s3 cp src dst                 # not: _aws_s3 s3 cp ...
_aws_s3 sync src dst               # not: _aws_s3 s3 sync ...

Issue 6 — Wrong Wasabi backup prefix

Symptom: STEP 1: Locating latest backup on Wasabi finds nothing, drill exits immediately with “No backup found”.

Root cause: BACKUP_PREFIX="mongodb/full" in mongodb-restore-drill.sh but actual backups are at s3://p24-infra/mongodb-backups/.

To list the actual structure:

aws s3 ls s3://p24-infra/mongodb-backups/ \
  --endpoint-url https://s3.eu-central-2.wasabisys.com --region eu-central-2
# Expected: w3_db-YYYY-MM-DD.mongodump.gz, w4_db-YYYY-MM-DD.mongodump.gz

Fix: BACKUP_PREFIX="mongodb-backups" and update date extraction to use grep -Eo (extracts date from filenames, not directory prefixes).

Issue 7 — Wrong restore format (directory vs archive)

Symptom: After the drill completes restore, collection counts are all 0 or the restore step fails silently.

Root cause: mongodb-restore-drill.sh was written for mongodump --out DIR --gzip format (per-collection .bson.gz files inside a directory). But actual backups are created with mongodump --archive=FILE --gzip format (single .mongodump.gz per database).

Fix: Restore with --archive flag:

# Inside Docker container:
mongorestore --archive="/tmp/${DB}.mongodump.gz" --gzip --drop --preserveUUID
# NOT: mongorestore --db "$DB" --dir "/dump/${DB}" --gzip ...

Issue 8 — aws CLI not installed on bms-4

Symptom: bash: aws: command not found during STEP 1 or 2.

Fix:

curl -s https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip -o /tmp/awscliv2.zip
cd /tmp && unzip -q awscliv2.zip
./aws/install --install-dir /usr/local/aws-cli --bin-dir /usr/local/bin
aws --version  # verify

Running the drill manually

# On bms-4 as root:
sudo -u claude-runner /opt/p24-infra/scripts/mongodb-restore-drill-cron.sh
# Logs: /tmp/mongo-drill-$PID/drill.log
# Duration: ~90-120 min (w3_db 7.3GB + w4_db 2.9GB)
  • scripts/mongodb-restore-drill.sh — main drill logic
  • scripts/mongodb-restore-drill-cron.sh — wrapper for SOPS injection + alerting
  • scripts/mongodb-backup.sh — backup script on bms-3 (Wasabi, weekly → daily after #2367)
  • docs/playbooks/mongodb-rs0-full-restore.md — full DR playbook (Scenario B)
  • Issue #2367 — bms-3 SSH + daily cron fix
  • Issue #2148 — backup gap tracking
  • PR #2371 — all drill script fixes

Graph View

Backlinks