05 — Backups & Disaster Recovery

Backup coverage, retention policies, and disaster-recovery procedures for all stateful systems: MongoDB rs0, Supabase PostgreSQL, Traccar MySQL, and Wasabi S3.

MongoDB restore drill — now automated (#2148). A quarterly drill runs on bms-4 (p24-mongodb-restore-drill.timer, 1st of Jan/Apr/Jul/Oct 03:00 UTC), restoring the latest Wasabi dump into a throwaway Docker container (never touches the rs0 PRIMARY), counting documents, and pushing a freshness metric. Prometheus alerts MongoDBRestoreDrillStale (>95 days), MongoDBRestoreDrillFailed, and MongoDBRestoreDrillMetricMissing fire if a drill is overdue, fails, or has never run. See MongoDB Restore Drill below.

Key Documents

Document	Description
01-backups.md	Backup improvement plan — coverage gaps, recommendations

MongoDB Restore Drill

The drill proves the MongoDB backup is actually restorable — not just present in Wasabi.

Script: scripts/mongodb-restore-drill.sh Cron wrapper: scripts/mongodb-restore-drill-cron.sh (injects Wasabi + MongoDB-admin secrets via sops exec-env, then runs the drill with --compare-prod).

What it does (safe by design)

Downloads the latest mongodb/full/ dump from Wasabi (p24-infra bucket).
Spins up a throwaway MongoDB 7.0 Docker container on bms-4 — it does not connect to rs0 and never writes to bms-2/bms-3. Production is only ever read (optional read-only estimatedDocumentCount via --compare-prod).
Restores w3_db, w4_db, admin; counts documents; builds a drill report.
Logs the outcome to dev_r_ops_events, uploads the log to s3://p24-infra/mongodb/drill-logs/, sends a Discord notification, and on FAIL opens a human-action GitHub issue.
Pushes Prometheus metrics to the bms-4 pushgateway: mongodb_restore_drill_last_success_timestamp, mongodb_restore_drill_success, mongodb_restore_drill_duration_seconds.

Why bms-4, not bms-2: bms-2 is the rs0 PRIMARY and must not carry drill load. Restoring into an isolated container on bms-4 satisfies the verification intent without any risk to production — matching the DR-drill rule “Never run a DR drill against the production MongoDB PRIMARY.”

Automation (quarterly)

Unit	Schedule	Source
`p24-mongodb-restore-drill.timer`	`*-1,4,7,10-01 03:00 UTC` (quarterly)	`deploy/p24-mongodb-restore-drill.timer`
`p24-mongodb-restore-drill.service`	oneshot wrapper run	`deploy/p24-mongodb-restore-drill.service`

Installed on bms-4 by the bms4-workers Ansible role (tag mongodb-restore-drill), same single-source-of-truth pattern as the queue-worker units.

# Check / trigger on bms-4
systemctl list-timers p24-mongodb-restore-drill.timer
sudo -u claude-runner /opt/p24-infra/scripts/mongodb-restore-drill-cron.sh   # manual run
journalctl -u p24-mongodb-restore-drill.service --no-pager -n 80

Monitoring

Recording rule mongodb_last_restore_drill_age_days = (time() - mongodb_restore_drill_last_success_timestamp) / 86400 plus the three alerts in monitoring/prometheus/rules/backups.yml (mongodb-restore-drill group). MongoDBRestoreDrillStale pages once the age exceeds 95 days (quarterly cadence + margin); MongoDBRestoreDrillMetricMissing covers the never-run state.

Drill Log

Each scheduled/manual run also appends a row to dev_r_ops_events (actor mongodb-restore-drill). Record verified runs here for the at-a-glance audit trail:

Date (UTC)	Backup tested	Result	Duration	Run by	Notes
pending	—	—	—	quarterly timer / manual	First run delegated to the first scheduled or manual execution on bms-4 (multi-GB restore needs root + Docker + age key). The staleness alert tracks until it runs.

Backup Coverage by System

System	Backup method	Location	Frequency	Last verified
MongoDB rs0	mongodump → Wasabi S3	`p24-infra` bucket	nightly	Quarterly drill automated (#2148); see Drill Log — first verified run pending
Supabase PostgreSQL	Supabase managed backups (daily) + pg_dump	Supabase + Wasabi	daily	Unknown
Traccar MySQL	mysqldump → local + Wasabi	`p24-infra` bucket	nightly	Unknown
n8n PostgreSQL (bms-4)	pg_dump via cron	Wasabi	nightly	Unknown
Grafana config	Git (monitoring/)	GitHub	on commit	Current

Wasabi S3 Bucket Layout

Bucket: p24-infra — region eu-central-2 — endpoint s3.eu-central-2.wasabisys.com

Folder	Contents
`thanos/`	Prometheus TSDB blocks (2h chunks, uploaded by Thanos sidecar)
`pdfs/`	Generated fleet inspection PDFs
Bucket root	Backup status JSON files polled by backup-exporter

Backup Exporter

The backup-exporter container on vps-i1 polls Wasabi for backup status JSON files and exposes metrics on :9220. If the JSON is stale, Prometheus fires a BackupStale alert → Alertmanager sends email.

See monitoring-exporters-operations.md for configuration.

Disaster Recovery

No dedicated DR runbook exists yet — creating one is tracked in README.

Interim priorities for a DR scenario:

MongoDB rs0 — bms-2 is PRIMARY, bms-3 is SECONDARY, bms-4 is ARBITER. If PRIMARY fails, rs0 elects automatically.
Supabase — managed service; failover is Supabase’s responsibility.
vps-i1 (monitoring) — monitoring is non-critical for platform availability; restore from git + .env.bak.
bms-1 (Pinbox24 production) — no hot standby; restore from Docker image tags on ECR + last mongodump.

Cross-references

README — backup-exporter alert rules
README — per-server disk and storage status
README — Wasabi S3 service details

p24-infra Docs

Explorer

README