p24-infra Docs

wasabi-replication-repair

5 min read

Playbook — Wasabi 3-bucket replication repair (Pinbox24 W4)

Script: scripts/wasabi-replication-repair.py Issue: #2709 Owner role: infra-task / sys-admin on bms-1 (needs the s3-v2 IAM key + w4_app Mongo access)

What it does

Pinbox24 W4 uploads every file to three Wasabi buckets in three datacenters for geo-redundancy. The upload path (uploadMultiFiles() / promiseAny() in the s3-v2-v42-prod microservice) answers the client as soon as the first bucket succeeds and never retries the others, so ~87k files ended up with fewer than 3 copies (21% on a single bucket, no redundancy).

This script scans w4_db.files for docs whose storageInfo array has fewer than 3 bucket copies and, for each missing bucket, copies the object in from a bucket that already has it — restoring full 3-datacenter redundancy.

Bucket topology

Source of truth: infra-src/pinbox24/w4/s3-v2-persistent-patches/storage.config.js (STORAGE_PRIORITY).

tagbucketregion / datacenter
storage_1test-replicated-to-us-bucketeu-central-1 (Frankfurt)
storage_2p24-was-us-east-1us-east-1 (Ashburn, VA)
storage_3test-us-bucket-for-replication-testingus-east-2 (Columbus, OH)

All three are reachable by the single s3v2-prod-bms1 IAM key.

Scope guardrails (#2709 architect decision 2026-08-04)

  • Only touches docs where storageInfo exists and has < 3 entries.
  • Docs with no storageInfo field at all (the ~20k legacy + ~10 live-orphan class) are a distinct failure mode tracked separately on #2709 — not touched by this script.
  • Bucket rename (point 4), the uploadMultiFiles retry fix (point 3), and the all-3-failed no-rollback bug are out of scope — this tool only backfills missing copies.

Cross-region copy is done as get → put (not server-side copy_object): the three buckets are in three separate Wasabi endpoints, so a server-side CopySource copy is not reliable across them. Every object is HEAD-checked at source and destination first, so the job is idempotent — already-present destinations are skipped and re-running never duplicates work.

Required environment (values referenced by NAME only — never printed)

PurposeEnv var(s)Source
Wasabi IAM key (all 3 buckets)pinbox24PublicOfficeAccessKeyId / pinbox24PublicOfficeSecretAccessKey (or WASABI_REPL_ACCESS_KEY / WASABI_REPL_SECRET_KEY)s3-v2-environment.env on bms-1 / secrets/pinbox24-w4.env.sops
MongoDB (read-only)W4_MONGODB_URI (or NEW_MONGODB_URI)w4_app creds — secrets/bms-servers.env.sops → mongodb_w4_app_password
Fatal-error alertP24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URLmonitoring env
Progress webhook (optional)DISCORD_WEBHOOK_URLfalls back to the errors webhook
GH issue on fatal error (optional)GH_TOKENworker env

Do not hardcode any of these — export them from SOPS at run time using the safe extraction pattern (see repo CLAUDE.md §Secrets). Reading these secrets and running against production is a secret-manager + sys-admin/infra-task operation, not a dev-coder one.

Running it

Dry-run is the default — the script performs NO writes without --execute.

# 1. Smoke test: scan a handful of docs, report only.
python3 scripts/wasabi-replication-repair.py --limit 200
 
# 2. Full dry-run: report the complete gap picture (counts of would_copy / already_present / …).
python3 scripts/wasabi-replication-repair.py
 
# 3. Execute the repair (writes copies). Start narrow, then widen.
python3 scripts/wasabi-replication-repair.py --execute --limit 500
python3 scripts/wasabi-replication-repair.py --execute

Flags: --batch-size N (progress checkpoint cadence, default 100), --limit N (cap docs, 0 = unlimited), --office-id <id> (restrict to one office).

Output

Per-batch progress lines go to stdout and Discord; a final JSON summary reports scanned / already_full / would_copy / copied / already_present / source_missing / no_key / errors / bytes. Exit code 0 = clean, 2 = missing config, 3 = one or more per-object errors.

Expected cost

Filling the ~113k missing copies adds ~18.5 GB → roughly +$0.13/month Wasabi storage.

Metadata backfill — wasabi-storageinfo-backfill.py (#5649)

wasabi-replication-repair.py copies objects between buckets at the S3 layer only — it has no update_one/$push and never writes back to MongoDB. So after the live repair run (2026-08-06: 77,289 files copied, 0 errors) the w4_db.files docs still showed storageInfo.length < 3 (56,968 of them) even though the files are now physically 3/3-present. Any app logic that trusts storageInfo to decide “does this file need repair” — including a naive re-run of the repair tool’s own dry-run — keeps reporting these ~77k files as under-replicated even though they are fixed.

scripts/wasabi-storageinfo-backfill.py closes that gap. For every doc still showing storageInfo < 3 it HEAD-checks all three canonical buckets directly (it imports the repair tool’s topology + helpers, so the bucket list/logic can never drift) and, for any bucket physically present but absent from the doc’s storageInfo array, $push-es a matching entry so the metadata reflects reality. Pushed entries carry bucketName + key + region/endpoint/tag and a "backfilled": true provenance marker.

  • Idempotent: only buckets confirmed present-via-HEAD and not already recorded are pushed — a second full pass is a no-op.
  • source_missing safety (#2709 phantom-source, ~3,958 docs): a doc whose object is on none of the three buckets has nothing to push (every HEAD 404s) and is skipped untouched. The script never fabricates a storageInfo entry for a bucket that does not physically hold the object.
  • Same creds as the repair tool — Wasabi pinbox24PublicOfficeAccessKeyId/...SecretAccessKey and W4_MONGODB_URI. Execution additionally needs w4_app write access (the repair tool only read Mongo). Owner role for the --execute run: infra-task / secret-manager / sys-admin, not dev-coder.
# Dry-run is the default — reports scanned / already_full / no_gap / source_missing /
# would_backfill / entries_added / errors and writes nothing.
python3 scripts/wasabi-storageinfo-backfill.py --limit 200      # smoke test
python3 scripts/wasabi-storageinfo-backfill.py                  # full dry-run
python3 scripts/wasabi-storageinfo-backfill.py --execute --limit 500   # narrow real run
python3 scripts/wasabi-storageinfo-backfill.py --execute        # full backfill

Same --batch-size / --limit / --office-id flags as the repair tool. Exit code 0 = clean, 2 = missing config, 3 = one or more per-doc errors.

Follow-ups still open on #2709 (not this script)

  • Rename the test-* buckets (operational hazard — two independent teams hit the “test-* buckets are actually production” trap).
  • Fix uploadMultiFiles() to retry partial failures instead of answering on first success.
  • Fix the all-3-failed no-rollback bug that orphans FileModel docs with no storageInfo.

Graph View