p24-infra Docs

wasabi-replication-repair

8 min read

Playbook — Wasabi 3-bucket replication repair (Pinbox24 W4)

Script: scripts/wasabi-replication-repair.py Issue: #2709 Owner role: infra-task / sys-admin on bms-1 (needs the s3-v2 IAM key + w4_app Mongo access)

What it does

Pinbox24 W4 uploads every file to three Wasabi buckets in three datacenters for geo-redundancy. The upload path (uploadMultiFiles() / promiseAny() in the s3-v2-v42-prod microservice) answers the client as soon as the first bucket succeeds and never retries the others, so ~87k files ended up with fewer than 3 copies (21% on a single bucket, no redundancy).

This script scans w4_db.files for docs whose storageInfo array has fewer than 3 bucket copies and, for each missing bucket, copies the object in from a bucket that already has it — restoring full 3-datacenter redundancy.

Bucket topology

Source of truth: infra-src/pinbox24/w4/s3-v2-persistent-patches/storage.config.js (STORAGE_PRIORITY).

tagbucketregion / datacenter
storage_1test-replicated-to-us-bucketeu-central-1 (Frankfurt)
storage_2p24-was-us-east-1us-east-1 (Ashburn, VA)
storage_3test-us-bucket-for-replication-testingus-east-2 (Columbus, OH)

All three are reachable by the single s3v2-prod-bms1 IAM key.

Scope guardrails (#2709 architect decision 2026-08-04)

  • Only touches docs where storageInfo exists and has < 3 entries.
  • Docs with no storageInfo field at all (the ~20k legacy + ~10 live-orphan class) are a distinct failure mode tracked separately on #2709 — not touched by this script.
  • Bucket rename (point 4), the uploadMultiFiles retry fix (point 3), and the all-3-failed no-rollback bug are out of scope — this tool only backfills missing copies.

Cross-region copy is done as get → put (not server-side copy_object): the three buckets are in three separate Wasabi endpoints, so a server-side CopySource copy is not reliable across them. Every object is HEAD-checked at source and destination first, so the job is idempotent — already-present destinations are skipped and re-running never duplicates work.

Required environment (values referenced by NAME only — never printed)

PurposeEnv var(s)Source
Wasabi IAM key (all 3 buckets)pinbox24PublicOfficeAccessKeyId / pinbox24PublicOfficeSecretAccessKey (or WASABI_REPL_ACCESS_KEY / WASABI_REPL_SECRET_KEY)s3-v2-environment.env on bms-1 / secrets/pinbox24-w4.env.sops
MongoDB (read-only)W4_MONGODB_URI (or NEW_MONGODB_URI)w4_app creds — secrets/bms-servers.env.sops → mongodb_w4_app_password
Fatal-error alertP24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URLmonitoring env
Progress webhook (optional)DISCORD_WEBHOOK_URLfalls back to the errors webhook
GH issue on fatal error (optional)GH_TOKENworker env

Do not hardcode any of these — export them from SOPS at run time using the safe extraction pattern (see repo CLAUDE.md §Secrets). Reading these secrets and running against production is a secret-manager + sys-admin/infra-task operation, not a dev-coder one.

Running it

Dry-run is the default — the script performs NO writes without --execute.

# 1. Smoke test: scan a handful of docs, report only.
python3 scripts/wasabi-replication-repair.py --limit 200
 
# 2. Full dry-run: report the complete gap picture (counts of would_copy / already_present / …).
python3 scripts/wasabi-replication-repair.py
 
# 3. Execute the repair (writes copies). Start narrow, then widen.
python3 scripts/wasabi-replication-repair.py --execute --limit 500
python3 scripts/wasabi-replication-repair.py --execute

Flags: --batch-size N (progress checkpoint cadence, default 100), --limit N (cap docs, 0 = unlimited), --office-id <id> (restrict to one office).

Output

Per-batch progress lines go to stdout and Discord; a final JSON summary reports scanned / already_full / would_copy / copied / already_present / source_missing / no_key / errors / bytes. Exit code 0 = clean, 2 = missing config, 3 = one or more per-object errors.

Expected cost

Filling the ~113k missing copies adds ~18.5 GB → roughly +$0.13/month Wasabi storage.

Metadata backfill — wasabi-storageinfo-backfill.py (#5649)

wasabi-replication-repair.py copies objects between buckets at the S3 layer only — it has no update_one/$push and never writes back to MongoDB. So after the live repair run (2026-08-06: 77,289 files copied, 0 errors) the w4_db.files docs still showed storageInfo.length < 3 (56,968 of them) even though the files are now physically 3/3-present. Any app logic that trusts storageInfo to decide “does this file need repair” — including a naive re-run of the repair tool’s own dry-run — keeps reporting these ~77k files as under-replicated even though they are fixed.

scripts/wasabi-storageinfo-backfill.py closes that gap. For every doc still showing storageInfo < 3 it HEAD-checks all three canonical buckets directly (it imports the repair tool’s topology + helpers, so the bucket list/logic can never drift) and, for any bucket physically present but absent from the doc’s storageInfo array, $push-es a matching entry so the metadata reflects reality. Pushed entries carry bucketName + key + region/endpoint/tag and a "backfilled": true provenance marker.

  • Idempotent: only buckets confirmed present-via-HEAD and not already recorded are pushed — a second full pass is a no-op.
  • source_missing safety (#2709 phantom-source, ~3,958 docs): a doc whose object is on none of the three buckets has nothing to push (every HEAD 404s) and is skipped untouched. The script never fabricates a storageInfo entry for a bucket that does not physically hold the object.
  • Same creds as the repair tool — Wasabi pinbox24PublicOfficeAccessKeyId/...SecretAccessKey and W4_MONGODB_URI. Execution additionally needs w4_app write access (the repair tool only read Mongo). Owner role for the --execute run: infra-task / secret-manager / sys-admin, not dev-coder.
# Dry-run is the default — reports scanned / already_full / no_gap / source_missing /
# would_backfill / entries_added / errors and writes nothing.
python3 scripts/wasabi-storageinfo-backfill.py --limit 200      # smoke test
python3 scripts/wasabi-storageinfo-backfill.py                  # full dry-run
python3 scripts/wasabi-storageinfo-backfill.py --execute --limit 500   # narrow real run
python3 scripts/wasabi-storageinfo-backfill.py --execute        # full backfill

Same --batch-size / --limit / --office-id flags as the repair tool. Exit code 0 = clean, 2 = missing config, 3 = one or more per-doc errors.

Phantom-source metadata correction — wasabi-phantom-source-correction.py (#5650)

The #5650 investigation (posted on the issue, 2026-08-12) scanned all 56,968 w4_db.files docs with storageInfo < 3 and HEAD-checked every one against all three canonical Wasabi buckets. Two disjoint populations came out of that scan:

ClassDocsMeaningOwning tool
source_ok54,989Recorded bucket really holds the object — metadata just under-counts copies.wasabi-storageinfo-backfill.py (#5649)
unrecoverable (phantom source)1,979Recorded bucket 404s and all 3 canonical buckets 404. Genuine data loss — three historic bulk-insert batches (2020-10-02, 2020-10-28, 2021-10-22, one office each) wrote storageInfo referencing test-replicated-to-us-bucket without the object ever landing in storage.wasabi-phantom-source-correction.py (#5650, this section)

Nothing can be copied in for the 1,979 — there is no source anywhere. What #5650 fixes instead is that storageInfo still falsely claims a real Wasabi source for these docs (an app that trusts storageInfo presence has no way to tell “confirmed present” from “confirmed lost — never verified”). wasabi-phantom-source-correction.py closes that gap with an explicit, additive marker.

Business decision (2026-08-12, architect): correct the metadata quietly — do not notify the 3 affected Pinbox24 offices (any download of these 1,979 files already 404s regardless of what the metadata says; notification is a separate, deferred decision, tracked on #5650).

What it writes — additive only, nothing is deleted or shortened

storageInfo is never cleared, shortened, or nulled (its length is preserved deliberately — the investigation flagged an unchecked risk that some billing/reporting/quota logic might read storageInfo.length, and preserving it sidesteps that risk entirely rather than requiring a backend code audit first). The original entry’s fields (bucketName, dateTime, …) are left untouched — full audit trail of what the wrong original claim was. Only new fields are $set:

// doc-level
wasabiSourceVerifiedMissing:      true
wasabiSourceVerifiedMissingAt:    "<ISO8601>"
wasabiSourceVerifiedMissingIssue: "p24-infra#5650"
 
// storageInfo.$[entry] — every entry naming one of the 3 canonical buckets
storageInfo[].sourceVerifiedMissing:   true
storageInfo[].sourceVerifiedMissingAt: "<ISO8601>"

Idempotent by construction: the Mongo query excludes docs already carrying wasabiSourceVerifiedMissing: true, so a second full pass finds zero candidates and writes nothing.

Running it

Same safety pattern and credentials as the repair tool + backfill tool above (W4_MONGODB_URI, pinbox24PublicOfficeAccessKeyId/...SecretAccessKey). Execution additionally needs w4_app Mongo write access — owner role infra-task / secret-manager / sys-admin, not dev-coder.

# Dry-run is the default — reports scanned / source_ok / would_correct / no_key /
# no_recorded_source / errors and writes nothing. Report + matched-id sample written to
# /tmp/w5650_phantom_correction_report.json.
python3 scripts/wasabi-phantom-source-correction.py --limit 200      # smoke test
python3 scripts/wasabi-phantom-source-correction.py                  # full dry-run
python3 scripts/wasabi-phantom-source-correction.py --execute --limit 500   # narrow real run
python3 scripts/wasabi-phantom-source-correction.py --execute        # full correction

Same --batch-size / --limit / --office-id flags as the other two tools. Exit code 0 = clean, 2 = missing config, 3 = one or more per-doc errors. On --execute, every corrected doc also gets one line appended to /tmp/w5650_phantom_correction_audit.jsonl (_id, officeId, recorded buckets, timestamp, issue ref) — copy it off-host before the container/VM cycles, same rule as cleanup-deleted-records.js’s audit log.

This class of issue — phantom Wasabi source metadata, general fix path

If a future audit finds another w4_db.files (or similar) population where storageInfo/ equivalent metadata claims a source that a live HEAD check disproves on every configured bucket:

  1. Do not delete the record or the field. Add an explicit, additive “verified missing” marker (doc-level boolean + timestamp + issue ref, entry-level boolean + timestamp) so there is an audit trail of what was claimed and when it was disproven. This is the same conservative pattern wasabi-phantom-source-correction.py uses — copy it rather than re-deriving from scratch.
  2. Check for a billing/reporting/quota dependency on the field’s length/presence before deciding whether to also shrink/null the array. This script deliberately does not do that — verifying it requires a backend code audit that was out of scope for #5650’s read-only investigation.
  3. Business sign-off is required before any write, specifically on: (a) whether to notify affected end-users/offices, (b) whether the “verified missing” correction should also change application-visible behaviour (e.g. hide a broken download link) beyond the DB marker itself.
  4. Idempotency via query, not runtime state — exclude already-marked docs directly in the Mongo query (see build_query() in wasabi-phantom-source-correction.py) so a second full pass is provably a no-op without needing separate bookkeeping.

Follow-ups still open on #2709 (not this script)

  • Rename the test-* buckets (operational hazard — two independent teams hit the “test-* buckets are actually production” trap).
  • Fix uploadMultiFiles() to retry partial failures instead of answering on first success.
  • Fix the all-3-failed no-rollback bug that orphans FileModel docs with no storageInfo.

Graph View

Backlinks