p24-infra Docs

wasabi-bucket-growth-spike

5 min read

Playbook: WasabiBucketGrowthSpike

Trigger

WasabiBucketGrowthSpike alert fires when increase(wasabi_bucket_bytes[24h]) > 5 GB persists for 2 hours.

How to Confirm (exact commands)

# 1. Check current bucket size and recent trend from cost-exporter metrics
Invoke-RestMethod https://prometheus.vps-i1.infra.zintegrowana.online/api/v1/query?query=wasabi_bucket_bytes{bucket="p24-infra"}
# Divide the value by 1e9 to get GB
 
# 2. Check Thanos sidecar upload activity (look for recent block uploads)
ssh root@217.154.82.162 "docker logs monitoring-thanos-sidecar-1 --since=24h 2>&1 | grep -E 'upload|block|error' | tail -50"
 
# 3. Check compactor activity
ssh root@217.154.82.162 "docker logs monitoring-thanos-compactor-1 --since=24h 2>&1 | grep -E 'compact|error|halted' | tail -30"
 
# 4. Check object count in the thanos/ prefix of the bucket
# (use Wasabi console or boto3 script — see scripts/check-wasabi-bucket.py if it exists)

Most Common Cause: Post-Outage Thanos Catch-Up

When the vps-i1 monitoring stack is restarted after an outage, Thanos sidecar uploads all accumulated TSDB blocks at once:

  • Prometheus is configured with --tsdb.min-block-duration=2h --tsdb.max-block-duration=2h
  • Each 2h block for this stack is ~500 MB–1 GB compressed (50+ scrape targets)
  • A 13h outage creates ~6-7 blocks → 3-7 GB uploaded in minutes
  • cost-exporter samples wasabi_bucket_bytes once every 24h, so the full delta appears as a single spike
  • The alert fires because the increase(wasabi_bucket_bytes[24h]) > 5 GB threshold is exceeded

This is NOT a runaway upload — it resolves itself when the backlog is cleared.

How to confirm it is post-outage catch-up

  1. Check for recent monitoring-stack outage issues (label bug + monitoring)
  2. Check Thanos sidecar logs — you’ll see a burst of msg="uploading block" entries followed by silence
  3. Check that the alert resolved itself within a few hours after firing
  4. wasabi_object_count{bucket="p24-infra"} should stabilize (not keep growing)

Step-by-Step Fix

Case A: Post-outage catch-up (most likely)

  1. Confirm the cause — check Thanos sidecar logs (command above). If you see a burst of block uploads followed by normal activity, this is catch-up.
  2. No action needed on the bucket — Thanos compactor will deduplicate and apply retention automatically within 1-2 hours (--wait mode).
  3. Verify alert resolves — within 24-48h the increase(wasabi_bucket_bytes[24h]) value will drop back below 5 GB as Thanos stops uploading and bucket size stabilizes.
  4. Close the GitHub issue with comment: “Post-outage Thanos catch-up — no runaway upload. Bucket growth was 13h of accumulated TSDB blocks uploaded at stack restart. Alert resolved automatically.”

Case B: Genuine runaway upload

Signs: Thanos sidecar logs show continuous uploads over 24+ hours, wasabi_object_count keeps growing with no plateau, cost-exporter shows >5 GB growth every day.

  1. Identify the source:

    • thanos/ prefix → Prometheus is generating blocks faster than normal (check rate(prometheus_tsdb_head_samples_appended_total[1h]))
    • pdfs/ prefix → pdf-service is uploading runaway reports
    • Bucket root → backup-exporter sees new backup files (check backup-scheduler)

  2. Stop the source (do not delete data yet):

    ssh root@217.154.82.162 "docker stop monitoring-thanos-sidecar-1"
# or for pdf-service:
ssh root@217.154.82.162 "docker stop monitoring-pdf-service-1"

  3. Identify and remove stale objects (use Wasabi console — do NOT run aws s3 rm blindly):

    • List most recent objects sorted by size: Wasabi console → bucket p24-infra → sort by Last Modified
    • Identify unexpected large objects or prefix floods

  4. Restart the stopped service once the root cause is fixed.

  5. Check Thanos block retention — if Thanos raw retention (--retention.resolution-raw=90d) has accumulated 90+ days of blocks, compactor will delete them automatically. No manual intervention needed.

Case C: Backup-exporter or log-shipping loop

Signs: growth is in the bucket root (backup JSON files) or loki/ prefix.

  1. Check backup-exporter logs: docker logs monitoring-backup-exporter-1 --since=24h
  2. Check Loki Wasabi storage config: monitoring/loki/loki-config.yml
  3. Fix the loop (usually a misconfigured upload interval or a script called in a tight loop)

Thanos Compactor — What It Does

The thanos-compactor container (running in --wait mode, check interval 1h) automatically:

  • Deduplicates overlapping blocks from sidecar uploads
  • Applies retention: raw data 90d, 5m resolution 180d, 1h resolution 365d
  • Compacts 2h blocks into larger 8h → 48h blocks (reduces object count)

If the bucket grows continuously despite the compactor running, check: docker logs monitoring-thanos-compactor-1 --since=24h | grep error

Alert Threshold Reference

ParameterValueRationale
exprincrease(wasabi_bucket_bytes[24h]) > 5 GBNormal Thanos daily growth is ~300 MB; 5 GB flags a real anomaly
for2hFilters out post-outage catch-up bursts (which complete in <30 min); genuine runaway uploads persist for hours

Changed from for: 30m to for: 2h in PR #1133 to prevent false positives after stack restarts.

Escalation Path

  1. If bucket is growing at >10 GB/day with no outage explanation → escalate to infra review session
  2. If Wasabi monthly cost increases significantly → check wasabi_bucket_bytes trend in Grafana cost dashboard
  3. If thanos-compactor is halted or shows errors → restart it: docker compose restart thanos-compactor

Prevention

  • Keep thanos-compactor running (restart: unless-stopped — already set)
  • Monitor wasabi_object_count trend in Grafana for gradual drift
  • Document all vps-i1 outages in GitHub issues — helps diagnose future bucket spikes
  • The WasabiBucketGrowthSpike alert with for: 2h will not fire for single catch-up events under ~2h duration
  • First occurrence: 2026-06-23 — issue #1122, caused by vps-i1 outage #1065 (2026-06-22 ~21:00 UTC)

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="other",
    resource="wasabi-bucket",
    result="success",  # "success" | "failed" | "skipped"
    detail="Wasabi bucket growth spike investigated — lifecycle policy or cleanup applied",
    env="vps-i1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'other', 'wasabi-bucket', 'success', 'Wasabi bucket growth spike investigated — lifecycle policy or cleanup applied', 'vps-i1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks