p24-infra Docs

report-scheduler-rerun

3 min read

Playbook: Re-running a failed report-scheduler report (vps-i1)

Status: ACTIVE Applies to: the report-scheduler service on vps-i1 (217.154.82.162). Trigger: a ReportRunFailed Prometheus alert / p24-infra-nc-alert issue where report_last_run_status{report_name="<name>"} == 0, and the day’s report deliverable (Wasabi PDF + email) is missing.

Prior occurrences: #4622 (2026-07-30), #5083 (2026-08-02) — both were transient DNS/name-resolution failures reaching Supabase during a vps-i1 monitoring blackout, not report-code defects.

How the report-scheduler runs

  • Code: /opt/p24-infra/infra-src/report-scheduler/report_scheduler.py (Python 3.9, sync).
  • Configs: /opt/p24-infra/reports/configs/daily/<report_name>.json.
  • Schedule: /etc/cron.d/report-scheduler — one line per report, 0 2 * * * root ... (02:00 local CEST = 00:00 UTC). Log stamps in the file are local (UTC+2).
  • Invocation (exact): 
    /usr/bin/python3 /opt/p24-infra/infra-src/report-scheduler/report_scheduler.py \
  --env-file /opt/p24-infra/monitoring/.env \
  --config /opt/p24-infra/reports/configs/daily/<report_name>.json \
  >> /var/log/report-scheduler.log 2>&1
  • Log: /var/log/report-scheduler.log.
  • Metric textfile: /var/lib/node_exporter/textfile_collector/report_<report_name>.prom (node_exporter textfile collector; report_name dots→underscores).
  • Alert rule: monitoring/prometheus/rules/reports.yml — pairs report_last_run_status == 0 with a freshness guard time() - report_last_run_timestamp_seconds < 21600 (6 h). The alert self-silences 6 h after the failed run even though the gauge stays 0; a self-close by prometheus-alerts-ai-triage.py after that window is a guard-silence artifact, not a recovery. The gauge only returns to 1 on the next successful run.

The metric semantics — why NOT to --dry-run first

report_scheduler.py writes the metric in a finally block on every exit, and --dry-run sets status = 1 (it skips only Wasabi upload + email). So a dry-run falsely clears the alert without delivering the PDF/email. In a live run, status = 1 is set only after a successful Wasabi upload; email delivery is best-effort and does not affect the status. Go straight to a live re-run — it is self-consistent (metric matches actual delivery) and produces the missing deliverable.

Remediation steps

  1. Confirm the fault is transient and already over (do NOT re-run into a still-broken env):

    ssh -i ~/.ssh/vps_root_key root@217.154.82.162 \
  'getent hosts mwkqmgadqnkkihjdeqsi.supabase.co && getent hosts s3.eu-central-2.wasabisys.com && \
   curl -s -o /dev/null -w "supabase %{http_code}\n" --max-time 15 https://mwkqmgadqnkkihjdeqsi.supabase.co/rest/v1/'

    Expect both DNS lookups to resolve and Supabase to answer 401 (reachable). Also inspect the failure in /var/log/report-scheduler.log — a Name or service not known / httpx.ConnectError is a transient DNS fault; a KeyError/TypeError/template error is a code bug — file a dev-issue instead of re-running.

  2. Live re-run (defaults --date to today; use --date YYYY-MM-DD only for a back-fill):

    ssh -i ~/.ssh/vps_root_key root@217.154.82.162 \
  '/usr/bin/python3 /opt/p24-infra/infra-src/report-scheduler/report_scheduler.py \
     --env-file /opt/p24-infra/monitoring/.env \
     --config /opt/p24-infra/reports/configs/daily/<report_name>.json \
     >> /var/log/report-scheduler.log 2>&1; echo "exit=$?"'  # PLAYBOOK: report-scheduler-rerun.md

    Exit 0 = success (status=1).

  3. Verify the metric flipped and the PDF was uploaded:

    ssh -i ~/.ssh/vps_root_key root@217.154.82.162 \
  'grep report_last_run_status /var/lib/node_exporter/textfile_collector/report_<report_name>.prom; \
   grep "Uploaded to Wasabi.*<report_name>-$(date +%Y-%m-%d)" /var/log/report-scheduler.log | tail -1'

    Expect report_last_run_status{...} 1 and an Uploaded to Wasabi: s3://p24-infra/reports/... line. Prometheus scrapes the textfile within ~1 scrape interval; the ReportRunFailed alert clears on the next evaluation.

  4. Audit log the operation to infra_operations (see server-operation playbook §post-flight).

Do NOT

  • Do NOT --dry-run as a “safe first step” — it writes status=1 and masks the real state.
  • Do NOT re-run while DNS/Supabase/Wasabi are still failing on vps-i1 — fix connectivity first.
  • Do NOT treat an auto-close after the 6 h freshness window as a recovery — verify the gauge is 1.

Graph View