p24-infra Docs

email-api-waf-ip-allowlist-403

4 min read

email-api.zintegrowana.online — edge WAF 403 "Your request was blocked."

Superseded (2026-08-07, #5263): the “restore the IP in the WAF rule” fix below assumes a normal, editable Custom Rule allowlist. A deeper investigation on #4112 (2026-08-06) found the actual block is a hidden, non-enumerable Cloudflare Managed Ruleset entry that a Free-plan zone’s Custom Rules skip action cannot override — see cloudflare-waf-hidden-managed-rule.md for the full diagnosis and the ruleId non-enumerability check (§How to confirm, step 5) that tells the two failure classes apart. openai-monitor.py and Alertmanager have since been routed around this block entirely via the Worker’s workers.dev subdomain (same playbook, §Recommended next steps). Run the ruleId check from that playbook first on any new occurrence — only fall back to the steps below if it turns out to be a genuine, editable allowlist regression.

Symptom: anything calling the p24-email-sender Worker at https://email-api.zintegrowana.online/{send,alert} fails with:

HTTP Error 403: Forbidden
Email worker 403: Your request was blocked.

The OpenAI Monitor workflow surfaces it as a traceback from scripts/openai-monitor.py send_email(); Alertmanager surfaces it as silently-undelivered alert emails (Discord is a separate path and keeps working).

First seen: #4112 (the allowlist was originally established). Recurred: #5263 (2026-08-03) — 2 failed OpenAI Monitor runs; the daily failed-jobs audit classified the root cause as unknown.

Root cause

email-api.zintegrowana.online is a proxied (orange-cloud) Cloudflare Worker route. A WAF custom rule on the zintegrowana.online zone blocks all traffic to that host except a small IP allowlist (vps-i1 217.154.82.162, vps-h1 72.60.32.61), returning a custom 403 body Your request was blocked.. This is a second defence layer in front of the Worker’s own ALLOWED_IPS check (workers/email-sender/src/index.js).

The 403 appears when the source IP has dropped off (or no longer matches) that WAF allowlist — e.g. the rule was edited, or a server’s egress IP changed. openai-monitor.yml deliberately SSHes into vps-i1 to run the script because vps-i1 is meant to be allowlisted; when vps-i1 itself is blocked, that design assumption has broken.

This is edge infrastructure, not a code bug. send_email() already sets an explicit User-Agent, and the OpenAI cost pull earlier in the run succeeds.

Distinguish from the look-alike UA block (do this first — 60s, from an allowlisted host)

There are two different Cloudflare 403s. The body text tells them apart:

BodyClassFix
Your request was blocked.This playbook — hostname-wide IP-allowlist WAF rule. Blocks every UA (curl, browser, urllib) and every path.Restore the IP in the WAF rule (below).
error code: 1010Python-urllib default-UA edge block — see cloudflare-waf-python-urllib-403.md. Only the Python-urllib UA; curl/browser reach the Worker (401).Send an explicit User-Agent.

Probe from the host the caller runs on (vps-i1 for the OpenAI monitor):

# All of these returning 403 "Your request was blocked." => IP-allowlist block (this playbook)
for ua in "curl/8" "Python-urllib/3.9" "Mozilla/5.0"; do
  curl -s -o /dev/null -w "%{http_code} $ua\n" -X POST https://email-api.zintegrowana.online/send \
    -A "$ua" -H "Content-Type: application/json" -d '{}'
done
# Control — proves the host is not globally CF-blocked (curl UA => 401 from the Worker):
curl -s -o /dev/null -w "%{http_code}\n" -X POST https://p24-meta-dispatcher.radieu.workers.dev/queue-issue \
  -H "Content-Type: application/json" -d '{}'

401 from /send means the edge passed and the Worker ran — a different (auth/IP-at-Worker) problem, not this one.

Fix (requires WAF-scoped credentials)

  1. Restore the allowlisted IP in the WAF rule: CF dashboard → zintegrowana.online → Security → WAF → Custom rules → the rule guarding the email-api host (its block response is the custom body Your request was blocked.). Ensure the allowlisted-IP set includes 217.154.82.162 (vps-i1) and 72.60.32.61 (vps-h1) — keep it in sync with the Worker’s ALLOWED_IPS in workers/email-sender/src/index.js.
  2. Verify from an allowlisted host: POST /send should return 401 (reaches the Worker), not 403.
  3. Re-run the affected job: gh workflow run openai-monitor.yml --repo radieu/p24-infra.

Credential gotcha (hit during #5263)

The WAF-scoped token in SOPS, CF_EDIT_ALL_ZONES_API_TOKEN, was invalid (/user/tokens/verify → Invalid API Token) — so no automated WAF management is possible until secret-manager rotates it. The other CF tokens (CF_API_TOKEN, CLOUDFLARE_TOKEN_ZINTEGROWANA, CF_WORKERS_API_TOKEN) are DNS/Workers-scoped and return Authentication error on the rulesets API. The Global API key that can edit WAF rules lives in administration.env.sops (dev-only; infra-task workers cannot decrypt it) — so this fix is human-action / network-admin, not a worker auto-fix.

Graph View