p24-infra Docs

cloudflare-waf-python-urllib-403

3 min read

Cloudflare edge 403 on *.workers.dev calls from Python

Symptom: a Python script calling a p24 Cloudflare Worker (p24-meta-dispatcher, p24-auth-worker, monitoring-watchdog, p24-tg-notify-proxy, …) fails with:

HTTP Error 403: Forbidden

…while the exact same request from curl — same URL, method, body and API key — succeeds.

First seen: #4292 (2026-07-19) — daily-session-audit.yml created audit issue #4279 fine but every /queue-issue dispatch was rejected, so the DSA worker was never spawned. The daily cron had been failing at this step and the failed-jobs audit classified the root cause as unknown.

Root cause

The 403 is not from the Worker. It is emitted by Cloudflare’s edge (WAF / bot protection) before the request ever reaches Worker code.

urllib.request sends User-Agent: Python-urllib/3.x by default, and the zone’s managed bot rules block that UA outright.

This is easy to misdiagnose as a bad QUEUE_API_KEY, because 403 looks like an auth failure. It is not — the key is never evaluated.

Diagnosis — 30 seconds, no credentials needed

Our Workers’ auth guard returns 401 for a missing/bad key, and never 403. So the status code alone tells you which layer rejected you. Probe unauthenticated and vary only the UA:

URL="https://p24-meta-dispatcher.radieu.workers.dev"
 
# curl's own UA
curl -s -o /dev/null -w "%{http_code}\n" -X POST "$URL/queue-issue" \
  -H "Content-Type: application/json" -d '{"issue_number":1}'
# → 401  = request reached the Worker
 
# the Python default UA
curl -s -o /dev/null -w "%{http_code}\n" -X POST "$URL/queue-issue" \
  -A "Python-urllib/3.12" -H "Content-Type: application/json" -d '{"issue_number":1}'
# → 403  = blocked at the edge, Worker never ran
StatusRejected byMeaning
401Worker (verifyApiKey)Edge passed. Key missing/wrong — a genuine credential problem.
403Cloudflare edgeWorker never ran. Almost always the User-Agent. Key is irrelevant.
200 / 201WorkerSuccess (200 = already in flight, 201 = queued).

Never rotate QUEUE_API_KEY in response to a 403 — it will not help.

Fix

Send an explicit, non-default User-Agent on every outbound request:

USER_AGENT = "p24-infra-<component>/1.0 (+https://github.com/radieu/p24-infra)"
 
req = urllib.request.Request(
    f"{QUEUE_API_URL}/queue-issue",
    data=payload,
    headers={
        "Authorization": f"Bearer {QUEUE_API_KEY}",
        "Content-Type": "application/json",
        "User-Agent": USER_AGENT,          # ← required; the default UA is blocked
    },
    method="POST",
)

requests and httpx set their own non-blocked UA, so they do not hit this — but set an explicit one anyway so the caller is identifiable in Cloudflare’s request logs.

Also log the response body

A bare HTTP Error 403: Forbidden is what made #4292 land in the audit as unknown. Always read the body out of HTTPError — it is the only way to tell an edge block from a Worker rejection after the fact:

except urllib.error.HTTPError as e:
    detail = e.read().decode("utf-8", "replace")[:300].strip()
    raise RuntimeError(f"Queue API HTTP {e.code}: {detail or e.reason}") from e

Checklist for any new Python caller of a Worker endpoint

  • Explicit User-Agent header set
  • HTTPError handler reads and reports e.code and the body
  • Timeout passed to urlopen (a hung edge connection otherwise stalls the cron)
  • Regression test asserting the UA header — see scripts/tests/test_dsa_create_audit_issues.py

Known callers (audited 2026-07-19)

FileStatus
scripts/dsa/create-audit-issues.pyFixed — #4292
scripts/agent-push-error.pyFixed — #4292 (was failing silently, return False)
scripts/alert-ingest.pyN/A — targets Supabase REST + Discord + GitHub API, not a Worker
  • docs/playbooks/email-api-waf-ip-allowlist-403.md — the look-alike Your request was blocked. hostname-wide IP-allowlist WAF block (different body, different fix — check the body text first)
  • docs/playbooks/gh-actions-failed-jobs-taxonomy.md
  • infra-src/meta-dispatcher/src/index.ts — verifyApiKey, the 401-not-403 guard

Graph View

Backlinks