p24-infra Docs

app-down

2 min read

BrandPilot — App Down

Trigger: probe_success{job="blackbox",instance=~".*brandpilot.*"} = 0 OR /api/health returns non-200 for >2 consecutive probes (1 minute).

1. Confirm the alert is real

curl -I https://p24-brandpilot.zintegrowana.online/api/health
  • 200 + {"status":"ok",...} → false positive, probe misconfigured. Check Alertmanager.
  • 503 / connection refused / timeout → real outage, continue below.

2. Diagnose

2a. Check Vercel deployment status

gh api /v1/deployments?app=brandpilot --hostname vercel.com

Or open the Vercel dashboard → project brandpilot → Deployments tab.

Look for:

  • Last deployment status (READY / ERROR / BUILDING)
  • Whether a recent deploy introduced a regression
  • Build logs for compile/runtime errors

2b. Check if it’s a deployment failure vs app crash

Deployment failure — Vercel shows ERROR state on latest deployment; previous deployment was READY.

App crash — latest deployment is READY but the health endpoint returns error. Check Vercel runtime logs:

gh api /v1/deployments/<deployment-id>/events --hostname vercel.com

Or in Vercel UI: Deployment → Functions → Runtime Logs.

Common crash causes:

  • SUPABASE_SERVICE_ROLE_KEY missing/expired → app cannot initialize Supabase admin client
  • ANTHROPIC_API_KEY missing → Claude SDK fails to initialize
  • Edge function timeout (Vercel free tier: 10s limit)

2c. Check if it’s a DNS/Cloudflare issue

# From a machine outside the local network:
curl -H "Host: p24-brandpilot.zintegrowana.online" https://p24-brandpilot.zintegrowana.online/api/health
nslookup p24-brandpilot.zintegrowana.online

If DNS resolves incorrectly → Cloudflare zone issue → escalate to p24-infra session (requires Cloudflare access).

3. Auto-solve

Fix A: Redeploy last stable version

# Trigger redeploy of the last successful deployment via Vercel CLI or GH Actions
gh workflow run deploy.yml -R p24-infra/brandpilot

Or in Vercel UI: go to the last READY deployment → “Redeploy”.

Fix B: Roll back to previous deployment

In Vercel UI: find the last READY deployment before the failing one → “Promote to Production”.

Fix C: Force-push a known-good commit

# Only if you have identified the bad commit:
git revert <bad-commit-sha>
git push origin dev
# Then create PR → dev → merge → Vercel auto-deploys

4. Escalate to human when

  • DNS is the root cause (Cloudflare zone change needed) → p24-infra session
  • SSL cert expired and Cloudflare cannot renew → p24-infra session
  • The outage persists after 2 redeploys
  • Vercel account suspended / billing issue → human: check Vercel dashboard

5. Verify fix

curl -s https://p24-brandpilot.zintegrowana.online/api/health | python -m json.tool
# Expect: { "status": "ok", "supabase": { "ok": true }, ... }

Confirm Alertmanager clears EndpointDown within 2 probe intervals (~1 minute).

Graph View

Backlinks