p24-infra Docs

waha-traefik-down-ports-80-443

4 min read

[NIEAKTYWNE — WAHA zdecommissioned 2026-06-30] WAHA zastąpiona przez whatsup-android-chat-puller. Ten dokument zachowany jako archiwum historyczne. Patrz: issue #2007

WAHA gateway down — Traefik ports 80/443 closed (host still up)

Trigger: WAHAProbeDown (critical) fires — probe_success{job="blackbox_waha"} == 0, probe_http_status_code == 0 (connection timeout, not an HTTP error code). The public endpoint https://waha2.vps-h1.infra.zintegrowana.online/api/server/status times out (curl exit 28 / HTTP 000) from every vantage point.

This is a different and more severe signature than waha-session-stopped-after-reboot: there the WAHA session is STOPPED but the reverse proxy still answers (you get an HTTP response). Here the reverse proxy (root-traefik-1) itself is down, so nothing on vps-h1 answers on 80/443 at all.

Confirmation commands

Run from any host that can reach vps-h1 (72.60.32.61) — e.g. bms-4 or vps-i1:

# Host alive? SSH port should be OPEN even when Traefik is down.
timeout 8 bash -c 'cat </dev/null >/dev/tcp/72.60.32.61/22' && echo "22 OPEN"   # host up
timeout 8 bash -c 'cat </dev/null >/dev/tcp/72.60.32.61/443' || echo "443 DOWN" # proxy down
timeout 8 bash -c 'cat </dev/null >/dev/tcp/72.60.32.61/80'  || echo "80 DOWN"  # proxy down
 
# Authoritative probe state (run on vps-i1, the Prometheus host):
ssh root@217.154.82.162 'curl -s "http://localhost:9090/api/v1/query?query=probe_success%7Bjob%3D%22blackbox_waha%22%7D"'

Diagnosis matrix:

2280/443MeaningAction
OPENDOWNHost up, Traefik/WAHA container downrestart containers (below)
OPENOPEN, but probe still 0session STOPPED, proxy finesee session-stopped playbook
closedclosedHost down / rebootingwait / OVH-Hostinger console

Why this needs a human (current constraint, 2026-06-25)

vps-h1 is a PROTECTED, frozen WAHA-only host. It has no claude-runner and no automated agent has an SSH key to it — neither bms-4 nor vps-i1 can ssh root@72.60.32.61 (both get Permission denied (publickey)). Therefore an automated worker cannot restart the container and must escalate. The worker’s job is limited to: confirm the signature, fire the Discord P1 alert, and flag the issue human-action.

The fix itself requires the human’s SSH key (C:\Users\konar\.ssh\id_ed25519 → root@72.60.32.61).

Step-by-step fix (human, from the dev workstation)

ssh root@72.60.32.61
cd /root                       # compose file: /root/docker-compose.yml (tracked: hostinger/docker-compose.yml)
docker compose ps              # confirm root-traefik-1 / waha are Exited or missing
docker compose up -d           # bring the whole stack back up
docker compose logs --tail=50 root-traefik-1   # confirm 80/443 bound, no port conflict

Then verify the WAHA session came back (root cause from PR #1362 should auto-start it):

curl -s http://localhost:13000/api/sessions/default   # expect "status":"WORKING"

If the session is STOPPED (not WORKING), follow the session-stopped playbook — WHATSAPP_RESTART_ALL_SESSIONS=true + WHATSAPP_FILES_FOLDER=/app/.sessions (PR #1362) should make a docker compose up -d restore the session automatically; if not, start it via POST /api/sessions/default/start.

Verify resolution

From vps-i1 (authoritative): the probe clears within ~1–2 scrape cycles.

ssh root@217.154.82.162 'curl -s "http://localhost:9090/api/v1/query?query=probe_success%7Bjob%3D%22blackbox_waha%22%7D"'
# expect value "1"
curl -s https://waha2.vps-h1.infra.zintegrowana.online/api/server/status   # expect HTTP 200

The WAHAProbeDown alert auto-resolves; close the GitHub issue with a resolution note.

Prevention

  • The waha-session.service systemd unit (PR #1362) auto-starts the session after a reboot, but does not guarantee the Traefik proxy is up. Ensure the compose stack itself has restart: unless-stopped on root-traefik-1 and waha (check hostinger/docker-compose.yml) so a reboot/crash brings the proxy back without a human.
  • Durable gap: no automated agent can remediate vps-h1 because none has an SSH key. Tracked as a human-action item — granting a restricted, restart-only key (or a tiny on-host healer timer that runs docker compose up -d when 443 is down) would let this self-heal. Until then, every Traefik-down event on vps-h1 is human-gated.

Escalation path

  1. Discord P1 via P24_DISCORD_INFRA_SCRIPTS_ERRORS_WEBHOOK_URL (color red 15158332). Note: the value is empty in secrets/monitoring.env.sops but present in the on-server plaintext /opt/p24-infra/bms-4/.env — use that on bms-4.
  2. Label the GitHub issue human-action, set milestone to the human-review state.
  3. Human restores the stack per the fix above, then closes the issue.
  • docs/playbooks/waha-session-stopped-after-reboot.md — referenced by issue #1369’s design; covers the session STOPPED, proxy up case (root cause fixed in PR #1362). Note: as of 2026-06-25 that file did not yet exist in the repo despite being referenced.
  • docs/playbooks/waha-ingestion-failure.md — WAHA → waha-router → Supabase ingestion failures.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="traefik-waha",
    result="success",  # "success" | "failed" | "skipped"
    detail="Traefik outage on vps-h1 resolved — ports 80/443 restored",
    env="vps-h1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'traefik-waha', 'success', 'Traefik outage on vps-h1 resolved — ports 80/443 restored', 'vps-h1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks