Runbook — wa-web-collector recovery (#7480)
Recovery procedures for the wa-web-collector Baileys WhatsApp Web collector on bms-4
(54.36.123.110). Linked from every WaCollector* Prometheus alert annotation.
wa-web-collector is the ONLY WhatsApp capture path since the Android path was disabled 2026-10-05 (whatsup-android-chat-puller#223). When it is down, no messages are captured. Full operational context:
docs/whatsup-android-chat-puller-operations.md§1.
0. Orientation — where it runs
| Item | Value |
|---|---|
| Host | bms-4 (54.36.123.110), root via SSH key |
| Container | wa-web-collector, restart: unless-stopped, no systemd/pm2 unit |
| Compose project dir | /opt/p24-infra/wa-web-collector/ — moving to /opt/wa-web-collector/ after the #7480 Step 0 relocation. If the first path is absent, use the second. |
| Metrics | /metrics + /health on METRICS_PORT = 9477 |
| Session (QR) | auth volume (./auth → /app/auth) — deleting it forces a new human QR scan. Treat as secret-equivalent. |
| Logs | docker logs wa-web-collector (pino JSON) |
# resolve the project dir once per session
D=/opt/p24-infra/wa-web-collector; [ -d "$D" ] || D=/opt/wa-web-collector1. Alert → cause → action
| Alert | Likely cause | First action |
|---|---|---|
| WaCollectorContainerMissing (crit) | Container crashed and did not restart; host/Docker issue | ssh root@54.36.123.110 "cd $D && docker compose ps && docker compose up -d"; then §3 logs |
| WaCollectorRestartLoop (warn) | Repeated crash: bad auth state, OOM, unhandled exception | §3 logs for the crash cause; if auth-corrupt → §4 QR re-pair; if OOM → check host memory |
| WaCollectorTargetDown (warn) | Process hung, metrics server dead, or port/ufw lost. Container may still look “up” in cAdvisor | §2 read /metrics; if unreachable but container up → docker compose restart; verify ufw + 9477:9477 publish |
| WaCollectorDisconnected (crit) | WhatsApp socket dropped; in-process reconnect failing | §3 logs; if wa_logged_out==1 too → §4; else docker compose restart to force a fresh socket |
| WaCollectorLoggedOut (crit) | Session de-linked or banned — cannot self-heal | §4 QR re-pair. If a ban is suspected → §5 before reconnecting |
| WaCollectorProbeStale (warn) | Zombie socket: looks connected, delivers nothing | docker compose restart to force a fresh socket; re-check the self-probe recovers |
| WaCollectorNoMessagesBusinessHours (warn) | SOFT signal — traffic is bursty (1-23 msgs/h). May be genuine quiet, or silent breakage | Correlate with wa_connected/self-probe (§2) before acting. If connected & probe fresh → likely genuine quiet |
| WaCollectorPersistErrors (warn) | Supabase upsert failing — rows may be lost | §3 logs for the DB error; verify Supabase reachable + service key valid (secrets/wa-web-collector.env.sops) |
| WaCollectorMediaFailureRatio (warn) | Media uploads failing — attachments missing | §3 logs; verify Wasabi reachable + the write-only IAM key; check bucket wa-uploads |
| WaCollectorRetryBacklog (warn) | A downstream dep (Supabase/Wasabi) failing long enough that retries aren’t draining | Fix the underlying dep (PersistErrors / MediaFailureRatio); buffer drains once it recovers |
2. Reading /metrics and /health
From bms-4 (the port is allowlisted only to the Prometheus host, so read it locally):
ssh root@54.36.123.110 'curl -s localhost:9477/health; echo; curl -s localhost:9477/metrics | grep -E "^wa_"'Key gauges: wa_connected (1=socket usable), wa_logged_out (1=de-linked/banned),
wa_last_probe_ok_timestamp_seconds / wa_last_message_timestamp_seconds (compare to date +%s for
age), wa_upsert_errors_total / wa_media_upload_errors_total / wa_media_uploaded_total (counters),
wa_retry_buffer_rows (pending retries). In Prometheus/Grafana query the same names (job
wa_web_collector).
3. Reading logs
ssh root@54.36.123.110 "docker logs --since 1h wa-web-collector | tail -80"Healthy startup sequence: group allowlist resolved (with count) → opened connection to WA →
WhatsApp connection open. Each captured message logs batch upserted and
p24_whatsapp_messages batch upserted. A media failure logs media download/upload failed — row kept, media_url stays null (the row is still written). A de-link logs a loggedOut disconnect reason.
4. QR re-pair procedure (Docker)
Needed when the session is logged out / de-linked (WaCollectorLoggedOut, or a QR prompt appears on
startup). Requires physical access to the paired phone’s WhatsApp → Linked Devices.
D=/opt/p24-infra/wa-web-collector; [ -d "$D" ] || D=/opt/wa-web-collector
cd "$D"
docker compose down
# Move the dead session aside (do NOT delete until the new pairing is confirmed working)
mv auth "auth.bad.$(date +%Y%m%d%H%M%S)"
mkdir -p auth
# Run ATTACHED so the QR renders in the terminal; scan it from Linked Devices on the phone
docker compose run --rm wa-web-collector
# -> scan the QR, wait for "WhatsApp connection open", then Ctrl-C
docker compose up -d # restart detached with the now-paired auth volume
docker logs --since 5m wa-web-collector | tail -30 # confirm "WhatsApp connection open", NO QROnce stable for a few minutes, remove the auth.bad.* backup. If the QR never resolves to a connection,
suspect a ban (§5).
5. Suspected ban
A ban de-links the session AND affects every device linked to the same WhatsApp account at once
(including any other companions). If WaCollectorLoggedOut fires and re-pairing immediately logs out
again, or the account shows problems on the phone itself:
- Stop the container (
docker compose down) — do not hammer reconnects or repeatedly re-pair; that worsens a ban. - Check the WhatsApp account health directly on the phone.
- Escalate to the owner (radieu) — this is accepted risk, not an auto-recoverable fault. Risk context:
docs/whatsapp-web-collector-risk.md(whatsup-android-chat-puller repo).
6. Independent Discord alert (CF Worker heartbeat) vs Prometheus
wa-app-api’s alertCheck CF Worker cron (every 5 min) pages Discord when the collector’s heartbeat
is stale (>20 min, with an alerted_at storm guard). It is a separate safety net from the Prometheus
alerts here and keys off heartbeat freshness only. It does not catch loggedOut or zombie-socket
cases — the process stays alive and the heartbeat keeps ticking — which is exactly why
WaCollectorLoggedOut and WaCollectorProbeStale exist. If you see the Discord heartbeat alert, expect
WaCollectorContainerMissing/WaCollectorTargetDown to corroborate; if you see only WaCollectorLoggedOut
with a healthy heartbeat, the Discord alert will not have fired — trust the Prometheus one.
7. Applying a changed group allowlist
The whatsapp_groups allowlist (WA_COLLECTOR_GROUP_JIDS / the DB-sourced list) is fetched once at
startup. A restart is the accepted procedure to apply a change:
D=/opt/p24-infra/wa-web-collector; [ -d "$D" ] || D=/opt/wa-web-collector
ssh root@54.36.123.110 "cd $D && docker compose restart"
# confirm the new scope: look for 'group allowlist resolved' with the expected count
ssh root@54.36.123.110 "docker logs --since 2m wa-web-collector | grep 'group allowlist resolved'"A restart reuses the auth volume (no QR), so expect only a seconds-long reconnect.