p24-infra Docs

waha-ingestion-failure

5 min read

[NIEAKTYWNE — WAHA zdecommissioned 2026-06-30] WAHA zastąpiona przez whatsup-android-chat-puller. Ten dokument zachowany jako archiwum historyczne. Patrz: issue #2007

Playbook: wa-ai-to-inbox pipeline failure

Trigger scenarios

  • Messages pile up in new status and never reach na_zgloszeniu_email
  • Messages stuck in processing status > 30 minutes
  • Messages in processingError status
  • n8n workflow wa-ai-to-inbox errors every execution

Quick status check

# Message counts by status
psql "$SUPABASE_DB_URL" -c "
  SELECT status, COUNT(*) FROM p24_whatsapp_messages GROUP BY status;"
 
# Messages in processing with age
psql "$SUPABASE_DB_URL" -c "
  SELECT COUNT(*), MIN(processing_started_at), MAX(processing_started_at)
  FROM p24_whatsapp_messages WHERE status='processing';"
 
# Which groups are monitored (claimable)
psql "$SUPABASE_DB_URL" -c "
  SELECT id, alias, name, monitored FROM whatsapp_groups ORDER BY monitored DESC;"

Or via Supabase MCP:

SELECT status, COUNT(*) FROM p24_whatsapp_messages GROUP BY status;

Root cause catalogue

1. Messages stuck in processing (timeout scenario)

Cause: Claude CLI took longer than 300s for an oversized prompt (>20 messages in one group).

Confirm: Check n8n execution list for 300s error entries on wa-ai-to-inbox. Check claude-proxy logs on bms-4:

ssh root@54.36.123.110 "journalctl -u claude-proxy --since '1 hour ago' | tail -40"

Fix:

-- Reset all stuck processing messages back to new
UPDATE p24_whatsapp_messages
SET status = 'new', processing_started_at = NULL
WHERE status = 'processing';

The “Group by ChatId” n8n node now batches groups with >20 messages into 20-message sub-batches (added 2026-06-24). Each batch produces a prompt ≤ ~5KB which resolves in <120s.

2. processingError messages from watchdog

Cause: wa-processing-watchdog workflow (ID: 5SwkWzMl0BrU0QSz) calls reset_stuck_wa_messages() every 10 minutes. Default threshold was 10 min (increased to 30 min in migration 20260624_wa_claim_limit_and_watchdog_threshold). If messages are in processing longer than the threshold, they go to processingError.

Fix:

-- Reset processingError back to new for re-processing
UPDATE p24_whatsapp_messages
SET status = 'new', processing_started_at = NULL
WHERE status = 'processingError';

3. PGRST203 — PostgREST function overload ambiguity

Cause: PostgreSQL has two versions of claim_wa_messages() — the old no-args version AND the new p_limit integer DEFAULT 50 version. PostgREST can’t decide which to call.

Symptom: n8n “Claim Messages” node errors with 300 - PGRST203 "Could not choose the best candidate function".

Fix:

DROP FUNCTION IF EXISTS public.claim_wa_messages();
-- Leaves only: claim_wa_messages(p_limit integer DEFAULT 50)

Check remaining functions after fix:

SELECT proname, pg_get_function_arguments(oid) FROM pg_proc
WHERE pronamespace = 'public'::regnamespace AND proname = 'claim_wa_messages';

4. Claude proxy not running (504 timeout from n8n)

Cause: claude-proxy systemd service on bms-4 is down, or ufw blocks port 9999 from n8n Docker network.

Confirm:

ssh root@54.36.123.110 "systemctl status claude-proxy"
ssh root@54.36.123.110 "ss -tlnp | grep 9999"
# Test from n8n container:
ssh root@54.36.123.110 "docker exec bms-4-n8n-1 nc -zv 172.17.0.1 9999"

Fix — restart proxy:

ssh root@54.36.123.110 "systemctl restart claude-proxy"

Fix — ufw (if nc test fails but service is running):

ssh root@54.36.123.110 "ufw allow from 172.17.0.0/16 to any port 9999 comment 'n8n Docker bridge'"
ssh root@54.36.123.110 "ufw allow from 172.18.0.0/16 to any port 9999 comment 'n8n Docker bridge alt'"

Proxy service file: /etc/systemd/system/claude-proxy.service Proxy script: /opt/claude-proxy/server.py

Key requirement in server.py: stdin=subprocess.DEVNULL (prevents claude -p from waiting for stdin on first run with empty pipe warning). Timeout is 300s.

5. Supabase authorization error (403/401 from Claim Messages node)

Cause: The “Claim Messages” n8n node uses a hardcoded sb_secret_ key. New Supabase key format requires BOTH apikey AND Authorization: Bearer <key> headers.

Confirm: Check n8n execution error for “JWT expired” or “Invalid API key”.

Fix — headers required:

apikey: <sb_secret_...>
Authorization: Bearer <sb_secret_...>
Content-Type: application/json

The correct key is SUPABASE_SERVICE_ROLE_KEY from secrets/n8n-bms4.env.sops.

Long-term fix: Replace hardcoded key in the n8n node with $env.SUPABASE_SERVICE_ROLE_KEY (requires merging PR that updates the SOPS file and restarting n8n containers).

6. Messages from non-monitored groups never claimed

Cause: claim_wa_messages() only claims messages from groups where whatsapp_groups.monitored = true. Messages from groups not in whatsapp_groups, or with monitored = false, are permanently stuck in new.

Confirm:

SELECT wm.status, wg.monitored, COUNT(*)
FROM p24_whatsapp_messages wm
LEFT JOIN whatsapp_groups wg ON wm.chat_id = wg.id
GROUP BY wm.status, wg.monitored;

Fix: Either add the group to whatsapp_groups with monitored = true, or accept these messages won’t be processed.

Normal drain rate

With fixes from 2026-06-24:

  • 50 messages claimed per 5-minute execution
  • Each execution takes 30–200s depending on group count and message count
  • Rate: ~50 messages per 5 minutes = 600/hour
  • 500 backlogged messages: ~50 minutes to drain

Escalation path

  1. Check claude-proxy logs for auth expiry: journalctl -u claude-proxy --since '2h ago' | grep -i error
  2. If claude-runner auth expired on bms-4: see playbook claude-runner-auth-expiry.md
  3. If DB is unreachable: check Supabase status page and SUPABASE_URL env var in n8n-bms4.env

Prevention

  • Watchdog threshold: 30 minutes (allows full 50-message batch to complete)
  • Batch size: max 20 messages per Claude call (prevents oversized prompts)
  • claim_wa_messages(p_limit=50): bounds each execution to 50 messages
  • stdin=subprocess.DEVNULL in claude-proxy: prevents stdin hang

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="waha-ingestion",
    result="success",  # "success" | "failed" | "skipped"
    detail="WAHA ingestion failure resolved — session restored and queue cleared",
    env="vps-h1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'waha-ingestion', 'success', 'WAHA ingestion failure resolved — session restored and queue cleared', 'vps-h1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks