p24-infra Docs

socat-supabase-watchdog

3 min read

Playbook: socat-supabase watchdog

Service: socat-supabase.service — IPv4→IPv6 proxy on bms-4 :15432 Watchdog: p24-socat-watchdog.timer — 5-min TCP + zombie check

Problem pattern

socat with fork spawns a child per connection. If the parent hangs or zombie children accumulate past ~20, the proxy silently stops forwarding connections while systemctl status still shows active (running). This becomes a cascade failure vector: queue-analyst, psql migrations, and any service using the direct Supabase DB port all fail.

What the watchdog does

Every 5 minutes, p24-socat-watchdog.service (oneshot, User=root): 0. Duplicate-unit cleanup (#2309): if supabase-pg-proxy.service exists and is not already masked AND socat-supabase.service is active, systemctl disable --now + mask it. That unit was a duplicate :15432 listener (same upstream) that could never bind and crash-looped 52k+ times. The guard requires socat-supabase to be active first, so the only working proxy is never removed; the step is idempotent (skips once the unit reports masked).

  1. nc -z -w5 127.0.0.1 15432 — verifies port actually accepts TCP
  2. Counts zombie processes (ps ax -o stat= | grep "^Z")
  3. If port dead OR zombies > 20: restart socat-supabase.service + Discord + GH issue

Manual triage

# Check watchdog timer + last run
systemctl status p24-socat-watchdog.timer
journalctl -u p24-socat-watchdog --since "1 hour ago" --no-pager
 
# Check socat process tree
ps aux | grep socat
ps ax | awk '{print $3}' | grep -c "^Z"   # zombie count
 
# Test port manually
nc -z -w3 127.0.0.1 15432 && echo "OK" || echo "DEAD"
 
# Force watchdog run
systemctl start p24-socat-watchdog.service

Deploy / update

# Copy files (from local or bms-4 git pull)
cp deploy/p24-socat-watchdog.service /etc/systemd/system/
cp deploy/p24-socat-watchdog.timer   /etc/systemd/system/
cp scripts/socat-supabase-watchdog.sh /opt/p24-infra/scripts/
chmod +x /opt/p24-infra/scripts/socat-supabase-watchdog.sh
 
systemctl daemon-reload
systemctl enable --now p24-socat-watchdog.timer
systemctl status p24-socat-watchdog.timer

Escalation

If socat keeps restarting every 5 min despite restarts:

  1. Check Supabase DB status at status.supabase.com
  2. Check bms-4 outbound TCP to IPv6: ping6 db.mwkqmgadqnkkihjdeqsi.supabase.co
  3. Check socat binary: socat --version
  4. Fallback: use Supabase Management API (no proxy needed) for all DB ops until resolved

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="socat-supabase",
    result="success",  # "success" | "failed" | "skipped"
    detail="socat-supabase watchdog triggered — tunnel restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'socat-supabase', 'success', 'socat-supabase watchdog triggered — tunnel restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks