p24-infra Docs

socat-supabase-zombie-crash-loop

3 min read

socat-supabase zombie crash loop — n8n alert avalanche

What triggers this

socat-supabase.service on bms-4 provides a TCP IPv4→IPv6 proxy for Supabase PostgreSQL (0.0.0.0:15432 → db.mwkqmgadqnkkihjdeqsi.supabase.co:5432). n8n workflows that use direct PostgreSQL credentials (not REST API) connect through this proxy.

Trigger: the socat process dies (OOM, SIGKILL, or manual restart) but leaves a zombie or forked child holding the TCP LISTEN socket on 0.0.0.0:15432. Systemd restarts the service every 5 s; each new socat immediately fails with Address already in use. The crash loop runs indefinitely.

First seen: 2026-06-27 ~11:29 UTC — crash loop ran for 4+ hours, restart counter reached 3225+.

Symptoms (alert avalanche)

A single socat zombie cascades into ~15+ GitHub alert issues:

AlertRoot cause
N8nHighFailureRatio: >30% for 15mAll Supabase-PostgreSQL workflows fail
N8nWorkflowPersistentlyFailing: mezmo-alert-routerUses Supabase; 42 errors/h → feedback loop
N8nWorkflowPersistentlyFailing: WAHA MonitorUses Supabase
N8nWorkflowPersistentlyFailing: Agent Error AnalyzerUses Supabase
N8nWorkflowFailureSpikeTriggered by high failure count
HourlyTriageNotRunning / HourlyTriageMissedRunHourly triage runs via n8n + Supabase
ReportNotGenerated / WeeklyReportNotGeneratedReport workflows use Supabase
BackupExporterDownMay use socat tunnel

None of these indicate actual service outages. N8n itself is running; Supabase is up. The mezmo-alert-router failure creates a self-reinforcing feedback loop (fails → logs to Mezmo → triggers itself again).

Confirmation

# On bms-4:
ss -tlnp | grep 15432
# If you see TWO different socat PIDs, or the PID shown does NOT match
# `systemctl show socat-supabase --property=MainPID`, a zombie is holding the port.
 
systemctl status socat-supabase
# Look for: Active: activating (auto-restart) + restart counter > 10
 
journalctl -u socat-supabase -n 5 --no-pager
# Look for: E bind(5, {AF=2 0.0.0.0:15432}, 16): Address already in use

Fix

# 1. Find and kill the zombie socat holding port 15432
ZOMBIE_PID=$(ss -tlnp | grep 15432 | grep -oP 'pid=\K[0-9]+')
echo "Killing zombie PID: $ZOMBIE_PID"
kill $ZOMBIE_PID
 
# 2. Restart the service
sleep 1
systemctl restart socat-supabase
 
# 3. Verify
systemctl status socat-supabase --no-pager | head -5
ss -tlnp | grep 15432
# Should show a single socat PID matching MainPID

After fix: n8n workflows reconnect to Supabase within ~30 s. Alert conditions clear in ~15 min. Close the GitHub alert issues manually or wait for them to auto-close on next Alertmanager resolve.

Prevention (already applied 2026-06-27)

/etc/systemd/system/socat-supabase.service now includes:

ExecStartPre=-/bin/fuser -k 15432/tcp   # force-kill any process holding port before start
KillMode=mixed                           # systemd kills parent + all forked children on stop

The - prefix on ExecStartPre means failure is non-fatal (no zombie = fuser exits non-zero, that’s fine). With KillMode=mixed, systemd sends SIGTERM to the main process and SIGKILL to the entire cgroup after TimeoutStopSec, ensuring forked children can’t hold the socket open.

Escalation

If the service keeps restarting after the fix:

  1. Check connectivity: nc -zv db.mwkqmgadqnkkihjdeqsi.supabase.co 5432 from bms-4
  2. Check Supabase status: https://status.supabase.com
  3. Check if fuser binary is available: which fuser (install with apt install psmisc if missing)
  4. As last resort: reboot bms-4 (will clear all zombie processes); notify on Discord first

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="socat-supabase",
    result="success",  # "success" | "failed" | "skipped"
    detail="socat-supabase zombie crash loop resolved — stale process killed, service restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'socat-supabase', 'success', 'socat-supabase zombie crash loop resolved — stale process killed, service restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks