p24-infra Docs

vps-h1-traefik-iptables-stale-rules

3 min read

Playbook: vps-h1 Traefik — stale iptables rules after Docker crash

Trigger

After a Docker OOM kill or forced host reboot on vps-h1, Traefik is unreachable on ports 80/443 despite all containers showing as Up. Symptom: curl -I https://waha2.vps-h1.infra.zintegrowana.online times out. Alerts: WAHAContainerDown, WAHAProbeDown, EndpointDown.

Root cause

When Docker is killed by the OOM killer (not stopped gracefully), it cannot clean up its iptables DNAT rules. On the next Docker start, new containers get new IPs and new DNAT rules are added — but the stale rules from the crashed session remain at lower line numbers and take priority.

Result: port 80/443 traffic is forwarded to an old (now dead or reassigned) IP instead of Traefik.

Confirm

ssh root@72.60.32.61
iptables -t nat -L DOCKER --line-numbers | grep -E 'dpt:https|dpt:http[^-]'
docker inspect root-traefik-1 --format '{{.NetworkSettings.Networks.root_default.IPAddress}}'

Stale rules exist if there are MULTIPLE lines for dpt:http or dpt:https pointing to different IPs, and Traefik’s actual IP is NOT at the lowest-numbered rule.

Fix

  1. Note Traefik’s current IP:

    TRAEFIK_IP=$(docker inspect root-traefik-1 --format '{{.NetworkSettings.Networks.root_default.IPAddress}}')
echo "Traefik IP: $TRAEFIK_IP"

  2. List all DNAT rules for ports 80/443, identify the stale ones (pointing to an IP that is NOT Traefik):

    iptables -t nat -L DOCKER --line-numbers | grep -E 'dpt:https|dpt:http[^-]'

  3. Delete stale rules by line number (delete higher number first to avoid renumbering):

    # If rule 4 (port 443) and rule 3 (port 80) point to the wrong IP:
iptables -t nat -D DOCKER 4
iptables -t nat -D DOCKER 3

  4. Verify only correct rules remain:

    iptables -t nat -L DOCKER --line-numbers | grep -E 'dpt:https|dpt:http[^-]'
# Should show exactly 2 lines, both pointing to $TRAEFIK_IP

  5. Test WAHA endpoint:

    curl -sf --max-time 5 https://waha2.vps-h1.infra.zintegrowana.online/api/server/status \
  -o /dev/null -w 'HTTP %{http_code}\n'
# Expected: HTTP 401 (means routing works, auth required)

Escalation

If deleting the stale rules doesn’t help (port still timing out):

  • Check Traefik logs: docker logs root-traefik-1 --tail=50
  • Restart the full compose: cd /root && docker compose down && docker compose up -d
  • After restart, re-check and delete stale rules again (the restart itself may re-add them)

Prevention

The only reliable prevention is avoiding abrupt Docker/host kills on vps-h1. Keep Docker’s OOM score low so it’s not the first victim:

echo -100 > /proc/$(docker inspect --format {{.State.Pid}} root-traefik-1)/oom_score_adj

This must be re-applied after each container restart (add to a cron or startup script).

Audit trail

DateEvent
2026-06-25First occurrence — vps-i1 OOM (GitHub Actions runner) caused cascading Docker crash on vps-h1. Manual iptables rule deletion resolved it.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="traefik-vps-h1",
    result="success",  # "success" | "failed" | "skipped"
    detail="Stale Traefik iptables rules flushed and Traefik restarted on vps-h1",
    env="vps-h1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'traefik-vps-h1', 'success', 'Stale Traefik iptables rules flushed and Traefik restarted on vps-h1', 'vps-h1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks