p24-infra Docs

vps-h1-stale-docker-iptables

4 min read

Playbook: vps-h1 stale Docker iptables rules

Trigger

After docker compose down && docker compose up -d on vps-h1, the new Docker network gets new container IPs. Docker creates correct DNAT rules for the new network, but the old rules from the previous network are NOT cleaned up. The stale rules take priority and redirect public ports to dead IPs or wrong containers.

Symptoms:

  • WAHAProbeDown — HTTPS probe to waha2.vps-h1.infra.zintegrowana.online times out
  • WAHAContainerDown — container_last_seen{name="waha"} absent (cAdvisor unreachable)
  • PromtailDown — port 9080 scrape timing out from vps-i1
  • N8nWorkflowPersistentlyFailing — WAHA Monitor can’t reach WAHA API

Discovered: 2026-06-25 after PR #1362 deployment recreated vps-h1 compose stack.

Confirm

ssh root@72.60.32.61
 
# List all DNAT rules with IPs
iptables -t nat -L DOCKER -n -v --line-numbers
 
# List current container IPs
docker network inspect root_default --format '{{json .Containers}}' | \
  python3 -c 'import sys,json; d=json.load(sys.stdin); [print(v["Name"], ":", v["IPv4Address"]) for v in d.values()]'

Problem indicator: DNAT rules pointing to IPs (e.g. .7, .8) that don’t appear in root_default container list. Multiple rules for the same port (old + new network DNAT).

Step-by-step fix

Option A — Restart Docker (preferred for full cleanup, causes ~30s downtime)

ssh root@72.60.32.61
# All containers will stop, Docker cleans iptables, then restores correct rules on start
systemctl restart docker
# After restart, compose stack is NOT auto-started (restart:unless-stopped only applies within a run)
cd /root && docker compose up -d
# Restore WAHA session if needed:
systemctl restart waha-session  # starts the session if not auto-started

Option B — Surgical iptables fix (no downtime)

ssh root@72.60.32.61
 
# 1. Find stale DNAT rules (IPs not in current network)
iptables -t nat -L DOCKER -n -v --line-numbers
docker network inspect root_default --format '{{json .Containers}}' | \
  python3 -c 'import sys,json; d=json.load(sys.stdin); [print(v["Name"], ":", v["IPv4Address"]) for v in d.values()]'
 
# 2. Delete stale rules from highest number to lowest (to preserve rule positions)
# Example: if rules 3 and 4 point to dead IPs
iptables -t nat -D DOCKER 4
iptables -t nat -D DOCKER 3
 
# 3. Verify correct rules remain
iptables -t nat -L DOCKER -n --line-numbers | grep -E 'dpt:80|dpt:443|dpt:8080|dpt:9080'
 
# 4. Confirm WAHA reachable
curl -s -o /dev/null -w '%{http_code}' --max-time 5 \
  https://waha2.vps-h1.infra.zintegrowana.online/api/server/status
# Expected: 401 (WAHA requires API key — 401 = reachable)

Root cause

Docker tracks iptables rules per-network using the bridge interface name (e.g. br-2530276e6882). When compose recreates with a new network name, Docker adds new DNAT rules but does NOT flush the old rules — it only removes rules it believes it owns for that specific bridge, and the old bridge is already gone. The stale rules match before the correct ones (iptables first-match wins) and send traffic to non-existent container IPs.

Prevention

  1. Before deploy: docker compose down cleans up the old network if Docker is still tracking it. If compose was killed or the network was manually removed, stale rules persist.
  2. After any compose recreation on vps-h1: verify iptables -t nat -L DOCKER -n -v shows no duplicate rules for ports 80, 443, 8080, 9080.
  3. Safest deployment procedure for vps-h1: 
    cd /root
docker compose down          # removes containers + network + iptables rules
git pull                     # get latest config
docker compose up -d         # creates fresh network + correct iptables
systemctl restart waha-session  # ensure WAHA session is started

Escalation

If surgical rule deletion doesn’t fix connectivity:

  1. Check if the correct DNAT rules were re-added by Docker
  2. Check iptables -t nat -L POSTROUTING -n for MASQUERADE rules
  3. If still broken, restart Docker (Option A) — this guarantees clean iptables state
  4. If Docker restart fails, check journalctl -u docker --since -10m for errors

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="vps-h1-docker",
    result="success",  # "success" | "failed" | "skipped"
    detail="Stale Docker iptables rules flushed on vps-h1",
    env="vps-h1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'vps-h1-docker', 'success', 'Stale Docker iptables rules flushed on vps-h1', 'vps-h1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks