Playbook — p24-network-admin

Role: p24-network-admin
Scope: DNS (Cloudflare), firewall (ufw / firewalld), Caddy reverse proxy, CF Tunnel, SSL certs, SSH hardening
Status: ACTIVE

Headscale VPN fully removed 2026-07-08. All servers are reached via public IP + SSH key only — no VPN dependency remains. See docs/playbooks/server-ssh-connectivity.md. Any Headscale/Tailscale references below are historical / kept for incident-history context only.

When to Use This Role

Trigger keywords (any of these → load this playbook and adopt p24-network-admin):

DNS change, DNS record, Cloudflare DNS, A record, CNAME, TXT, DNS propagation
Firewall, ufw, firewalld, port block, allow port, deny IP, whitelist IP
Caddy, reverse proxy, vhost, HTTPS, Caddyfile, upstream
CF Tunnel, cloudflared, Cloudflare Tunnel
SSL cert, certificate, cert expiry, ACME, Let’s Encrypt
SSH hardening, authorized_keys, sshd_config, SSH access

Pre-Flight Checklist (mandatory before any network change)

Identify target server(s):

vps-i1  217.154.82.162  — monitoring stack (firewalld)
vps-h1  72.60.32.61     — general AI worker (ufw)
bms-1   94.23.26.113    — Pinbox24 production (ufw)
bms-2   145.239.133.104 — MongoDB PRIMARY (ufw)
bms-3   51.68.155.224   — MongoDB secondary (ufw)
bms-4   54.36.123.110   — n8n + AI agents (ufw)
lap1    lap1-ssh.zintegrowana.online — CF Tunnel, variable uptime

Every server above also has a backup SSH path via Cloudflare Tunnel (added 2026-08-11, independent of the public IP — useful if the primary IP is unreachable): bms1-ssh / bms2-ssh / bms3-ssh / bms4-ssh / i1-ssh / h1-ssh .zintegrowana.online, SSH config aliases bms1-tunnel … h1-tunnel. Full detail: docs/playbooks/fleet-backup-ssh-cf-tunnel.md.

Verify SSH reachability before change:
```
ssh -i C:\Users\konar\.ssh\id_ed25519 -o ConnectTimeout=10 -o BatchMode=yes root@<IP> "echo ok"
```
If SSH fails → do NOT proceed with firewall changes. Diagnose connectivity first.
Read current firewall state:
- ufw servers: ssh root@<IP> "ufw status numbered"
- vps-i1 (firewalld): ssh root@217.154.82.162 "firewall-cmd --list-all"
Confirm recovery path: what is the fallback if SSH breaks?
- vps-i1 / vps-h1: IONOS / Hostinger web console (always available)
- bms-1 / bms-2 / bms-3 / bms-4: OVH KVM console (always available)
- lap1: CF Tunnel is the only access — loss = manual physical access needed

Common Tasks

1. DNS Change (Cloudflare)

Credentials: CF_API_TOKEN from secrets/monitoring.env.sops (delegate read to secret-manager if needed).
Zone ID: CF_ZONE_ID from secrets/monitoring.env.sops.

# Step 1 — Check current record (PowerShell, safe — no secret values)
$headers = @{Authorization = "Bearer $env:CF_API_TOKEN"; "Content-Type" = "application/json"}
$zone_id = $env:CF_ZONE_ID
$resp = Invoke-RestMethod -Uri "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records?name=<name>&type=A" `
  -Headers $headers -Method GET
$resp.result | Select-Object name, type, content, ttl, id
 
# Step 2 — Create or update record (example: A record)
$body = @{type="A"; name="<subdomain>.zintegrowana.online"; content="<IP>"; ttl=120; proxied=$false} | ConvertTo-Json
Invoke-RestMethod -Uri "https://api.cloudflare.com/client/v4/zones/$zone_id/dns_records" `
  -Headers $headers -Method POST -Body $body
 
# Step 3 — Verify propagation
Resolve-DnsName <subdomain>.zintegrowana.online -Type A -Server 1.1.1.1

When modifying existing records:

Reduce TTL to 120s at least 10 minutes before the change
After change: verify with dig or Resolve-DnsName (two independent resolvers)
Wait for TTL to expire before deleting old records

2. Firewall Rule Change

CRITICAL: always verify SSH connectivity before and after.

ufw (bms-1, bms-2, bms-3, bms-4, vps-h1)

# View current rules
ssh root@<IP> "ufw status numbered"
 
# Allow a port
ssh root@<IP> "ufw allow <PORT>/tcp && ufw reload && ufw status"
 
# Deny (block) an IP
ssh root@<IP> "ufw insert 1 deny from <ATTACKER_IP> to any && ufw reload"
 
# Remove a rule by number
ssh root@<IP> "ufw delete <RULE_NUMBER> && ufw reload"
 
# Verify SSH still works AFTER the change
ssh -o ConnectTimeout=5 -o BatchMode=yes root@<IP> "echo connectivity-ok"

firewalld (vps-i1)

# View current state
ssh root@217.154.82.162 "firewall-cmd --list-all"
 
# Open a port permanently
ssh root@217.154.82.162 "firewall-cmd --permanent --add-port=<PORT>/tcp && firewall-cmd --reload"
 
# Block an IP (rich rule)
ssh root@217.154.82.162 "firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=<IP> reject' && firewall-cmd --reload"
 
# Verify SSH connectivity after change
ssh -o ConnectTimeout=5 -o BatchMode=yes root@217.154.82.162 "echo connectivity-ok"

3. Caddy Vhost Add/Update

Location: /opt/p24-infra/monitoring/Caddyfile on vps-i1.
Restart: via docker compose restart caddy in /opt/p24-infra/monitoring.

# Step 1 — View current Caddyfile
ssh root@217.154.82.162 "cat /opt/p24-infra/monitoring/Caddyfile"
 
# Step 2 — Edit Caddyfile (use heredoc or scp from local)
# Example: add new vhost block
 
# Step 3 — Reload Caddy (graceful)
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose exec caddy caddy reload --config /etc/caddy/Caddyfile"
 
# Step 4 — Verify HTTPS
curl -I --max-time 10 https://<new-subdomain>.zintegrowana.online

If reload fails: docker compose restart caddy (full restart).
Caddy auto-provisions Let’s Encrypt certs via DNS challenge using CF_API_TOKEN.

4. CF Tunnel Update (cloudflared)

Context: cloudflared runs on lap1 (dev-laptop) and tunnels lap1-ssh.zintegrowana.online to SSH.

# Check cloudflared status on lap1 (via CF Tunnel itself — bootstrapping)
ssh -o StrictHostKeyChecking=no root@lap1-ssh.zintegrowana.online "systemctl status cloudflared"
 
# Restart cloudflared
ssh root@lap1-ssh.zintegrowana.online "systemctl restart cloudflared"
 
# Check tunnel in CF dashboard via API
$resp = Invoke-RestMethod -Uri "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/tunnels" `
  -Headers $headers -Method GET
$resp.result | Where-Object { $_.name -like "lap1*" } | Select-Object name, status

⚠️ Creating a CF Access Service Token — the create-call returns client_secret INLINE (#7056). POST /accounts/<ACCOUNT_ID>/access/service_tokens returns the client_secret exactly once, in the response body, at creation time. Never print or display that raw response — it leaks a live credential into your transcript (this is exactly what happened in #7056). Capture the response, extract only the secret field, hand it straight to the secret-manager path (write to the appropriate secrets/*.env.sops), and never echo it:
resp=$(curl -sf -X POST "https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/access/service_tokens" \
  -H "Authorization: Bearer $CF_API_TOKEN" -H "Content-Type: application/json" \
  -d '{"name":"<descriptive-name>"}')                # ALWAYS name the token — an empty name is a mistake
client_id=$(printf '%s' "$resp"     | jq -r '.result.client_id')      # non-secret — safe to log
client_secret=$(printf '%s' "$resp" | jq -r '.result.client_secret')  # SECRET — never echo
# → deliver $client_secret to secret-manager for SOPS storage, then: unset resp client_secret
Report back only non-secret identifiers (token id, name, created-at). Same discipline as reading an existing secret, extended to creation — see standards/common/secrets.md §Rules.

5. SSL Cert Check

# Check expiry date
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null | openssl x509 -noout -dates
 
# Check full chain
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null | openssl x509 -noout -issuer -subject
 
# Force Caddy ACME renewal (if cert is close to expiry)
ssh root@217.154.82.162 "cd /opt/p24-infra/monitoring && docker compose exec caddy caddy renew"

Alert threshold: warn when cert expires in < 30 days. Create GH issue with label infra if < 14 days.

Emergency Procedures

SSH Locked Out of a Server

Recovery paths (in priority order):

Server	Recovery
Any server (bms-1/2/3/4, vps-i1, vps-h1)	Try the CF Tunnel backup first — `ssh <label>-tunnel` (e.g. `ssh bms4-tunnel`), independent of the locked-out public IP. See `docs/playbooks/fleet-backup-ssh-cf-tunnel.md`. If that’s also down, fall through to the provider console below.
vps-i1	IONOS web console: cloud.ionos.de → Server → KVM Console
vps-h1	Hostinger web console: hpanel.hostinger.com → VPS → Access
bms-1/2/3/4	OVH Manager: ca.ovh.com → Bare Metal → KVM
lap1	Physical access (Berlin); or reboot machine — CF Tunnel starts automatically via systemd

Common SSH lockout causes:

ufw rule blocked port 22 — fix via console: ufw allow 22/tcp && ufw reload
sshd_config error after edit — fix via console: sshd -t; systemctl restart sshd
Wrong key in authorized_keys — add correct key via console

Prevention: always test SSH in a second terminal before closing the session that made the change.

DNS Propagation Stuck

# Check multiple resolvers
Resolve-DnsName <domain> -Type A -Server 1.1.1.1   # Cloudflare
Resolve-DnsName <domain> -Type A -Server 8.8.8.8   # Google
Resolve-DnsName <domain> -Type A -Server 9.9.9.9   # Quad9
 
# If CF API shows new value but DNS still serves old:
# 1. Check TTL — wait for TTL to expire (original TTL may be 1h)
# 2. Check if record is proxied (orange cloud) — CF caches aggressively
# 3. Force uncached lookup: dig @<AUTHORITATIVE_NS> <domain>

Role Boundaries — What to Hand Off

Task	Hand off to
Write/update SOPS secret	secret-manager
Restart Docker app containers	sys-admin
Security anomaly / pen-test	sys-security
Feature code change	dev-coder
vps-h1 new service beyond allowed scope	Human (create `human-action` GH issue)
MongoDB operations	sys-admin

References

Role file: C:\Users\konar\.claude\agent-prompts\roles\p24-network-admin.md
SSH connectivity: docs/playbooks/server-ssh-connectivity.md
SOPS key reference: docs/templates/network-admin-keys.template.env
Server IPs: CLAUDE.md §Server IPs — quick reference
Secret manager playbook: docs/playbooks/secret-manager.md

p24-infra Docs

Explorer

p24-network-admin