p24-infra Docs

bms1-nginx-proxy-missing-cert-symlinks

3 min read

Playbook: bms-1 nginx-proxy — missing top-level cert symlinks (DRAFT)

Status: DRAFT — written live during first occurrence, 2026-07-12.

Trigger

A vhost served by nginx-proxy on bms-1 presents the internal self-signed CN=letsencrypt-nginx-proxy-companion placeholder certificate instead of its real Let’s Encrypt certificate, even though nginx-proxy-letsencrypt logs show no renewal errors (e.g. Domains not changed. Skip, Next renewal time is: <future date>).

Root cause

nginx-proxy-letsencrypt (jrcs/letsencrypt-nginx-proxy-companion) issues/renews certs into /etc/nginx/certs/<domain>/{cert,chain,fullchain,key}.pem, then normally creates top-level convenience symlinks /etc/nginx/certs/<domain>.{crt,key,chain.pem,dhparam.pem} that nginx-proxy actually reads. nginx-proxy falls back to its self-signed placeholder ONLY when those top-level .crt/.key files are absent for a vhost — it does not look inside the per-domain subdirectory itself.

The symlink-creation step appears to run only as part of an actual issuance/renewal event, not on every companion loop cycle. If a vhost’s cert directory exists with valid PEM files but the top-level symlinks are missing (observed for multiple bms-1 vhosts: api.w4.pinbox24.com, git-deploy-api.w4.pinbox24.com, pdf-gen-api.w4.pinbox24.com, rapi.w4.pinbox24.com, s3-api.w4.pinbox24.com, s3-v2-api.w4.pinbox24.com — only mailgun-api.w4.pinbox24.com, most recently renewed Jul 7, had them), the symlinks were either never created or were lost (possibly related to the nginx-proxy jwilder→nginxproxy image swap on 2026-07-08, see docs/playbooks/nginx-proxy-upgrade-jwilder-to-nginxproxy.md — not confirmed, needs further investigation) and will not self-heal until the next real renewal event for that specific domain (which acme.sh will skip for months since it believes the cert is still fresh).

Confirm

# 1. Check what cert nginx is actually serving
echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
# BAD: subject=issuer=CN=letsencrypt-nginx-proxy-companion
 
# 2. Check the companion's renewal state (should show "skip", not an actual error)
ssh root@94.23.26.113 "docker logs nginx-proxy-letsencrypt --since 2h 2>&1 | grep -A3 'Creating/renewal <domain> certificates'"
 
# 3. Compare top-level symlinks against a known-working sibling vhost
ssh root@94.23.26.113 "docker exec nginx-proxy-letsencrypt ls -la /etc/nginx/certs/ | grep <domain>"
# BAD: only a directory entry, no <domain>.crt / <domain>.key / <domain>.chain.pem symlinks
 
# 4. Confirm the real PEM files exist and are valid
ssh root@94.23.26.113 "docker exec nginx-proxy-letsencrypt sh -c 'ls -la /etc/nginx/certs/<domain>/; openssl x509 -in /etc/nginx/certs/<domain>/cert.pem -noout -dates'"

Fix — recreate the missing symlinks (safe, additive, no data touched)

# PLAYBOOK: docs/playbooks/bms1-nginx-proxy-missing-cert-symlinks.md
ssh root@94.23.26.113 "docker exec nginx-proxy-letsencrypt sh -c '
cd /etc/nginx/certs &&
ln -sf ./<domain>/fullchain.pem <domain>.crt &&
ln -sf ./<domain>/key.pem <domain>.key &&
ln -sf ./<domain>/chain.pem <domain>.chain.pem &&
ln -sf ./dhparam.pem <domain>.dhparam.pem &&
ls -la <domain>.crt <domain>.key <domain>.chain.pem <domain>.dhparam.pem
'"
 
# nginx-proxy watches the certs volume via docker-gen and reloads automatically on change.
# If it doesn't pick it up within ~10s, force a reload:
ssh root@94.23.26.113 "docker exec nginx-proxy nginx -s reload"

Verify

echo | openssl s_client -connect <domain>:443 -servername <domain> 2>/dev/null | openssl x509 -noout -subject -issuer -dates
# Expected: issuer contains "Let's Encrypt", subject CN=<domain>, notAfter ~90 days out

Follow-up (not yet done)

  • Audit all bms-1 vhosts for the same missing-symlink pattern (list above from the 2026-07-12 incident) — several may be legacy/unused, but worth confirming which are still live traffic.
  • Root-cause why the companion’s symlink step didn’t fire for these domains — check jrcs/letsencrypt-nginx-proxy-companion’s create_links.sh behavior and whether the 2026-07-08 nginx-proxy image swap (bind-mount based, should have been non-destructive to /etc/nginx/certs) is actually implicated, or if this predates that swap entirely.
  • Consider a periodic health check (blackbox exporter probe checking cert issuer, not just HTTPS reachability) so a placeholder-cert regression like this alerts instead of silently breaking non-browser clients.
  • #3879 — first occurrence, api.w4.pinbox24.com
  • docs/playbooks/nginx-proxy-upgrade-jwilder-to-nginxproxy.md
  • docs/playbooks/p24-network-admin.md

Graph View