p24-infra Docs

pinbox24-nginx-emergency-path-block

13 min read

Playbook: Pinbox24 nginx-proxy — Emergency Exact-Path Block (bms-1)

Status: REUSABLE RUNBOOK — the #5093 block itself was removed 2026-08-02 once the GitLab code fix (pinbox24/p24-v-3.2!73, commit ae638913) shipped and was verified live (see #5093 closing comment). This document stays active as the general procedure for the next exact-path emergency block on this host. Server: bms-1 (94.23.26.113) — Pinbox24 production (P0 — handle with care) Container: nginx-proxy (nginxproxy/nginx-proxy:latest) First applied: 2026-08-02 (issue #5093 — W3 GET /api/profile/ unscoped MD5-password-hash dump) Mechanism corrected: 2026-08-02, after PR #5109’s rewrite was rejected on review — see “How the vhost.d/<host> include actually works” below for the evidence-based correction.

Verification results (2026-08-02, with a fresh JWT for W3_TEST_USER_EMAIL)

RequestBefore fixAfter fix
GET /api/profile/200, ~4.4 MB (unscoped array of all users’ login+pass)403
GET /api/profile (no trailing slash)200, ~4.4 MB — same handler as above403
GET /api/profile/contactData200, 190 bytes (own data only)200, 190 bytes (unchanged)
POST /api/auth (login)200 with JWT200 with JWT (unchanged)
GET /api/i18n/langs (public)200200 (unchanged)

nginx -t passed (only pre-existing harmless ssl_stapling OCSP warnings across all certs on this host). nginx -s reload completed gracefully, no dropped connections.

When to use this playbook

An actively-exploitable route needs to be blocked at the nginx layer immediately, scoped to one exact path on one exact vhost, while the real code fix goes through GitLab MR review (human-merge-only, see docs/w3-w4-stack-operations.md). This is a stopgap, not a permanent fix — remove the block once the upstream code fix ships.

Why NOT the two existing nginx-proxy patterns for the initial, fastest block

Existing patternWhy it doesn’t fit an immediate exact-path block
conf.d/*.conf global snippet (pinbox24-nginx-client-max-body-size.md)Only valid for http-context directives (e.g. client_max_body_size). location blocks are illegal outside a server {} context — cannot be added via a bare conf.d/*.conf file.
vhost.d/<host> per-vhost overrideWorks, but is not instant — it requires a docker-gen regeneration cycle to take effect (see the corrected explanation below). It is the durable long-term location once you have a minute to harden the block; it is not the fastest first move under active-exploit pressure, which is why the emergency Fix section below still edits default.conf directly.
vhost.d/default (shared)Wired into all 13 vhosts — a rule here leaks scope to every other vhost on bms-1 unless gated by $host, which is fragile to improvise safely. It is also companion-owned (## Start/End of configuration add by letsencrypt container markers) — nginx-proxy-letsencrypt may rewrite it on ACME renewal, silently dropping any manually-added directive. Confirmed on 2026-08-02: the file’s entire content is currently just the ACME challenge block.

How the vhost.d/<host> include actually works (corrected 2026-08-02)

The version of this playbook first applied during #5093 recorded vhost.d/<host> as “confirmed dead on this host” — that finding is wrong and is corrected here with direct evidence, after a later PR (#5109) tried to flip the conclusion outright and was rejected on review for asserting the opposite without evidence. This section documents what was actually verified, so a third correction shouldn’t be needed.

The include is host-aware, but it is resolved once, at docker-gen render time — not per-request. nginx-proxy’s template (/app/nginx.tmpl inside the nginx-proxy container) contains:

{{- if (exists (printf "/etc/nginx/vhost.d/%s" $vhostFileName)) }}
include /etc/nginx/vhost.d/{{ $vhostFileName }};
{{- else if (exists "/etc/nginx/vhost.d/default") }}
include /etc/nginx/vhost.d/default;
{{- end }}

docker-gen runs this template — and writes the literal include line into /etc/nginx/conf.d/default.conf — only when it (re)generates the file, which happens on a watched Docker event (connect/disconnect for containers with VIRTUAL_HOST set) or a manual trigger (see Hardening section below). Whichever branch was true at that render moment is baked into default.conf as plain text until the next regeneration. Consequences:

  • If /etc/nginx/vhost.d/<host> does not exist yet when docker-gen last rendered, the server block’s include line points at vhost.d/default — creating the file afterwards has no effect until the next render. This is what the original 2026-08-02 finding actually observed (checked before the file existed), and why it looked “dead.”
  • If /etc/nginx/vhost.d/<host> does exist at render time, the include line points directly at that host’s file. Verified directly on bms-1 after a later remediation pass in the same incident created /docker/vol/etc/nginx/vhost.d/api.w3.pinbox24.com and forced a regen: default.conf line 219 read include /etc/nginx/vhost.d/api.w3.pinbox24.com; for that one server block, while the other 12 untouched vhosts still read include /etc/nginx/vhost.d/default;. nginx -t passed and the block served 403 from that file. Confirmed independently by re-reading the shipped nginx.tmpl template on the running container.
  • The “existing stub file vhost.d/mailgun-v42-prod sits unused” observation in the original finding is not evidence either way — that filename does not match any real server_name on this host (the actual mailgun vhost is mailgun-api.w4.pinbox24.com), so docker-gen was never going to look for a file called mailgun-v42-prod regardless of whether the mechanism works. It was a mislabeled/orphaned artifact, not a test of the include logic. Don’t use it as evidence for or against this mechanism again.

Bottom line: vhost.d/<host> is a real, working, per-host include point on this host, but it is eventually consistent with docker-gen’s render cycle, not instant like a direct default.conf edit + nginx -s reload. Use default.conf for the first, fastest block (below); promote it to vhost.d/<host> once you can verify the regen actually happened (Hardening section below) if you want it to survive a docker-gen regeneration.

Chosen approach — direct edit of conf.d/default.conf, scoped to one server block

/etc/nginx/conf.d/default.conf is bind-mounted at /docker/vol/etc/nginx/conf.d/default.conf on the host. Insert two location = (exact match) blocks directly inside the target vhost’s server { listen 443 ssl; server_name <host>; ... } block, just before its closing }. Exact-match location = always wins over the vhost’s location / proxy block regardless of declaration order or position, so this is safe to append at the end.

This does NOT require any docker-gen regeneration — nginx -s reload re-parses default.conf fresh from disk, so a manual edit takes effect on reload exactly like any other nginx config change.

⚠️ Persistence risk — READ BEFORE RELYING ON THIS LONG-TERM

default.conf is docker-gen-owned and gets fully regenerated from its template on any relevant Docker event (container start/stop/create) for containers with VIRTUAL_HOST set. bms-1 is a busy host (24+ containers, active CI/CD) — the file’s mtime was already Aug 2 07:42 (same-day) when this playbook was first applied, confirming regen happens often. A future regen will silently drop this manual location block, reopening the vulnerability at the nginx layer (the code-level bug remains present until the GitLab fix ships either way).

Mitigation until the GitLab fix lands:

  1. Re-run the “Verify” section below periodically (daily, or after any known bms-1 container deploy/restart) and re-apply this playbook if the block reverts to 200.
  2. Promote the block to vhost.d/<host> (see “Hardening” section below) — that location does survive a docker-gen regeneration, once you’ve confirmed the regen actually picked it up. This is the recommended step as soon as the initial default.conf edit is verified working and you have a moment to spare.
  3. Recommended fast-follow (not yet implemented, out of scope for the nginx-only emergency response): a blackbox/Prometheus probe hitting the blocked path with a throwaway/invalid token and alerting if the response ever stops being 403/401.
  4. The durable fix is the GitLab MR (source-level auth scoping) — this nginx block should be removed once that ships and is verified, not left in place indefinitely (stale security-relevant nginx rules are themselves a maintenance hazard).

Fix

# On bms-1 as root. Path is bind-mounted -- editable directly on the host, no docker exec needed.
python3 - <<'PYEOF'
path = '/docker/vol/etc/nginx/conf.d/default.conf'
with open(path) as f:
    content = f.read()
 
# Anchor: unique to the api.w3.pinbox24.com :443 server block (the :80 redirect block
# has no ssl_certificate directive, so this anchor cannot match the wrong block).
anchor = 'ssl_certificate /etc/nginx/certs/api.w3.pinbox24.com.crt;'
idx = content.index(anchor)
start = content.rfind('server {', 0, idx)
 
# Brace-depth walk from the server block's opening '{' to its matching closing '}'.
depth = 0
i = start
block_end = None
while i < len(content):
    c = content[i]
    if c == '{':
        depth += 1
    elif c == '}':
        depth -= 1
        if depth == 0:
            block_end = i
            break
    i += 1
assert block_end is not None
 
insertion = (
    "    # Issue #5093 (P0) -- emergency block, unscoped profile dump. "
    "See docs/playbooks/pinbox24-nginx-emergency-path-block.md. Remove after GitLab fix ships.\n"
    "    location = /api/profile/ {\n"
    "        return 403;\n"
    "    }\n"
    "    location = /api/profile {\n"
    "        return 403;\n"
    "    }\n"
)
 
new_content = content[:block_end] + insertion + content[block_end:]
 
with open(path + '.bak-5093', 'w') as f:
    f.write(content)          # backup of the pre-edit file
with open(path, 'w') as f:
    f.write(new_content)
 
print("edited, block_end was at byte", block_end)
PYEOF
 
# Validate BEFORE reloading
docker exec nginx-proxy nginx -t
 
# Graceful reload (no downtime, no dropped connections)
docker exec nginx-proxy nginx -s reload

Hardening (optional, recommended once the emergency block is verified) — promote to vhost.d/<host>

Do this once the immediate default.conf block above is confirmed working (403s verified) and you have a moment — it makes the block survive the next docker-gen regeneration instead of being silently dropped by it (see persistence risk above).

# On bms-1 as root. Same bind-mount, different path -- no docker exec needed to write it.
cat > /docker/vol/etc/nginx/vhost.d/api.w3.pinbox24.com <<'EOF'
# Issue #5093 (P0) -- emergency block, unscoped profile dump, promoted to durable vhost.d
# include. See docs/playbooks/pinbox24-nginx-emergency-path-block.md. Remove after GitLab fix ships.
location = /api/profile/ {
    return 403;
}
location = /api/profile {
    return 403;
}
EOF

Force docker-gen to regenerate default.conf now — do NOT wait for the next Docker event, and do NOT restart or signal the nginx-proxy container itself:

# CORRECT -- signals docker-gen (PID of the `docker-gen` process inside the container), which
# is launched with -notify "nginx -s reload", so a forced regen also triggers a graceful reload
# automatically. No container restart, no dropped connections.
docker exec nginx-proxy sh -c 'kill -HUP $(pidof docker-gen)'

⚠️ Do NOT run docker kill -s HUP nginx-proxy (or any signal targeted at the container itself/PID 1). Inside this image PID 1 is forego (a process supervisor running both docker-gen and nginx per its Procfile), not nginx and not docker-gen. Sending SIGHUP to forego does not behave like a graceful nginx reload — during the #5093 response this exact command stopped the whole container, taking every bms-1 vhost down for ~15 seconds. Always target the docker-gen process specifically, as shown above.

Verify the include actually switched — this is the step the original hardening pass skipped verifying explicitly; don’t assume the regen picked up the file:

docker exec nginx-proxy grep -n 'include /etc/nginx/vhost.d/api.w3.pinbox24.com;' \
  /etc/nginx/conf.d/default.conf
# Expect exactly one match, inside the api.w3.pinbox24.com :443 server block.
# If it prints nothing, the regen has not happened yet -- retry the kill -HUP step, or fall
# back to re-confirming the direct default.conf edit (above) is still in place.

Then re-run the nginx -t / verify steps below against the live host to confirm the block still returns 403 after the switch.

Verify

# Confirm the two blocks are present -- check whichever file is currently the live source.
# After the emergency Fix step, that's conf.d/default.conf:
docker exec nginx-proxy grep -n -A2 'location = /api/profile' /etc/nginx/conf.d/default.conf
 
# After the Hardening step (promoted to vhost.d/<host>), default.conf will only have the
# `include vhost.d/api.w3.pinbox24.com;` line -- check the actual rule there instead:
docker exec nginx-proxy grep -n -A2 'location = /api/profile' /etc/nginx/vhost.d/api.w3.pinbox24.com

From a client with a fresh valid JWT (W3_TEST_USER_EMAIL/W3_TEST_USER_PASSWORD, client-side MD5 hash, POST /api/auth):

GET https://api.w3.pinbox24.com/api/profile/          -> expect 403 (was 200, ~4.4MB)
GET https://api.w3.pinbox24.com/api/profile           -> expect 403 (was 200, ~4.4MB -- same handler)
GET https://api.w3.pinbox24.com/api/profile/contactData -> expect 200, caller's own data only (unaffected)

Also spot-check that unrelated vhosts on the same host are unaffected (e.g. api.w4.pinbox24.com health/login) since this is a same-file edit on a shared config.

Rollback / Removal

Check which location is actually live before touching anything — depending on whether the Hardening step above was ever run, the block may still be the raw default.conf edit, or it may have been promoted to vhost.d/<host> (in which case default.conf itself won’t contain the location = rules at all — only the include vhost.d/<host>; line). Don’t assume the task brief’s stated location is still accurate; verify on the live server first:

docker exec nginx-proxy grep -c 'location = /api/profile' /etc/nginx/conf.d/default.conf
docker exec nginx-proxy grep -c 'location = /api/profile' /etc/nginx/vhost.d/api.w3.pinbox24.com 2>/dev/null

If the block is in default.conf (bad-edit correction, or the Hardening step was never run):

cp /docker/vol/etc/nginx/conf.d/default.conf.bak-5093 /docker/vol/etc/nginx/conf.d/default.conf
docker exec nginx-proxy nginx -t && docker exec nginx-proxy nginx -s reload

Note: a docker-gen regen after this playbook runs will also “roll back” this file’s contents (see persistence risk above) — this manual rollback is for correcting a bad edit, not the expected long-term removal path.

If the block was promoted to vhost.d/<host> (durable path, the state actually found live on bms-1 at #5093 closure — confirmed 2026-08-02): edit that file directly, not conf.d/default.conf. Strip just the two location = blocks and the # Issue #5093 comment, leaving any pre-existing content (e.g. the letsencrypt ACME-challenge block) untouched, then:

docker exec nginx-proxy nginx -t && docker exec nginx-proxy nginx -s reload

This does not require forcing a docker-gen regen — nginx -s reload re-parses every included file fresh from disk, same as the emergency-Fix path. The include line in default.conf doesn’t need to change; only the content of the file it points at does.

  • docs/playbooks/pinbox24-nginx-client-max-body-size.md — the conf.d/*.conf global-snippet pattern this playbook explicitly does NOT use, and why
  • docs/playbooks/pinbox24-nginx-roundrobin-s3.md — prior precedent for nginx -t + nginx -s reload graceful-reload pattern and python-via-ssh-stdin edit technique on this same host
  • docs/playbooks/nginx-proxy-upgrade-jwilder-to-nginxproxy.md — nginx-proxy bind-mount layout on bms-1
  • docs/playbooks/w4-auth-password-model.md — verbatim-MD5-compare auth model (why the leaked hash is a working credential)
  • Issue #5093 — the P0 vulnerability this playbook was written to stop-gap; its closing comment recorded the on-server ground truth that the vhost.d/<host> include point is real for this vhost, correcting the “confirmed dead” finding this doc originally shipped with
  • PR #5109 — an earlier attempt to rewrite this playbook around the vhost.d/<host> mechanism. Rejected on review: it reversed the “confirmed dead” finding with no verification evidence, contained a self-contradiction (a stale paragraph claiming docker kill -s HUP nginx-proxy “does not stop the container”, directly opposite its own Step 3 warning), and its removal step executed exactly the forbidden signal command. The direction (documenting the durable vhost.d path) was right; the execution wasn’t safe to merge as-is. This revision keeps the original emergency default.conf procedure as the fast path, adds the vhost.d promotion as an explicit optional Hardening step with the corrected docker-gen-targeted signal, and cites concrete on-server evidence (the rendered nginx.tmpl conditional, and the live include line observed on bms-1) rather than asserting either conclusion outright.

Graph View