p24-infra Docs

pinbox24-container-runtime-hotfix

4 min read

Playbook: Pinbox24 Container Runtime Hotfix (bms-1)

Last updated: 2026-07-02 Applies to: bms-1 (94.23.26.113) — v42-prod and microservice containers Trigger: Non-fatal bug in compiled JS inside a running container that cannot be fixed via a full GitLab CI redeploy (ECR token expired, runner broken, or fix is urgent).

When to use this playbook

  • A TypeError or logic error in compiled JS that logs on startup but does not crash the container
  • The fix is a small null-guard or format correction in a single file
  • A full image rebuild + redeploy would take hours or is blocked by GitLab runner issues
  • The container is v42-prod, s3-v2-v42-prod, or a similar Pinbox24 microservice on bms-1

Not for: changes that require new NPM packages, schema changes, or container restarts (use docs/playbooks/pinbox24-bms1-manual-deploy.md for those).

Approach

  1. Read the compiled JS from inside the container via docker exec cat
  2. Apply a string-replace patch in a Python script
  3. Write the patched file back via docker exec -i sh -c 'cat > file'
  4. Reload the affected PM2 process (not a container restart)
  5. Verify the error no longer appears in PM2 logs

Step-by-step

1. Inspect the target file

# On bms-1 (via SSH)
docker exec v42-prod cat /app/dist/apps/<module>/<file>.helper.js | head -100

2. Write the Python hotfix script

Follow the pattern in docs/pinbox24/hotfix-w4-s3-response-format.sh (Python):

  • Check if already patched (idempotent)
  • Verify the target string exists; print context if not found
  • content.replace(old, new) — single exact-string replacement
  • Write back via docker exec -i sh -c 'cat > /app/dist/apps/...'
  • pm2 reload <process-name> inside the container

3. Deploy and run the hotfix script

From the local Windows dev machine (PowerShell):

# Copy script to bms-1
scp -i C:\Users\konar\.ssh\id_ed25519 docs\pinbox24\hotfix-<name>.py root@94.23.26.113:/tmp/ # PLAYBOOK: pinbox24-container-runtime-hotfix.md
 
# Run it
$out = ssh -i C:\Users\konar\.ssh\id_ed25519 -o BatchMode=yes -o LogLevel=ERROR root@94.23.26.113 "python3 /tmp/hotfix-<name>.py" 2>&1 # PLAYBOOK: pinbox24-container-runtime-hotfix.md
if ($LASTEXITCODE -ne 0) { Write-Host "FAILED: $out" } else { Write-Host "OK: $($out[-3..-1] -join '; ')" }

4. Verify

# Check PM2 logs — the TypeError line should not appear after reload
$logs = ssh ... root@94.23.26.113 "docker exec v42-prod pm2 logs v42-prod_backend --lines 30 --nostream" 2>&1 # PLAYBOOK: pinbox24-container-runtime-hotfix.md
Write-Host $logs | Select-String "officeCron|TypeError|Error while making"

PM2 process names in v42-prod

PM2 processSource app
v42-prod_backendMain backend (officeCron, routes, sockets)

Confirm with: docker exec v42-prod pm2 list

Durability warning

This hotfix patches the compiled JS inside the running container only. It is lost when:

  • The container is recreated (docker compose up -d pulls the ECR image)
  • The server reboots and Docker auto-restarts the container

After applying: update the GitLab issue / GH issue with “hotfix applied, needs permanent fix via image rebuild” and link this playbook.

Permanent fix: the null guard must be merged into the TypeScript source on GitLab (gitlab.com/pinbox24/ — access via GITLAB_ADMIN_PAT in secrets/administration.env.sops) and a new image deployed via docs/playbooks/pinbox24-bms1-manual-deploy.md.

Escalation

If pm2 reload fails or the container crashes after patching:

  1. docker exec v42-prod pm2 logs v42-prod_backend --lines 50 --nostream
  2. If container is in restart loop: docker restart v42-prod (reverts to original image — safe rollback)
  3. Check docker ps to confirm container came back up
  4. Create GH issue in radieu/p24-infra with label bug, pinbox24

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="deploy",
    resource="pinbox24-bms1",
    result="success",  # "success" | "failed" | "skipped"
    detail="Pinbox24 container runtime hotfix applied on bms-1",
    env="bms-1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'deploy', 'pinbox24-bms1', 'success', 'Pinbox24 container runtime hotfix applied on bms-1', 'bms-1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks