Playbook: Pinbox24 bms-1 Manual Deploy (GitLab CI broken)

Last updated: 2026-06-29 Applies to: bms-1 (94.23.26.113) — Pinbox24 production and staging microservices Services: p24-ms-mailgun, pinbox24-ms-s3-v2

Trigger

Use this playbook when GitLab CI/CD cannot deploy Pinbox24 microservices on bms-1 because:

The GitLab runner logs forbidden errors continuously and picks up no jobs
A hotfix or urgent deploy must land before the runner is repaired
The GitLab runner version is too old to authenticate with GitLab.com

Symptoms

journalctl -u gitlab-runner -n 50
# Expected bad output:
# FATAL: Runner forbidden  runner=wNhJ5wS_
# (repeated all day, no jobs executed)

Additional indicators:

gitlab-runner status shows the service running but no jobs processed
All 12 runners registered on bms-1 are orphaned from old/deleted GitLab projects
concurrent = 1 in /etc/gitlab-runner/config.toml means only one job at a time anyway

Diagnosis

SSH into bms-1 as root:

ssh root@94.23.26.113
 
# Check runner service
gitlab-runner status
journalctl -u gitlab-runner -n 50
 
# Check registered runners and their tokens
cat /etc/gitlab-runner/config.toml | grep -E "(token|url|name|tag)"

Root causes found 2026-06-29:

Stale runner token wNhJ5wS_ — runner was deleted from GitLab.com UI but block remained in config.toml
gitlab-runner version 13.8.0 — too old for current GitLab.com API, returns forbidden even with valid tokens

Manual Deploy Steps

1. Locate build directories on bms-1

# s3-v2
ls /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2/
 
# mailgun
ls /root/builds/-X_FwMVU/0/pinbox24/p24-ms-mailgun/

2. Fix git remote (HTTPS does not work — bms-1 has no valid SSL client cert for gitlab.com)

cd /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2
git remote set-url origin git@gitlab.com:pinbox24/pinbox24-ms-s3-v2.git
 
cd /root/builds/-X_FwMVU/0/pinbox24/p24-ms-mailgun
git remote set-url origin git@gitlab.com:pinbox24/p24-ms-mailgun.git

3. Pull latest code

cd <build-dir>
git stash                          # preserve any local runner-generated files
git checkout development
git pull origin development

4. Build Docker image locally

# s3-v2
cd /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2
docker build -t s3-v2-v42 .
 
# mailgun
cd /root/builds/-X_FwMVU/0/pinbox24/p24-ms-mailgun
docker build -t p24-ms-mailgun .

Note: The s3-v2 Dockerfile uses node:10.16.3-alpine. See the TypeScript gotchas section below if the build fails.

5. Deploy staging (ISOLATED directory — critical, see Gotchas)

mkdir -p /root/s3v2-stage
cp /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2/docker-compose.yml /root/s3v2-stage/
cp /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2/docker-deploy-stage.sh /root/s3v2-stage/
 
cd /root/s3v2-stage
IMAGE_NAME=s3-v2-v42-stage CLUSTER_NETWORK=test-net CONTAINER_NAME=s3-v2-v42-stage \
  bash docker-deploy-stage.sh \
    s3-v2-v42-stage \
    s3-v2-v42-stage \
    s3-v2-test.dev.pinbox24.com \
    v41-stage.dev.pinbox24.com

6. Deploy production

ECR token is typically expired during manual deploys — Docker pull fails. Use the locally built image instead:

cd /root/builds/JsHnVmmJ/0/pinbox24/pinbox24-ms-s3-v2
 
# docker-compose.yml references the image by name — it will use local cache
bash docker-deploy-prod.sh

For mailgun production restart (no code change, just restart):

cd /root/builds/-X_FwMVU/0/pinbox24/p24-ms-mailgun
docker-compose up -d

Gotchas

docker-compose staging/prod collision (CRITICAL)

docker-compose.yml defines a single service named s3-v2 with container_name: $CONTAINER_NAME. Running the staging deploy script from the same directory as the production deploy will stop the production container because docker-compose reconciles by service name.

Rule: Always copy docker-compose.yml and the stage deploy script to an isolated directory (e.g. /root/s3v2-stage/) before running staging deploys. Never run staging from the build directory that also runs production.

TypeScript build errors on Node 10

The s3-v2 Dockerfile pins node:10.16.3-alpine. Code written for Node 14+ will fail to compile. Fixes applied 2026-06-29:

Error	Fix
`import fsp from "fs/promises"` — module not found	`import { promises as fsp } from "fs"`
`return res.status(500).json(...)` inside `for..of` — type mismatch	Split into two statements: assign result, then `return`
`catch (e)` — eslint `id-length` violation	Rename to `catch (err)`
Upload function exceeds `max-statements: 25` eslint rule	Add `// eslint-disable-next-line max-statements` above function

nginx-proxy routing (test-net)

Staging containers are routed by jwilder/nginx-proxy on the test-net Docker network. Requirements:

Container must be on the test-net network
Container must have VIRTUAL_HOST env var set to the target domain
SSL certificate for new domains takes a few minutes (Let’s Encrypt via nginx-proxy-letsencrypt)

Verify routing after deploy:

docker exec nginx-proxy cat /etc/nginx/conf.d/default.conf | grep -A5 's3-v2-test.dev.pinbox24.com'

ECR token expiry

GitLab CI normally refreshes ECR credentials before each pull. During manual deploys the cached token is expired. Always build the image locally with docker build first, then run the deploy script — docker-compose will use the local image cache without pulling.

Health Checks

# s3-v2 production
curl -I https://s3-v2-api.w4.pinbox24.com/
 
# s3-v2 staging (from bms-1, bypassing DNS)
curl -H "Host: s3-v2-test.dev.pinbox24.com" http://localhost/
 
# mailgun production
docker ps | grep mailgun

Expected: s3-v2 prod returns HTTP/2 200. Mailgun container shows Up X minutes.

Runner Fix (permanent)

Do this after the emergency deploy to restore CI/CD:

1. Update gitlab-runner to current version

curl -L https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.rpm.sh | bash
yum update gitlab-runner
gitlab-runner --version

2. Clean config.toml — remove orphaned runner blocks

Edit /etc/gitlab-runner/config.toml:

Remove every [[runners]] block whose token corresponds to a deleted GitLab runner
Keep only runners that exist in the GitLab project’s CI/CD Settings → Runners page
Set concurrent = 4 (was 1 — allows parallel jobs)

concurrent = 4
check_interval = 0

3. Re-register active runner

In GitLab UI: Project → Settings → CI/CD → Runners → New project runner. Generate a registration token, then on bms-1:

gitlab-runner register \
  --url https://gitlab.com/ \
  --registration-token <TOKEN-FROM-GITLAB-UI> \
  --executor shell \
  --description "bms-1-prod" \
  --tag-list "autodeploy,prod-deploy-p24" \
  --non-interactive

4. Restart and verify

systemctl restart gitlab-runner
journalctl -u gitlab-runner -f
# Watch for: "Checking for jobs... nothing" (healthy) vs "forbidden" (stale token still present)

Prevention

Risk	Prevention
Runner goes stale silently	Add Prometheus alert on `gitlab_runner_jobs_total` stagnant for > 4h
Node 10 build surprises	Upgrade Dockerfile to `node:18-alpine`; track in issue backlog
ECR token expiry during manual ops	Document local-build fallback in CI scripts; add note to CI README
Staging kills prod containers	Enforce isolated working directory in `docker-deploy-stage.sh` via path check at script top
Single `concurrent = 1` bottleneck	Set `concurrent = 4` and add a monitoring check

Escalation

If manual deploy fails and health check does not pass:

Check docker logs <container> --tail 50 for runtime errors
Check docker ps -a for containers in Exited state
Verify the test-net network exists: docker network ls | grep test-net
If nginx-proxy is missing the route after 5 minutes, restart it: docker restart nginx-proxy
Create a GH issue in radieu/p24-infra with label bug and pinbox24 before attempting further changes

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="deploy",
    resource="pinbox24-bms1",
    result="success",  # "success" | "failed" | "skipped"
    detail="Pinbox24 manual deployment executed on bms-1",
    env="bms-1",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'deploy', 'pinbox24-bms1', 'success', 'Pinbox24 manual deployment executed on bms-1', 'bms-1')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

p24-infra Docs

Explorer

pinbox24-bms1-manual-deploy