p24-infra Docs

pinbox24-bms1-redis-wrongpass

4 min read

Playbook: Pinbox24 bms-1 Redis WRONGPASS / PM2 restart loop

Trigger: v42-prod (w4.pinbox24.com) shows elevated PM2 restarts in Grafana, WRONGPASS errors in PM2 logs, backend returns 500/login failures.

Confirm the problem

# From any session with bms-1 access (Tailscale: ssh -J root@217.154.82.162 root@100.118.64.105)
docker exec v42-prod sh -c "pm2 list" | grep -E "restarts|WRONGPASS|error"
docker exec v42-prod sh -c "pm2 logs --lines 20" 2>&1 | grep -i "WRONGPASS\|redis\|auth"

Key metric: restarts column in pm2 list — any value > 0 is a regression (w4 was stable at 0 restarts before).

Root cause history

2026-07-05: Someone edited /home/p24-server-scripts/v4/v42/backend-environment.env manually using nano at 15:29 UTC. They added REDIS_HOST, REDIS_PASSWORD, and REDIS_PORT keys with a wrong password value (10 chars) instead of the correct 28-char value from SOPS.

Result: v42-prod crashed 54 times with WRONGPASS invalid username-password pair or user is disabled.

Fix procedure

Step 1 — Identify the correct password

The correct Redis password lives in secrets/pinbox24-w4.env.sops, key V42_REDIS_PASSWORD. The running env file uses the same key stripped of the V42_ prefix — i.e., REDIS_PASSWORD.

# Windows dev machine (p24-infra session only)
$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$env:REDIS_PASS = (sops --decrypt --input-type dotenv --output-type dotenv `
    C:\code_2026\p24-infra\secrets\pinbox24-w4.env.sops | `
    Select-String "^V42_REDIS_PASSWORD=").ToString().Split("=",2)[1].Trim()
Write-Host "Redis password loaded: $($env:REDIS_PASS.Length) chars"  # expect 28

Step 2 — Check current env file on bms-1

# On bms-1 (via jump host)
ssh -J root@217.154.82.162 root@100.118.64.105 \
    "grep -E 'REDIS_' /home/p24-server-scripts/v4/v42/backend-environment.env"

Note: only shows key names + values. If REDIS_PASSWORD length != 28 chars, it’s wrong.

# Count chars (safe — doesn't echo value)
ssh -J root@217.154.82.162 root@100.118.64.105 \
    "grep '^REDIS_PASSWORD=' /home/p24-server-scripts/v4/v42/backend-environment.env | cut -d= -f2- | wc -c"
# Expect: 29 (28 chars + newline)

Step 3 — Fix the env file (safe method — never echoes value)

Use Python stdin pipe to update the password line without it appearing in shell history or logs:

# Windows — build the Python update command, never echo the value
$envFile = "/home/p24-server-scripts/v4/v42/backend-environment.env"
$pythonScript = @"
import sys, re
content = open('$envFile').read()
new_pass = sys.stdin.readline().rstrip('\n')
updated = re.sub(r'^REDIS_PASSWORD=.*$', f'REDIS_PASSWORD={new_pass}', content, flags=re.MULTILINE)
open('$envFile', 'w').write(updated)
print('done')
"@
 
# Pipe password via stdin — it never appears in SSH command or shell log
$env:REDIS_PASS | ssh -J root@217.154.82.162 root@100.118.64.105 "python3 -c '$pythonScript'"

Verify (length check, not value):

ssh -J root@217.154.82.162 root@100.118.64.105 \
    "grep '^REDIS_PASSWORD=' /home/p24-server-scripts/v4/v42/backend-environment.env | cut -d= -f2- | wc -c"
# Expect: 29

Step 4 — Force-recreate the container

The env file change only takes effect after a full container stop + start. Simple restart is not enough.

ssh -J root@217.154.82.162 root@100.118.64.105 "
cd /home/p24-server-scripts/v4/v42
 
# If docker-compose can't stop v42-prod (started outside compose project), do it manually:
docker stop v42-prod 2>/dev/null || true
docker rm v42-prod 2>/dev/null || true
 
# Start fresh via compose (picks up updated env file)
docker-compose up -d backend
sleep 10
docker exec v42-prod sh -c 'pm2 list'
"

Step 5 — Verify recovery

ssh -J root@217.154.82.162 root@100.118.64.105 "
docker exec v42-prod sh -c 'pm2 list'
# restarts column must be 0 (or at least not growing)
 
docker exec v42-prod sh -c 'pm2 logs --lines 10 2>&1 | grep -i redis'
# No WRONGPASS lines
"

End-to-end test: curl -sf https://w4.pinbox24.com/api/health should return HTTP 200.

Cleanup env

$env:REDIS_PASS = ""

Prevention

  1. Never use nano / vim directly on backend-environment.env — it’s the running production config. Changes go through SOPS: update secrets/pinbox24-w4.env.sops, then run secrets-sync to bms-1.

  2. Monitor WRONGPASS errors: pinbox24_pm2_restarts_total Prometheus metric + Pinbox24BackendRestartLoop Grafana alert. Alert fires when restarts > threshold in 5min window.

  3. Backup before manual edits: always cp file file.bak.$(date +%Y%m%d%H%M%S) first. On 2026-07-05, a backup existed at backend-environment.env.bak.20260705152933 which helped confirm when the edit occurred.

  4. Related issue (#2855): long-term fix is wiring SOPS into the bms-1 deploy pipeline so SOPS is the single source of truth for all container env vars.

  • pinbox24-bms1-operations.md — general bms-1 operations
  • secrets/pinbox24-w4.env.sops — source of truth for V42_REDIS_PASSWORD
  • docs/priorities.md — issue #2855 (SOPS deploy pipeline for bms-1)

Graph View

Backlinks