p24-infra Docs

mongodb-exporter-uri-special-chars

3 min read

Playbook: mongodb_exporter URI fails — special chars in password

Trigger

mongodb_up{cluster_role="mongod"} 0 in Prometheus. Container logs show:

Cannot connect to MongoDB: invalid dsn: error parsing uri: unescaped @ sign in user info

Confirm

ssh root@54.36.123.110 "docker logs bms-4-mongodb-exporter-bms2-1 --tail 5 2>&1"
ssh root@54.36.123.110 "curl -s http://localhost:9216/metrics | grep mongodb_up"

Expected broken: mongodb_up{cluster_role=""} 0 Expected fixed: mongodb_up{cluster_role="mongod"} 1

Root cause

The MongoDB URI mongodb://user:PASSWORD@host/db treats @ as a delimiter. If MONGODB_RS0_PROMETHEUS_PASSWORD contains an @, the URI parser fails. This surfaces when containers are recreated (new env var expansion).

Fix

Step 1 — Add URL-encoded password to SOPS on Windows dev workstation:

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$secretsDir = "C:\code_2026\p24-infra\secrets"
$decrypted = sops --decrypt --input-type dotenv --output-type dotenv "$secretsDir\n8n-bms4.env.sops"
$rawPwd = ($decrypted | Where-Object { $_ -match "^MONGODB_RS0_PROMETHEUS_PASSWORD=" }).Split("=",2)[1]
$encodedPwd = [Uri]::EscapeDataString($rawPwd)
$newContent = ($decrypted + @("MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED=$encodedPwd")) -join "`n"
[System.IO.File]::WriteAllText("$secretsDir\n8n-bms4-edit.env.sops", $newContent + "`n", [System.Text.UTF8Encoding]::new($false))
$enc = sops --encrypt --input-type dotenv --output-type dotenv "$secretsDir\n8n-bms4-edit.env.sops"
[System.IO.File]::WriteAllText("$secretsDir\n8n-bms4.env.sops", ($enc -join "`n") + "`n", [System.Text.UTF8Encoding]::new($false))
Remove-Item "$secretsDir\n8n-bms4-edit.env.sops" -Force

Step 2 — Add encoded var to bms-4 .env immediately (no wait for CI):

ssh root@54.36.123.110 "
RAW=\$(grep '^MONGODB_RS0_PROMETHEUS_PASSWORD=' /opt/p24-infra/bms-4/.env | cut -d= -f2-)
ENCODED=\$(python3 -c \"import urllib.parse, sys; print(urllib.parse.quote(sys.argv[1], safe=''))\" \"\$RAW\")
echo \"MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED=\$ENCODED\" >> /opt/p24-infra/bms-4/.env
"

Step 3 — Update docker-compose.yml (on server for immediate fix):

ssh root@54.36.123.110 "python3 -c \"
with open('/opt/p24-infra/bms-4/docker-compose.yml') as f: c = f.read()
c = c.replace('\${MONGODB_RS0_PROMETHEUS_PASSWORD}@', '\${MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED}@')
with open('/opt/p24-infra/bms-4/docker-compose.yml', 'w') as f: f.write(c)
\""

Step 4 — Restart containers:

ssh root@54.36.123.110 "cd /opt/p24-infra/bms-4 && docker compose up -d --no-deps mongodb-exporter-bms2 mongodb-exporter-bms3"

Step 5 — Verify after 30 seconds:

ssh root@54.36.123.110 "curl -s http://localhost:9216/metrics | grep mongodb_up"

Step 6 — Commit the git changes (bms-4/docker-compose.yml + secrets/n8n-bms4.env.sops) via PR to main.

Prevention

If rotating MONGODB_RS0_PROMETHEUS_PASSWORD to a value that contains special URI chars (@, #, /, ?):

  • Always update MONGODB_RS0_PROMETHEUS_PASSWORD_ENCODED in SOPS at the same time
  • Use [Uri]::EscapeDataString() (PowerShell) or urllib.parse.quote(pwd, safe='') (Python) to encode

Escalation

If mongodb_up stays 0 after the fix:

  1. Check the prometheus user still exists in MongoDB: mongosh admin --eval "db.getUser('prometheus')"
  2. Check network: nc -zv 145.239.133.104 27017 from bms-4
  3. Check MongoDB logs on bms-2: ssh ubuntu@145.239.133.104 "sudo tail -50 /var/log/mongodb/mongod.log"

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="config_change",
    resource="mongodb-exporter",
    result="success",  # "success" | "failed" | "skipped"
    detail="MongoDB exporter URI fixed — special characters in password encoded in SOPS",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'config_change', 'mongodb-exporter', 'success', 'MongoDB exporter URI fixed — special characters in password encoded in SOPS', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks