p24-infra Docs

queue-dispatcher-bms4-env-crlf

3 min read

Playbook: bms-4 queue-dispatcher SUPABASE_SERVICE_ROLE_KEY unbound variable

What triggers this problem

queue-dispatcher.sh on bms-4 crashes every run with:

SUPABASE_SERVICE_ROLE_KEY: unbound variable

Despite EnvironmentFile=/opt/p24-infra/bms-4/.env being correctly configured in /etc/systemd/system/p24-queue-dispatcher.service.

Root cause

/opt/p24-infra/bms-4/.env was deployed with CRLF line endings (\r\n instead of \n). systemd 249 (Ubuntu 22.04 on bms-4) does not fully strip \r when parsing EnvironmentFile, causing the variable to be silently ignored or its value corrupted.

When the bash script uses set -euo pipefail and references ${SUPABASE_SERVICE_ROLE_KEY}, bash throws unbound variable because systemd never loaded the key into the service environment.

Confirming CRLF presence (no values shown):

grep -cP "\r" /opt/p24-infra/bms-4/.env   # returns N > 0 if CRLF present

Confirming the variable is empty in the service environment:

bash -c 'source /opt/p24-infra/bms-4/.env 2>/dev/null; echo "SUPA_LEN=${#SUPABASE_SERVICE_ROLE_KEY}"'
# Returns SUPA_LEN=41 if OK, SUPA_LEN=0 if CRLF bug

How to confirm

# On bms-4 as root:
grep -cP "\r" /opt/p24-infra/bms-4/.env
# > 0 = CRLF problem
 
tail -5 /var/log/p24-infra-dispatcher.log
# Shows: "SUPABASE_SERVICE_ROLE_KEY: unbound variable"
 
systemctl status p24-queue-dispatcher.service
# Shows repeated failures

Fix

Fast fix: re-run secrets-sync.yml targeting bms-4

# From dev machine:
gh workflow run secrets-sync.yml --repo radieu/p24-infra -f target=bms-4

The workflow includes sed -i 's/\r$//' /tmp/n8n-bms4.env before deploying, which strips CRLF. After the workflow completes, the service will work on its next 2-minute timer tick.

Verify:

# On bms-4:
grep -cP "\r" /opt/p24-infra/bms-4/.env   # should return 0
systemctl restart p24-queue-dispatcher.service
systemctl status p24-queue-dispatcher.service
tail -3 /var/log/p24-infra-dispatcher.log
# Should show: "Server bms-4: dispatched N jobs this cycle"

Manual fix (if workflow unavailable)

# On bms-4 as root — strip CRLF in place:
sed -i 's/\r$//' /opt/p24-infra/bms-4/.env
systemctl restart p24-queue-dispatcher.service

Re-queuing failed jobs after fix

Jobs that were dispatched while the dispatcher was broken may have status failed_permanently with spawn_failures=0 (marked without any worker execution). Reset them via Python on bms-4:

# /tmp/reset-jobs.py — run on bms-4 as root
import subprocess, json, urllib.request
 
key = subprocess.check_output(["bash", "-c",
    "source /opt/p24-infra/bms-4/.env 2>/dev/null; echo $SUPABASE_SERVICE_ROLE_KEY"
]).decode().strip()
SUPA_URL = "https://mwkqmgadqnkkihjdeqsi.supabase.co"
 
def reset_job(row_id):
    url = SUPA_URL + f"/rest/v1/dev_r_worker_queue?id=eq.{row_id}"
    body = json.dumps({"status": "queued", "spawn_failures": 0, "next_attempt_at": None}).encode()
    req = urllib.request.Request(url, data=body, method="PATCH",
        headers={"apikey": key, "Authorization": "Bearer " + key,
                 "Content-Type": "application/json", "Prefer": "return=representation"})
    with urllib.request.urlopen(req) as r:
        return json.loads(r.read())
 
# Find failed_permanently rows with 0 spawn_failures (never executed)
url = SUPA_URL + "/rest/v1/dev_r_worker_queue?status=eq.failed_permanently&spawn_failures=eq.0&select=id,issue_number,status&limit=50"
req = urllib.request.Request(url, headers={"apikey": key, "Authorization": "Bearer " + key})
with urllib.request.urlopen(req) as r:
    rows = json.loads(r.read())
 
for row in rows:
    result = reset_job(row["id"])
    print(f"#{row['issue_number']} id={row['id']}: reset to queued")

Note on unique constraint: PATCH by issue_number=eq.N&status=eq.failed_permanently will fail with 409 if multiple failed rows exist for the same issue. Always patch by id=eq.N to target a single row.

Prevention

The secrets-sync.yml workflow already includes sed -i 's/\r$//' before deploying. CRLF only appears if the env file was manually deployed or the workflow was run before the sed step was added.

Check .gitattributes in secrets/: secrets/*.env.sops text eol=lf ensures SOPS files are committed with LF. The plaintext .env on-server is not git-tracked, so protection is CI-side only (via the sed step).

Escalation

If CRLF persists after secrets-sync re-run, check:

  1. The bms-4 GH runner is healthy: gh run list --workflow secrets-sync.yml --limit 5
  2. The Ship to bms-4 step completed without error
  3. Manually strip CRLF in place (see Manual fix above)

Graph View