p24-infra Docs

dispatcher-metadata-json-crash

2 min read

Playbook: Dispatcher Crashes — AttributeError on metadata column

Symptom: queue-dispatcher-loop.py crashes every 2 minutes with:

AttributeError: 'str' object has no attribute 'get'
  File "queue-dispatcher-loop.py", line NNN, in spawn_worker
    pr_number = meta.get("pr_number", "")

Dispatcher log shows “Dispatching nnnn” immediately followed by the traceback. Workers never spawn.

Trigger

A queued row in dev_r_worker_queue has a non-null metadata field. The Supabase REST API returns metadata as a JSON-encoded string (text column type), but the dispatcher calls .get() directly on it without json.loads().

Confirm

# Check dispatcher logs
journalctl -u p24-queue-dispatcher.service --no-pager -n 20
 
# Should show repeating "AttributeError" on every run

Fix

  1. Edit scripts/queue-dispatcher-loop.py — find every meta = job.get("metadata") or {} and replace with: 
    _meta_raw = job.get("metadata") or {}
meta = json.loads(_meta_raw) if isinstance(_meta_raw, str) else _meta_raw
  2. PR → main → merge → git pull origin main on vps-i1 → restart dispatcher: 
    systemctl restart p24-queue-dispatcher.timer
systemctl start p24-queue-dispatcher.service

Reset stale rows after fix

Failed/claimed rows that accumulated during the crash must be reset to queued:

# Run via docker exec monitoring-queue-exporter-1 python3 /tmp/reset.py
import os, urllib.request, json
url = os.environ["SUPABASE_URL"] + "/rest/v1/dev_r_worker_queue"
key = os.environ["SUPABASE_SERVICE_KEY"]
hdrs = {"apikey": key, "Authorization": "Bearer " + key,
        "Content-Type": "application/json", "Prefer": "return=representation"}
for row_id in [130, 131, ...]:  # IDs of stuck rows
    body = json.dumps({"status": "queued", "server_node": None,
                       "started_at": None, "error_message": None, "retry_count": 0}).encode()
    req = urllib.request.Request(url + "?id=eq." + str(row_id), data=body, headers=hdrs, method="PATCH")
    result = json.load(urllib.request.urlopen(req))
    for r in result:
        print("RESET id=%s issue=%s" % (r["id"], r["issue_number"]))

Prevention

Any new code path in queue-dispatcher-loop.py that reads job["metadata"] must use the safe pattern:

_raw = job.get("metadata") or {}
meta = json.loads(_raw) if isinstance(_raw, str) else _raw

First fixed in PR #1657 (2026-06-27), issue #1660.

Audit Log — Log to infra_operations

After this operation completes, log it to the infra_operations audit table.

Python (Linux server — bms-4, vps-i1, vps-h1, or similar):

import sys
sys.path.insert(0, '/opt/p24-infra')
from scripts.lib.log_op import log_op
 
log_op(
    actor="claude",  # "radieu" for manual human ops, "claude" for agent
    op_type="restart",
    resource="meta-dispatcher",
    result="success",  # "success" | "failed" | "skipped"
    detail="Dispatcher metadata JSON crash resolved — malformed job purged, service restarted",
    env="bms-4",
    gh_issue=2730,
)

PowerShell (Windows dev machine):

$env:SUPABASE_URL = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_URL=").ToString().Split("=",2)[1].Trim()
$env:SUPABASE_SERVICE_KEY = (Get-Content "C:\code_2026\p24-infra\.env.local" | Select-String "^SUPABASE_SERVICE_KEY=").ToString().Split("=",2)[1].Trim()
python -c "
import os, sys
sys.path.insert(0, 'C:/code_2026/p24-infra')
from scripts.lib.log_op import log_op
log_op('claude', 'restart', 'meta-dispatcher', 'success', 'Dispatcher metadata JSON crash resolved — malformed job purged, service restarted', 'bms-4')
"
$env:SUPABASE_URL = ''; $env:SUPABASE_SERVICE_KEY = ''

Graph View

Backlinks