n8n-queue-worker-restart

Playbook — Direct-SQL n8n workflow edits: update the ACTIVE version, then restart (bms-4)

Status: active

Corrected 2026-10-05. The first version of this playbook (#7388, PR #7389) blamed “stale queue-mode worker containers” for a direct-SQL fix not taking effect. That diagnosis was wrong. Restarting all four containers changed nothing: the next scheduled run still ran the old code. The real cause is n8n 2.x workflow versioning, described below. The filename is kept because other docs link to it.

Why a direct-SQL edit to workflow_entity.nodes is not enough

bms-4 runs n8n 2.x, which versions workflows:

Table / column	What it holds	Who reads it
workflow_entity.nodes / .connections	the editor copy — what the UI shows	manual / editor runs (POST /rest/workflows/<id>/run), the UI
workflow_entity."activeVersionId"	pointer to the published version	the trigger/scheduler
workflow_history row where "versionId" = activeVersionId	the active (published) version — nodes + connections	scheduled/trigger runs, including Schedule Trigger cron registration

A UI Save + Publish writes a new workflow_history row and moves activeVersionId to it. A direct UPDATE workflow_entity SET nodes = … (the “surgical” pattern used to avoid the import:workflow REDACTED_CREDENTIAL clobber, see atrax-daily-workflows-paired-item-and-stale-key.md) changes only the editor copy. Consequences:

• Scheduled runs keep executing the old code indefinitely. execution_data.workflowData of a trigger run shows the old nodes, while workflow_entity.nodes shows the new ones.
• Manual runs use the new code, so a manual “verification” run falsely shows the fix working.
• A cron change made this way (e.g. switching the Schedule Trigger to * * * * * to force a test run) is never registered. The trigger looks “dead”, but it is still firing on the old cron.
• Restarting main and/or the queue-mode workers does not help. They reload the same active version.

Confirmed incident (2026-09-29 → 2026-10-05, p24_atrax_drivers_daily)

• #7304 fixed the map_drivers keys in workflow_entity.nodes only (2026-09-29). Active version e6128bc7-bf8e-4208-b0db-39efb2954b36 (sole workflow_history row, created 2026-08-09) kept the old keys (dayDrivingTime, thisWeekRemainingTime, last2WeeksDrivingTime, last2WeeksRemainingTime).
• Every scheduled run 2026-10-02 → 2026-10-05 wrote NULL for day_driving_min, this_week_remaining_min, last_2w_driving_min, last_2w_remaining_min.
• #7388 restarted all containers on 2026-10-04 and “verified” with a manual run (557892), which read the editor copy and looked fixed. The next scheduled run (560389, 2026-10-05 04:00 UTC) was NULL again.
• #7391 (“Schedule Trigger never fires”) was the same bug: its * * * * * test cron was written to workflow_entity.nodes only.
• Fixed 2026-10-05 by copying the editor copy into the active workflow_history row (procedure below). Verified with a scheduled run (cron temporarily set to 12 10 * * * in both tables): execution 561228, mode trigger, success, workflowData contains the new keys. stat_date 2026-10-04 now has 260/262 rows non-NULL in all four columns. Cron restored to 0 6 * * *.
• stat_date 2026-10-02 stays NULL and cannot be backfilled: /api/rtdrivers only returns live counters. The 2026-10-04 row holds counters as of the 10:12 Warsaw run on 2026-10-05, not end of day 2026-10-04.

Preferred method: UI Save + Publish

If anyone can open the n8n editor, open the workflow, Save, then Publish/Activate. n8n creates the history row and moves activeVersionId itself. Use SQL only when the UI is not an option.

Direct-SQL procedure

Run on bms-4 against the n8n Postgres (bms-4-n8n-postgres-1). Replace <WF_ID>.

# PLAYBOOK: n8n-queue-worker-restart.md
PSQL='docker exec -i bms-4-n8n-postgres-1 sh -c "psql -U \$POSTGRES_USER -d \$POSTGRES_DB -At -v ON_ERROR_STOP=1"'
 
# 1. Inspect: which version is active, and does it differ from the editor copy?
ssh root@54.36.123.110 "$PSQL" <<'SQL'
select e."activeVersionId", e."versionId", e."updatedAt",
       (h.nodes::text = e.nodes::text) as active_matches_editor
  from workflow_entity e
  left join workflow_history h on h."versionId" = e."activeVersionId"
 where e.id = '<WF_ID>';
SQL
 
# 2. Back up the active version locally before touching it.
ssh root@54.36.123.110 "$PSQL" <<'SQL' > workflow_history_<WF_ID>_backup.json
select json_build_object('versionId',h."versionId",'nodes',h.nodes,'connections',h.connections)
  from workflow_history h join workflow_entity e on h."versionId" = e."activeVersionId"
 where e.id = '<WF_ID>';
SQL
 
# 3. Make the edit in BOTH places, in one transaction. Either edit workflow_entity first and then
#    copy it across (below), or apply the same replace() to both tables.
ssh root@54.36.123.110 "$PSQL" <<'SQL'
begin;
update workflow_history h
   set nodes = e.nodes, connections = e.connections, "updatedAt" = now()
  from workflow_entity e
 where e.id = '<WF_ID>' and h."versionId" = e."activeVersionId";
commit;
SQL
 
# 4. Restart main + all queue-mode workers so the trigger re-registers from the active version.
ssh root@54.36.123.110 \
  "docker restart bms-4-n8n-1 bms-4-n8n-worker-1-1 bms-4-n8n-worker-2-1 bms-4-n8n-worker-3-1"

Overwriting the active history row in place keeps the same versionId, so nothing else that references it breaks. The trade-off is that the old version is lost from history, which is why step 2 backs it up. Inserting a new workflow_history row and repointing activeVersionId is the alternative, but it is more error-prone by hand. Prefer the UI for that.

Before step 4, check that no other deploy is touching the same containers (ps -eo etime,cmd | grep 'compose .*up'). A concurrent docker compose up -d recreates them underneath you.

Verification

1. docker ps --filter name=bms-4-n8n: all containers Up, and curl -s localhost:5678/healthz returns {"status":"ok"}.
2. Step 1’s query now returns active_matches_editor = t.
3. Wait for a scheduled run, not a manual one (manual runs read the editor copy and prove nothing). To force one, change the cron to a few minutes ahead in both tables with the same replace(), restart main + workers, let it run once, then revert both and restart again.
4. Confirm the trigger execution’s execution_data.workflowData contains the new code, and check the output data (e.g. Supabase) for the expected values.

Related

• atrax-daily-workflows-paired-item-and-stale-key.md: the REDACTED_CREDENTIAL import landmine that motivates direct-SQL edits, and the main-restart reload gotcha.
• ssh-agent-pattern.md: silent-unless-error SSH pattern.
• .claude/task-playbooks/server-operation.md: SSH pre-flight and the # PLAYBOOK: annotation.

---

First documented 2026-10-04 (#7388, wrong root cause); corrected 2026-10-05.