p24-infra Docs

n8n-alertmanager-incidents

7 min read

Playbook: n8n alertmanager-incidents Webhook — GH Issue Branches

Applies to: n8n queue-mode instance on bms-4 (https://n8n.bms-4.infra.zintegrowana.online), workflow alertmanager-incidents (webhook path /webhook/alertmanager-incidents) Last updated: 2026-07-09 See also:

  • docs/playbooks/n8n/n8n-workflow-creation.md — general n8n API patterns, credential loading, quirks
  • monitoring/alertmanager/alertmanager.yml.tpl — routes/receivers that POST to this webhook
  • monitoring/prometheus/rules/pinbox24.yml — Pinbox24BackendRestartLoop alert (issue #2567)
  • Issue #2567 / PR #2570 — origin of the pm2-restart-loop category and this doc’s requirement

0. Why this playbook exists

alertmanager.yml.tpl routes several alert categories to the shared alertmanager-incidents n8n webhook (critical catch-all, resource-incident, and — since issue #2567 — pm2-restart-loop). The webhook itself is not versioned in this repo; it only exists as a workflow built in the n8n UI/API on bms-4. Every alertmanager.yml.tpl comment that introduced a new branch pointed here for “how to add it”, but this file never existed until now (gap found while re-verifying issue #2567’s acceptance criteria on 2026-07-09 — the code side had been merged since PR #2570 on 2026-07-02, but the n8n branch was left as a manual TODO and nobody had written down what “manual” meant).

This playbook is the concrete, copy-paste spec for the pm2-restart-loop branch. Follow it exactly once to close out #2567’s remaining acceptance criteria:

  • GH issue created in radieu/p24-infra (label bug) when Pinbox24BackendRestartLoop fires
  • No duplicate GH issues on alert flap (dedup check against existing open issues first)

Role note: creating/editing this workflow requires the n8n API key from secrets/n8n-bms4.env.sops (BMS4_N8N_API_KEY). Per CLAUDE.md §Role Enforcement, a dev-coder session must not decrypt SOPS or call the live n8n API directly — this is secret-manager/sys-admin territory (or a human with n8n UI access). This doc is written so either can execute it without re-deriving the design.

1. Alertmanager payload this branch must handle

alertmanager.yml.tpl’s pm2-restart-loop route/receiver (added in PR #2570):

    - matchers:
        - category = pm2-restart-loop
      receiver: pm2-restart-loop-gh
      group_by: ['alertname', 'container']
      group_wait: 0s
      repeat_interval: 30m
      continue: true       # ALSO flows to the normal critical (email+Discord) receiver
  - name: pm2-restart-loop-gh
    webhook_configs:
      - url: 'https://n8n.bms-4.infra.zintegrowana.online/webhook/alertmanager-incidents'
        send_resolved: false

Alertmanager’s native webhook payload (v4 format) looks like this for a Pinbox24BackendRestartLoop firing:

{
  "receiver": "pm2-restart-loop-gh",
  "status": "firing",
  "alerts": [
    {
      "status": "firing",
      "labels": {
        "alertname": "Pinbox24BackendRestartLoop",
        "category": "pm2-restart-loop",
        "container": "v42-prod",
        "severity": "critical"
      },
      "annotations": {
        "summary": "v42-prod PM2 crash-loop -- 8 restarts in 10m on bms-1",
        "description": "v42-prod PM2 is crash-looping (8 restarts in 10m).\nINVESTIGATE: ssh root@94.23.26.113 ..."
      },
      "startsAt": "2026-07-09T10:15:00Z",
      "generatorURL": "https://prometheus.vps-i1.infra.zintegrowana.online/graph?..."
    }
  ],
  "groupLabels": { "alertname": "Pinbox24BackendRestartLoop", "container": "v42-prod" },
  "commonLabels": { "category": "pm2-restart-loop", "severity": "critical" },
  "commonAnnotations": { "summary": "v42-prod PM2 crash-loop -- 8 restarts in 10m on bms-1" }
}

send_resolved: false means this webhook only ever receives status: "firing" — there is no resolved callback to handle for this category (resolution is tracked via the GH issue lifecycle, not by Alertmanager).

2. Branch logic to add to the alertmanager-incidents workflow

Locate the workflow first (do not assume an ID — it changes if recreated):

$env:SOPS_AGE_KEY_FILE = "C:\Users\konar\.age\p24-infra-keys.txt"
$n8nHost = "https://n8n.bms-4.infra.zintegrowana.online"
$env:BMS4_N8N_API_KEY = (
  sops --decrypt --input-type dotenv --output-type dotenv C:\code_2026\p24-infra\secrets\n8n-bms4.env.sops |
  Select-String "^BMS4_N8N_API_KEY="
).ToString().Split("=", 2)[1]
$headers = @{ "X-N8N-API-KEY" = $env:BMS4_N8N_API_KEY; "Content-Type" = "application/json" }
 
$all = Invoke-RestMethod -Uri "$n8nHost/api/v1/workflows?limit=250" -Headers $headers -Method GET
$wf  = $all.data | Where-Object { $_.name -eq "alertmanager-incidents" } | Select-Object -First 1
Write-Host "ID: $($wf.id) | Active: $($wf.active)"
$env:BMS4_N8N_API_KEY = ""   # clear immediately after use

Fetch its current definition (GET /api/v1/workflows/{id}) before editing — never rebuild it from scratch, since it already handles the critical catch-all and resource-incident branches for other alerts. Add the following as a new parallel branch off whatever node currently receives the raw webhook payload (commonly a Webhook trigger node feeding a Switch/IF chain keyed on $json.commonLabels.category or the first alert’s labels.category).

Node A — IF: category is pm2-restart-loop?

SettingValue
Typen8n-nodes-base.if (typeVersion: 1, per n8n-http-request-quirks.md — string equality, not isTrue)
Condition{{ $json.commonLabels.category }} equals pm2-restart-loop
  • True -> Node B
  • False -> existing branches, unchanged

Node B — HTTP Request: check for an existing open GH issue (dedup)

SettingValue
Typen8n-nodes-base.httpRequest
MethodGET
URLhttps://api.github.com/search/issues?q=repo:radieu/p24-infra+is:issue+is:open+in:title+%22%5BAlert%5D+v42-prod+PM2+restart+loop%22
HeadersAuthorization: Bearer <GH_TOKEN> (repo-write PAT, same one used by github-auto-trigger workflow — see secrets/n8n-bms4.env.sops key GH_TOKEN), Accept: application/vnd.github.v3+json

Response total_count is the dedup signal.

Node C — IF: total_count == 0?

SettingValue
Typen8n-nodes-base.if
Condition{{ $json.total_count }} equals 0
  • True (no open issue yet) -> Node D
  • False (one already open) -> stop this branch; the running crash-loop is already tracked. Optionally add a comment to the existing issue instead of a no-op — see §4 “Future improvement”.

Node D — HTTP Request: create GH issue

SettingValue
Typen8n-nodes-base.httpRequest
MethodPOST
URLhttps://api.github.com/repos/radieu/p24-infra/issues
HeadersAuthorization: Bearer <GH_TOKEN>, Accept: application/vnd.github.v3+json
Body (specifyBody: "json", built via a Code node upstream so it’s a real JSON string — see Quirk 1 in n8n-http-request-quirks.md)see below

Code node (“Build GH Issue Body”) feeding Node D:

// runOnceForAllItems
const alert = $('Webhook').first().json.alerts[0];
const container = alert.labels.container;
const summary = alert.annotations.summary;
const description = alert.annotations.description || '';
const generatorUrl = alert.generatorURL || '';
 
const title = `[Alert] v42-prod PM2 restart loop -- ${summary.match(/\d+/)?.[0] || '?'} restarts in 10m`;
const body = [
  `**${summary}**`,
  '',
  description,
  '',
  `Alertmanager fired at ${alert.startsAt}.`,
  generatorUrl ? `Prometheus: ${generatorUrl}` : '',
  '',
  '_Created automatically by the n8n `alertmanager-incidents` workflow, `pm2-restart-loop` branch (issue #2567)._',
].filter(Boolean).join('\n');
 
return [{ json: {
  gh_body: JSON.stringify({ title, body, labels: ['bug'] }),
}}];

Node D’s jsonBody parameter: ={{ $json.gh_body }}.

3. Connections summary

Webhook
  -> IF: category == pm2-restart-loop?
       true  -> HTTP: search open GH issues (dedup)
                  -> IF: total_count == 0?
                       true  -> Code: build GH issue body -> HTTP: create GH issue
                       false -> (no-op / optional: comment on existing issue)
       false -> [existing critical / resource-incident branches, unchanged]

4. Testing

Simulated webhook payload (no live Prometheus alert needed)

$testPayload = @{
  receiver = "pm2-restart-loop-gh"
  status   = "firing"
  alerts   = @(@{
    status      = "firing"
    labels      = @{ alertname = "Pinbox24BackendRestartLoop"; category = "pm2-restart-loop"; container = "v42-prod"; severity = "critical" }
    annotations = @{ summary = "v42-prod PM2 crash-loop -- 8 restarts in 10m on bms-1 (TEST)"; description = "Simulated test alert -- safe to close the resulting GH issue immediately." }
    startsAt    = (Get-Date).ToUniversalTime().ToString("yyyy-MM-ddTHH:mm:ssZ")
  })
  commonLabels = @{ category = "pm2-restart-loop"; severity = "critical" }
} | ConvertTo-Json -Depth 10
 
Invoke-RestMethod -Uri "https://n8n.bms-4.infra.zintegrowana.online/webhook/alertmanager-incidents" `
  -Method POST -Body $testPayload -ContentType "application/json"

Verify:

  1. A GH issue titled [Alert] v42-prod PM2 restart loop -- 8 restarts in 10m appears in radieu/p24-infra with label bug.
  2. Re-POST the same payload -> Node C’s dedup check must find the issue just created and take the false branch (no second issue created).
  3. Close the test issue manually afterward.

End-to-end (real alert)

Requires a real Pinbox24BackendRestartLoop firing — do not force this in production; rely on the simulated payload above for verification, and confirm against real firings retroactively via the n8n executions list (https://n8n.bms-4.infra.zintegrowana.online -> workflow -> Executions).

5. Escalation

ProblemAction
Webhook workflow inactivePOST /api/v1/workflows/{id}/activate — see n8n-workflow-creation.md §7
GH issue not created, no error in n8nCheck GH_TOKEN in secrets/n8n-bms4.env.sops hasn’t expired — see docs/playbooks/n8n/n8n-bms4-api-key-rotation.md for the rotation pattern (same file, different key)
Duplicate issues still appearingDedup search query too narrow/wide — verify the GitHub search API query string manually against https://github.com/radieu/p24-infra/issues?q=... before blaming the workflow
Alert never reaches n8n at allCheck Alertmanager routing tree matched category = pm2-restart-loop first (routes are evaluated top-down; a broader earlier route can shadow it) — docker exec alertmanager amtool config routes on vps-i1

6. Future improvement (not required by #2567)

Instead of a no-op on the dedup “already open” branch, add a GitHub API comment to the existing issue with the new restart count, so a still-ongoing crash-loop accumulates evidence in one thread rather than only the first occurrence. Track as a follow-up issue if desired — out of scope for the original #2567 acceptance criteria, which only requires no duplicate issues.

Graph View

Backlinks