Grafana Dashboard Authoring Playbook

Role: grafana-dashboard-builder · Job type: grafana-dashboard Issue refs: #7890, #7930

How to dispatch a dashboard task

Add the grafana-dashboard label to any issue (at creation or later). dispatch-to-queue.yml detects the label in both the issues.milestoned and issues.reopened handlers and calls queue_dispatch with job_type=grafana-dashboard, which routes the job to a grafana-dashboard-builder worker rather than a generic dev-coder.

Keywords that suggest this label: grafana, dashboard, panel.

Overview

Dashboards in p24-infra are provisioned via JSON files under monitoring/grafana/provisioning/dashboards/. Grafana reads them on restart and automatically syncs changes when update_interval_seconds fires (default: 30 s). The grafana-dashboard-builder role owns these files end-to-end — author in git, merge via PR, and sys-admin handles the live reload.

Datasource UIDs

Name	UID	Plugin
Prometheus	`prometheus`	grafana-prometheus-datasource
Supabase (Infinity)	`supabase`	yesoreyeram-infinity-datasource

Never use supabase-rest — that UID is stale. Check monitoring/grafana/provisioning/datasources/ for the current set.

Step 1 — Understand the metric space

Before building a panel, confirm the metric exists:

# From vps-i1 (via sys-admin for SSH access):
# curl -s http://localhost:9090/api/v1/label/__name__/values | jq -r '.data[]' | grep <keyword>
 
# Locally (if Prometheus is reachable):
# promtool query instant http://<prometheus_host>:9090 '<expr>'

If you cannot validate the PromQL (no Prometheus access from the worker), note it in the PR body: “PromQL unvalidated — requires Prometheus access on vps-i1.”

Step 2 — Scaffold the dashboard JSON

Minimal valid dashboard skeleton:

{
  "__inputs": [],
  "__requires": [],
  "annotations": {"list": []},
  "editable": false,
  "fiscalYearStartMonth": 0,
  "graphTooltip": 1,
  "id": null,
  "links": [],
  "panels": [],
  "refresh": "1m",
  "schemaVersion": 38,
  "tags": ["p24-infra"],
  "templating": {"list": []},
  "time": {"from": "now-6h", "to": "now"},
  "timepicker": {},
  "timezone": "browser",
  "title": "My Dashboard",
  "uid": "my-dashboard-v1",
  "version": 1
}

UID naming: <slug>-v1 where slug is kebab-case and unique across monitoring/grafana/provisioning/dashboards/*.json.

Step 3 — Add panels

Time-series panel (Prometheus)

{
  "datasource": {"type": "datasource", "uid": "prometheus"},
  "fieldConfig": {
    "defaults": {"color": {"mode": "palette-classic"}, "custom": {"lineWidth": 1}},
    "overrides": []
  },
  "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
  "id": 1,
  "options": {"legend": {"displayMode": "list", "placement": "bottom"}, "tooltip": {"mode": "multi"}},
  "targets": [
    {
      "datasource": {"type": "datasource", "uid": "prometheus"},
      "expr": "rate(my_metric_total[5m])",
      "legendFormat": "{{label_name}}",
      "refId": "A"
    }
  ],
  "title": "My Panel Title",
  "type": "timeseries"
}

Stat panel

{
  "datasource": {"type": "datasource", "uid": "prometheus"},
  "fieldConfig": {"defaults": {"thresholds": {"mode": "absolute", "steps": [{"color": "green", "value": null}]}}, "overrides": []},
  "gridPos": {"h": 4, "w": 6, "x": 0, "y": 0},
  "id": 2,
  "options": {"colorMode": "background", "graphMode": "none", "reduceOptions": {"calcs": ["lastNotNull"]}},
  "targets": [{"datasource": {"type": "datasource", "uid": "prometheus"}, "expr": "my_gauge", "refId": "A"}],
  "title": "Current Value",
  "type": "stat"
}

Step 4 — Validate PromQL

If promtool is available locally:

# Validate expressions in isolation (parse-only, no running Prometheus needed)
echo 'rate(my_metric_total[5m])' | promtool query instant --no-color http://localhost:9090 /dev/stdin 2>&1 || true
 
# Check a recording rule file
promtool check rules prometheus/rules/my_rules.yml

Step 5 — File placement and provider

Dashboard JSON → monitoring/grafana/provisioning/dashboards/<dashboard-name>.json
Provider config → monitoring/grafana/provisioning/datasources/dashboards.yml (this file is rarely changed; only update if adding a new provider folder)

Step 6 — Compliance checklist

Before opening a PR:

Dashboard UID is unique (grep monitoring/grafana/provisioning/dashboards/*.json)
All panel datasource refs use uid: prometheus or uid: supabase (not hardcoded URLs)
No secret values embedded in JSON
dev_r_services row created/updated in Supabase
Ops doc created or updated in docs/
EU AI Act check: does this dashboard score/rank human workers? → flag, do not ship

Step 7 — PR and handoff

Open the PR with Implements: #<issue> (full scope) or Partially implements: (subset).

In the PR body, note:

**Live reload:** after merge, sys-admin must restart Grafana on vps-i1 OR wait for the
30 s auto-sync: `docker compose restart grafana` from /opt/p24-infra/monitoring.

The implementing worker does NOT restart Grafana — that is sys-admin scope.

Escalation

Situation	Action
New metric needed (no exporter exists)	Create a `dev-issue` for `dev-coder`
Need to push JSON directly to live server	Comment on issue, spawn/request `sys-admin`
Datasource credential misconfigured	Spawn `secret-manager` to check SOPS keys

p24-infra Docs

Explorer

grafana-dashboard-authoring