Variables and Dashboard Design

Template variables and layout

Dashboard variables make dashboards reusable: a job variable populated by label_values(up, job) and an instance variable that depends on it let one dashboard serve every service. Use them in queries as $job. Lay out dashboards top-down: key health indicators (traffic, errors, latency, saturation) in the first row, details below, and link to drill-down dashboards. Add descriptions to panels so newcomers know what each one means.

Variable queries

Grafana Prometheus variable definitions.

job       label_values(up, job)
instance  label_values(up{job="$job"}, instance)      (multi-value, include All)
route     label_values(http_requests_total{job="$job"}, route)

Panel query using multi-value variables:
  sum by (instance) (rate(http_requests_total{job="$job", instance=~"$instance"}[$__rate_interval]))

Use =~ with multi-value variables

Multi-select variables expand to a regex such as (a|b), so match them with =~.

Quick check: What does label_values(up, job) provide for a variable?

• The number of jobs
• The list of job label values found on the up metric
• A PromQL alert
• A new data source