Lesson 19 / 25

Data Sources and Panels

From query to visualisation.

Connect Prometheus, then build panels

Add Prometheus as a data source (its URL, for example http://prometheus:9090), then create a dashboard of panels. Each panel runs one or more queries and shows them as a time series graph, stat, gauge, bar chart, table or heatmap (good for histogram buckets). Set the correct unit (seconds, bytes, percent, requests per second) and thresholds so colours highlight problems. Explore mode is for ad-hoc querying during investigations, and Grafana can also alert on queries.

Visualise and explore

Grafana turns PromQL queries into dashboards that help people understand systems quickly.

Three ideas: data sources and panels, variables and good layout, dashboards as code.
Figure 7.1 — Panels, variables and provisioning.

Panels for a service overview

Queries for a RED dashboard.

Panel "Request rate"   (time series, unit req/s)
  sum by (route) (rate(http_requests_total{job="$job"}[$__rate_interval]))
Panel "Error ratio"    (stat, unit percent 0-1, thresholds 1% amber, 5% red)
  job:http_requests_error_ratio:rate5m{job="$job"}
Panel "p95 latency"    (time series, unit seconds)
  histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket{job="$job"}[$__rate_interval])))
Panel "Latency heatmap" (heatmap, format: heatmap)
  sum by (le) (increase(http_request_duration_seconds_bucket{job="$job"}[$__rate_interval]))

Use $__rate_interval

Grafana's $__rate_interval adapts the rate window to the zoom level and scrape interval, avoiding empty or jagged graphs.

Quick check: Which panel type best shows a latency distribution over time?

  • Text
  • Stat
  • Pie chart
  • Heatmap
Answer

Heatmap — Buckets over time.