Lesson 15 / 25

Recording Rules

Precompute expensive queries.

Store query results as new series

A recording rule evaluates a PromQL expression on a schedule and saves the result as a new metric. Dashboards and alerts then read the cheap precomputed series instead of recalculating heavy aggregations over many series. The naming convention is level:metric:operations, for example job:http_requests:rate5m. Rules live in groups in rule files referenced by prometheus.yml and are validated with promtool check rules.

A rule file with recording rules

Rates and error ratio per job.

groups:
  - name: orders-api-recording
    interval: 30s
    rules:
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_requests_errors:rate5m
        expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
      - record: job:http_requests_error_ratio:rate5m
        expr: job:http_requests_errors:rate5m / job:http_requests:rate5m

Record what dashboards load most

Start with the queries on your most-viewed dashboards and in your alerts.

Quick check: What is the main benefit of a recording rule?

  • Expensive queries are precomputed, so dashboards and alerts are fast
  • It deletes old data
  • It sends notifications
  • It discovers targets
Answer

Expensive queries are precomputed, so dashboards and alerts are fast — Computed once, read many times.