SkillByAIOpen interactive version →

Lesson 14 / 25

Binary Operators and Vector Matching

Combining two sets of series.

on, ignoring and group_left

Arithmetic and comparison operators between two instant vectors match series with identical label sets. Use on(labels) to match only on some labels or ignoring(labels) to ignore some; for many-to-one matches add group_left (or group_right) and optionally copy labels from the "one" side. A common pattern joins an info metric (value 1, with descriptive labels) to add labels such as a version to other series. Comparison operators filter series unless bool is added.

Matching examples

PromQL.

# memory used as a fraction of the container limit (both share pod and container labels)
sum by (pod, container) (container_memory_working_set_bytes)
  / on (pod, container)
sum by (pod, container) (kube_pod_container_resource_limits{resource="memory"})

# add the app version from an info metric to request rates
sum by (instance) (rate(http_requests_total[5m]))
  * on (instance) group_left (version)
app_build_info

# keep only hosts above 90% disk usage
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) > 0.9

Check labels on both sides first

Run each side separately and compare label sets; most "no data" results are matching problems.

Quick check: What does group_left enable?

  • Removing all labels
  • Sorting results
  • Joining logs
  • Many-to-one matching, where several left series match one right series
Answer

Many-to-one matching, where several left series match one right series — Needed for info-metric joins.