Lesson 14 / 25
Binary Operators and Vector Matching
Combining two sets of series.
on, ignoring and group_left
Arithmetic and comparison operators between two instant vectors match series with identical label sets. Use on(labels) to match only on some labels or ignoring(labels) to ignore some; for many-to-one matches add group_left (or group_right) and optionally copy labels from the "one" side. A common pattern joins an info metric (value 1, with descriptive labels) to add labels such as a version to other series. Comparison operators filter series unless bool is added.
Matching examples
PromQL.
# memory used as a fraction of the container limit (both share pod and container labels)
sum by (pod, container) (container_memory_working_set_bytes)
/ on (pod, container)
sum by (pod, container) (kube_pod_container_resource_limits{resource="memory"})
# add the app version from an info metric to request rates
sum by (instance) (rate(http_requests_total[5m]))
* on (instance) group_left (version)
app_build_info
# keep only hosts above 90% disk usage
(1 - node_filesystem_avail_bytes / node_filesystem_size_bytes) > 0.9Check labels on both sides first
Run each side separately and compare label sets; most "no data" results are matching problems.
Quick check: What does group_left enable?
- Removing all labels
- Sorting results
- Joining logs
- Many-to-one matching, where several left series match one right series
Answer
Many-to-one matching, where several left series match one right series — Needed for info-metric joins.