Service Discovery and Relabelling

Find targets automatically, then shape labels

In dynamic environments, targets come from service discovery: kubernetes_sd_configs, EC2, GCE, Azure, Consul, DNS or file-based discovery. Discovered targets carry metadata labels starting with __meta_. relabel_configs run before scraping to keep or drop targets and turn metadata into useful labels; metric_relabel_configs run after scraping to drop or rewrite series. On Kubernetes, the Prometheus Operator offers ServiceMonitor and PodMonitor resources to declare targets.

Scraping annotated Kubernetes pods

A widely used opt-in pattern.

scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_.*"
        action: drop          # drop series we never use

Drop unused series early

metric_relabel_configs with action: drop reduces storage and cardinality for metrics nobody queries.

Quick check: When do metric_relabel_configs run?

• Only in Grafana
• Before target discovery
• After scraping, on the scraped series
• When alerts fire