# Service Discovery and Relabelling — Prometheus + Grafana

Source: https://www.skillbyai.com/en/prometheus-grafana/c-discovery

> Dynamic targets.

## Find targets automatically, then shape labels

In dynamic environments, targets come from **service discovery**: `kubernetes_sd_configs`, EC2, GCE, Azure, Consul, DNS or file-based discovery. Discovered targets carry metadata labels starting with `__meta_`. **relabel_configs** run before scraping to keep or drop targets and turn metadata into useful labels; **metric_relabel_configs** run after scraping to drop or rewrite series. On Kubernetes, the Prometheus Operator offers ServiceMonitor and PodMonitor resources to declare targets.

## Scraping annotated Kubernetes pods

A widely used opt-in pattern.

```yaml
scrape_configs:
  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: "true"
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: "go_gc_.*"
        action: drop          # drop series we never use
```

## Drop unused series early

metric_relabel_configs with action: drop reduces storage and cardinality for metrics nobody queries.

**Quiz:** When do metric_relabel_configs run?

- [ ] Only in Grafana
- [ ] Before target discovery
- [x] After scraping, on the scraped series
- [ ] When alerts fire

*Answer:* After scraping, on the scraped series. relabel_configs run before scraping.