Lesson 7 / 25

Scrape Configuration

prometheus.yml.

Global settings and scrape jobs

prometheus.yml sets global defaults (scrape_interval, evaluation_interval), lists rule files, points to Alertmanager and defines scrape_configs: each job has a name, targets (static or discovered), and optional settings such as metrics_path, scheme and timeouts. Every target gets an up series that is 1 when the scrape succeeded. Reload the configuration with a SIGHUP or the /-/reload endpoint when enabled, and validate it with promtool check config.

Targets, exporters and discovery

Scrape configuration, exporters and service discovery decide what Prometheus sees.

Three ideas: scrape configuration, exporters, service discovery and relabelling.
Figure 3.1 — Scrape config, exporters and discovery.

A minimal prometheus.yml

Two jobs and Alertmanager.

global:
  scrape_interval: 30s
  evaluation_interval: 30s

rule_files:
  - "rules/*.yml"

alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

scrape_configs:
  - job_name: "orders-api"
    static_configs:
      - targets: ["orders-1:8000", "orders-2:8000"]
        labels:
          env: "prod"
  - job_name: "node"
    static_configs:
      - targets: ["web-1:9100", "web-2:9100"]

Validate before reloading

promtool check config and promtool check rules catch syntax errors before they reach production.

Quick check: What does the up metric indicate?

  • Whether the last scrape of a target succeeded (1) or failed (0)
  • CPU usage
  • Uptime in days
  • The number of alerts
Answer

Whether the last scrape of a target succeeded (1) or failed (0) — A built-in health signal.