Lesson 13 / 35

How Prometheus Works

Explain Prometheus's pull model, service discovery, the up metric and where its local storage fits.

By · 15 years in web development

Pull, store, evaluate

Prometheus is an open-source monitoring system and time-series database, and a graduated CNCF project. Its core loop is simple. Using service discovery (static lists, Kubernetes, cloud APIs, DNS) it finds targets. Every scrape_interval it sends an HTTP GET to each target's /metrics endpoint, parses the text, attaches labels such as job and instance, and appends the samples to its local TSDB on disk.

Pulling has useful side effects: Prometheus knows when a target is unreachable, and records this in an automatic metric up (1 for a successful scrape, 0 for a failed one). Short-lived batch jobs that finish before a scrape can push to a Pushgateway instead, but that is the exception.

Prometheus also evaluates recording rules (precompute expensive queries) and alerting rules on a schedule, sending firing alerts to Alertmanager. A single Prometheus server is not designed for long-term, highly available storage; projects such as Thanos, Cortex, Grafana Mimir and VictoriaMetrics add that, typically fed by remote_write.

One Prometheus evaluation cycle

Service discovery produces a target list, Prometheus scrapes each /metrics endpoint, samples land in the TSDB, rules are evaluated against stored data, and firing alerts go to Alertmanager.

Five stages in order: service discovery finds targets, Prometheus scrapes them, samples are stored in the TSDB, rules are evaluated, alerts are sent to Alertmanager.
Figure 5.1: service discovery -> scrape -> TSDB -> rules -> Alertmanager

A prometheus.yml

Two jobs: one with a static target list and an extra env label, one that discovers every pod through the Kubernetes API. In practice the pod job also needs relabel_configs to choose which pods and ports to scrape.

global:
  scrape_interval: 15s       # how often to pull every target
  evaluation_interval: 15s   # how often to evaluate rules

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: "checkout"
    metrics_path: /metrics
    static_configs:
      - targets: ["checkout-1:8000", "checkout-2:8000"]
        labels:
          env: "prod"

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod

Alert on up == 0 first

If a target cannot be scraped, all its other metrics simply stop, and rules like "error rate > 5%" quietly evaluate to nothing. A basic up == 0 alert (and absent() for critical metrics) protects you from the silent failure where monitoring itself is blind.

Quick check: What does the automatically generated `up` metric tell you?

  • Whether the last scrape of a target succeeded
  • The uptime of the host in seconds
  • How many users are currently online
  • The number of alerts currently firing
Answer

Whether the last scrape of a target succeeded — Prometheus sets up to 1 when a scrape succeeds and 0 when it fails, which makes it the simplest way to detect a dead or unreachable target.