Lesson 13 / 35
How Prometheus Works
Explain Prometheus's pull model, service discovery, the up metric and where its local storage fits.
By Rahul Agarwal · 15 years in web development
Pull, store, evaluate
Prometheus is an open-source monitoring system and time-series database, and a graduated CNCF project. Its core loop is simple. Using service discovery (static lists, Kubernetes, cloud APIs, DNS) it finds targets. Every scrape_interval it sends an HTTP GET to each target's /metrics endpoint, parses the text, attaches labels such as job and instance, and appends the samples to its local TSDB on disk.
Pulling has useful side effects: Prometheus knows when a target is unreachable, and records this in an automatic metric up (1 for a successful scrape, 0 for a failed one). Short-lived batch jobs that finish before a scrape can push to a Pushgateway instead, but that is the exception.
Prometheus also evaluates recording rules (precompute expensive queries) and alerting rules on a schedule, sending firing alerts to Alertmanager. A single Prometheus server is not designed for long-term, highly available storage; projects such as Thanos, Cortex, Grafana Mimir and VictoriaMetrics add that, typically fed by remote_write.
One Prometheus evaluation cycle
Service discovery produces a target list, Prometheus scrapes each /metrics endpoint, samples land in the TSDB, rules are evaluated against stored data, and firing alerts go to Alertmanager.
A prometheus.yml
Two jobs: one with a static target list and an extra env label, one that discovers every pod through the Kubernetes API. In practice the pod job also needs relabel_configs to choose which pods and ports to scrape.
global:
scrape_interval: 15s # how often to pull every target
evaluation_interval: 15s # how often to evaluate rules
rule_files:
- "rules/*.yml"
scrape_configs:
- job_name: "checkout"
metrics_path: /metrics
static_configs:
- targets: ["checkout-1:8000", "checkout-2:8000"]
labels:
env: "prod"
- job_name: "kubernetes-pods"
kubernetes_sd_configs:
- role: podAlert on up == 0 first
If a target cannot be scraped, all its other metrics simply stop, and rules like "error rate > 5%" quietly evaluate to nothing. A basic up == 0 alert (and absent() for critical metrics) protects you from the silent failure where monitoring itself is blind.
Quick check: What does the automatically generated `up` metric tell you?
- Whether the last scrape of a target succeeded
- The uptime of the host in seconds
- How many users are currently online
- The number of alerts currently firing
Answer
Whether the last scrape of a target succeeded — Prometheus sets up to 1 when a scrape succeeds and 0 when it fails, which makes it the simplest way to detect a dead or unreachable target.