How Prometheus Works

Scrape, store, query, alert

The Prometheus server periodically scrapes (HTTP GET) a /metrics endpoint on each target, typically every 15 to 60 seconds, and stores the samples in its local time-series database (TSDB). Targets come from static configuration or service discovery (Kubernetes, cloud APIs, Consul). Prometheus evaluates recording and alerting rules, sends firing alerts to Alertmanager, and answers PromQL queries from its web UI, its HTTP API and Grafana. Short-lived batch jobs can push to a Pushgateway, which Prometheus then scrapes.

The main components

How data flows.

apps / exporters  --(/metrics, pulled every scrape_interval)-->  Prometheus server
                                                                   |  TSDB (local disk)
                                                                   |  rules (recording, alerting)
                                         Alertmanager  <--alerts---|
                                     (email, Slack, PagerDuty)     |
                                                    Grafana  --PromQL queries-->  Prometheus

Pull makes health visible

Because Prometheus scrapes targets, a failed scrape is itself a signal: the up metric becomes 0.

Quick check: How does Prometheus normally collect metrics?

• It pulls them by scraping HTTP endpoints
• Applications push every sample to it
• It reads log files
• It queries databases directly