# How Prometheus Works — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-prometheus-model

> Explain Prometheus's pull model, service discovery, the up metric and where its local storage fits.

## Pull, store, evaluate

**Prometheus** is an open-source monitoring system and time-series database, and a graduated CNCF project. Its core loop is simple. Using **service discovery** (static lists, Kubernetes, cloud APIs, DNS) it finds **targets**. Every `scrape_interval` it sends an HTTP GET to each target's `/metrics` endpoint, parses the text, attaches labels such as `job` and `instance`, and appends the samples to its local **TSDB** on disk.

Pulling has useful side effects: Prometheus knows when a target is unreachable, and records this in an automatic metric `up` (1 for a successful scrape, 0 for a failed one). Short-lived batch jobs that finish before a scrape can push to a **Pushgateway** instead, but that is the exception.

Prometheus also evaluates **recording rules** (precompute expensive queries) and **alerting rules** on a schedule, sending firing alerts to **Alertmanager**. A single Prometheus server is not designed for long-term, highly available storage; projects such as Thanos, Cortex, Grafana Mimir and VictoriaMetrics add that, typically fed by `remote_write`.

## One Prometheus evaluation cycle

**Service discovery** produces a target list, Prometheus **scrapes** each `/metrics` endpoint, samples land in the **TSDB**, **rules** are evaluated against stored data, and firing alerts go to **Alertmanager**.

![Five stages in order: service discovery finds targets, Prometheus scrapes them, samples are stored in the TSDB, rules are evaluated, alerts are sent to Alertmanager.](assets/figures/observability/section-5-map.svg) — Figure 5.1: service discovery -> scrape -> TSDB -> rules -> Alertmanager

## A prometheus.yml

Two jobs: one with a static target list and an extra `env` label, one that discovers every pod through the Kubernetes API. In practice the pod job also needs `relabel_configs` to choose which pods and ports to scrape.

```yaml
global:
  scrape_interval: 15s       # how often to pull every target
  evaluation_interval: 15s   # how often to evaluate rules

rule_files:
  - "rules/*.yml"

scrape_configs:
  - job_name: "checkout"
    metrics_path: /metrics
    static_configs:
      - targets: ["checkout-1:8000", "checkout-2:8000"]
        labels:
          env: "prod"

  - job_name: "kubernetes-pods"
    kubernetes_sd_configs:
      - role: pod
```

## Alert on up == 0 first

If a target cannot be scraped, all its other metrics simply stop, and rules like "error rate > 5%" quietly evaluate to nothing. A basic `up == 0` alert (and `absent()` for critical metrics) protects you from the silent failure where monitoring itself is blind.

**Quiz:** What does the automatically generated `up` metric tell you?

- [x] Whether the last scrape of a target succeeded
- [ ] The uptime of the host in seconds
- [ ] How many users are currently online
- [ ] The number of alerts currently firing

*Answer:* Whether the last scrape of a target succeeded. Prometheus sets up to 1 when a scrape succeeds and 0 when it fails, which makes it the simplest way to detect a dead or unreachable target.
