# Monitoring Saturation — Cost & Capacity Estimation

Source: https://www.skillbyai.com/en/capacity-estimation/ce-capacity-monitoring

> Track utilisation, saturation and errors per resource so estimates stay accurate after launch.

## Measure what you estimated

After launch, monitoring replaces assumptions with data. A useful checklist is the **USE method**: for every resource (CPU, memory, disk, network, connection pools, queues), track **Utilisation** (how busy it is), **Saturation** (how much work is waiting, such as run-queue length, I/O wait or queue depth) and **Errors**. Pair it with request-level signals: traffic, latency percentiles and error rate. For capacity, the key graphs are **peak utilisation over weeks**, which shows trend and headroom, and **saturation signals**, which warn before latency degrades. Alert on sustained trends (for example, peak usage forecast to cross a threshold in a few weeks) rather than only on momentary spikes. Finally, feed observed numbers back into the estimation model: real bytes per object, real QPS per server and real peak factors make the next estimate far more accurate.

## A Prometheus alert on disk headroom

This Prometheus alerting rule uses the PromQL function predict_linear and fires if a filesystem is predicted to fill within four days based on the last six hours of growth. It uses node_exporter metric names.

```yaml
groups:
  - name: capacity
    rules:
      - alert: DiskWillFillIn4Days
        expr: predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs"}[6h], 4 * 24 * 3600) < 0
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Filesystem on {{ $labels.instance }} predicted to fill within 4 days"
```

## Average CPU hides hotspots

A cluster at 40% average CPU can still have one hot shard at 95%. Look at the maximum and the distribution across nodes, not only the fleet average.

**Quiz:** In the USE method, what does 'Saturation' measure?

- [ ] The fraction of time a resource is busy
- [ ] The number of failed operations
- [ ] The total cost of a resource
- [x] The amount of work waiting because the resource cannot keep up

*Answer:* The amount of work waiting because the resource cannot keep up. Saturation is queued or waiting work, such as run-queue length or queue depth. Utilisation is busy time and Errors counts failures.
