# Storing and Querying Logs — Logging, Metrics, Tracing & Alerting

Source: https://www.skillbyai.com/en/observability/ob-log-storage-query

> Compare full-text indexing with label-based log stores and write basic LogQL queries.

## Two ways to store logs

Log backends make a fundamental trade-off between **ingest cost** and **query speed**.

**Full-text indexing** (Elasticsearch, OpenSearch) builds an inverted index over the content of every log line, like the index at the back of a textbook. Searches for any word are fast, but indexing is CPU- and storage-heavy, and the index can be larger than the raw data.

**Label-based storage** (Grafana Loki is the best-known example) indexes only a small set of **labels** such as `app`, `env` and `namespace`. The log lines themselves are compressed into chunks, often in cheap object storage. A query first selects streams by label, then scans those chunks with filters. Ingest is much cheaper, but a query over a wide time range with weak label filters scans a lot of data.

Whichever you use, set a **retention** policy: keep recent logs searchable for days or weeks, and archive or delete older data. Logs kept forever become a cost and compliance liability.

## LogQL by example

In Loki's LogQL, `{...}` selects streams by label, `|=` filters lines by text, `| json` extracts fields, and range functions like `count_over_time` turn logs into numbers.

```text
# 1. All lines from checkout in prod containing "timeout"
{app="checkout", env="prod"} |= "timeout"

# 2. Parse JSON and keep only server errors
{app="checkout"} | json | status >= 500

# 3. Error lines per minute, grouped by route (a metric derived from logs)
sum by (route) (count_over_time({app="checkout"} | json | status >= 500 [1m]))
```

## Labels are not for IDs

In a label-based store, every unique combination of labels creates a separate **stream**. Putting `user_id` or `request_id` in labels creates millions of tiny streams and hurts performance. Keep IDs in the log body and filter them with `|=` or a parsed field.

**Quiz:** In Loki, why should `request_id` be a field inside the log line rather than a label?

- [ ] Labels can only hold numbers
- [ ] LogQL cannot filter on labels
- [ ] Request IDs are never needed in queries
- [x] Each unique label value creates a new stream, so high-cardinality labels explode the index

*Answer:* Each unique label value creates a new stream, so high-cardinality labels explode the index. Loki indexes label combinations as streams. Unbounded values such as request IDs create huge numbers of streams; keep them in the line and filter at query time.
