# Dashboards, Filters and Alerts — LLM Observability (Langfuse / LangSmith)

Source: https://www.skillbyai.com/en/llm-observability/lo-dashboards

> Build useful dashboards and alerts by slicing traces on tags, metadata and scores.

## From traces to trends

Individual traces are for debugging; **dashboards** are for trends. Both Langfuse and LangSmith provide built-in charts for trace counts, latency percentiles, token usage, cost and scores over time, and both let you filter or group by model, tags, metadata, user and environment. The skill is choosing **slices that map to decisions**: cost per feature tells you what to optimise; p95 latency per model tells you whether a cheaper model is also slower; average feedback score per prompt version tells you whether a release helped. Filters on traces are also a workflow tool: "all traces with negative feedback this week" or "all traces where the agent hit the step limit" become review queues. For **alerts**, use whatever the tool offers or export metrics to your existing alerting stack. Alert on symptoms users notice: error rate, p95 latency, a sudden cost spike or a drop in quality scores, not on every single bad answer.

## Hospital vital-signs monitor

A nurse does not read each heartbeat; the monitor shows trends and beeps when pulse or oxygen leaves a safe range. Then the doctor examines the patient in detail. Dashboards and alerts are the monitor; traces are the detailed examination.

## Beware of high-cardinality metadata

Grouping by a value that is unique per request (like a raw query string or timestamp) makes charts useless. Group by bounded values such as model, feature, prompt version or customer tier, and keep unique values as searchable fields.

**Quiz:** Which alert is most useful for an LLM support bot?

- [ ] Alert whenever any single answer gets a thumbs-down
- [ ] Alert when the number of tags changes
- [ ] Alert every time a new trace is created
- [x] Alert when average user-feedback score drops sharply compared with the previous week

*Answer:* Alert when average user-feedback score drops sharply compared with the previous week. Trend-based alerts on user-visible symptoms catch real regressions. Single thumbs-downs are normal noise and belong in a review queue, not a pager.