# Cheat Sheet: LLM Observability in One Page — LLM Observability (Langfuse / LangSmith)

Source: https://www.skillbyai.com/en/llm-observability/lo-cheatsheet

> Recall the core vocabulary, SDK entry points and production rules of LLM observability at a glance.

## The one-page summary

**Why**: LLM failures are usually successful responses with wrong content, and outputs are non-deterministic, so you must record inputs, outputs and context. **Data model**: a *trace* is one request; it holds nested *spans* (Langfuse *observations*, LangSmith *runs*); a model call is a *generation* or `llm` run with model, parameters, messages, token usage and cost; *sessions* group a conversation; *user ID*, *tags* and *metadata* enable filtering. **Langfuse**: open source, cloud or self-hosted; `@observe()`, `langfuse.openai` drop-in, `start_as_current_span/generation`, `create_score`, `get_prompt` with labels, `flush()`. **LangSmith**: managed by the LangChain team; `LANGSMITH_TRACING`, `@traceable(run_type=...)`, `wrap_openai`, auto tracing for LangChain and LangGraph, `create_feedback`, datasets and `evaluate`. **Metrics**: tokens and cost by feature, p50/p95 latency and TTFT, feedback and judge scores by version. **Production**: mask in the SDK, sample per trace, propagate context, separate environments, validate judges.

## The full loop at a glance

Everything in this course fits one loop: instrument the app, trace requests, measure cost and latency, evaluate quality, and improve prompts or code.

![Five connected stages: instrument, trace, measure, evaluate, improve.](assets/figures/llm-observability/section-10-map.svg) — Figure 10.1: instrument -> trace -> measure cost and latency -> evaluate quality -> improve

## Terms examiners and interviewers mix up

Trace vs span: a trace is the whole request, a span is one step. Generation vs span: a generation is a span for a model call with usage. Offline vs online evaluation: fixed dataset before release vs live traffic after. Feedback vs score: user reaction vs any attached quality value (feedback is one source of scores).

**Quiz:** Which pairing of term and meaning is correct?

- [ ] Session: a single model call
- [ ] Generation: the root of every trace
- [x] Trace: one end-to-end request made of nested spans
- [ ] Dataset: a live stream of production requests

*Answer:* Trace: one end-to-end request made of nested spans. A trace is one request's tree of spans. A session groups many traces, a generation is a model-call span, and a dataset is a fixed set of examples.