Lesson 29 / 31
Cheat Sheet: LLM Observability in One Page
Recall the core vocabulary, SDK entry points and production rules of LLM observability at a glance.
By Rahul Agarwal · 15 years in web development
The one-page summary
Why: LLM failures are usually successful responses with wrong content, and outputs are non-deterministic, so you must record inputs, outputs and context. Data model: a trace is one request; it holds nested spans (Langfuse observations, LangSmith runs); a model call is a generation or llm run with model, parameters, messages, token usage and cost; sessions group a conversation; user ID, tags and metadata enable filtering. Langfuse: open source, cloud or self-hosted; @observe(), langfuse.openai drop-in, start_as_current_span/generation, create_score, get_prompt with labels, flush(). LangSmith: managed by the LangChain team; LANGSMITH_TRACING, @traceable(run_type=...), wrap_openai, auto tracing for LangChain and LangGraph, create_feedback, datasets and evaluate. Metrics: tokens and cost by feature, p50/p95 latency and TTFT, feedback and judge scores by version. Production: mask in the SDK, sample per trace, propagate context, separate environments, validate judges.
The full loop at a glance
Everything in this course fits one loop: instrument the app, trace requests, measure cost and latency, evaluate quality, and improve prompts or code.
Terms examiners and interviewers mix up
Trace vs span: a trace is the whole request, a span is one step. Generation vs span: a generation is a span for a model call with usage. Offline vs online evaluation: fixed dataset before release vs live traffic after. Feedback vs score: user reaction vs any attached quality value (feedback is one source of scores).
Quick check: Which pairing of term and meaning is correct?
- Session: a single model call
- Generation: the root of every trace
- Trace: one end-to-end request made of nested spans
- Dataset: a live stream of production requests
Answer
Trace: one end-to-end request made of nested spans — A trace is one request's tree of spans. A session groups many traces, a generation is a model-call span, and a dataset is a fixed set of examples.