Cheat Sheet: LLM Observability in One Page

The one-page summary

Why: LLM failures are usually successful responses with wrong content, and outputs are non-deterministic, so you must record inputs, outputs and context. Data model: a trace is one request; it holds nested spans (Langfuse observations, LangSmith runs); a model call is a generation or llm run with model, parameters, messages, token usage and cost; sessions group a conversation; user ID, tags and metadata enable filtering. Langfuse: open source, cloud or self-hosted; @observe(), langfuse.openai drop-in, start_as_current_span/generation, create_score, get_prompt with labels, flush(). LangSmith: managed by the LangChain team; LANGSMITH_TRACING, @traceable(run_type=...), wrap_openai, auto tracing for LangChain and LangGraph, create_feedback, datasets and evaluate. Metrics: tokens and cost by feature, p50/p95 latency and TTFT, feedback and judge scores by version. Production: mask in the SDK, sample per trace, propagate context, separate environments, validate judges.

The full loop at a glance

Everything in this course fits one loop: instrument the app, trace requests, measure cost and latency, evaluate quality, and improve prompts or code.

Terms examiners and interviewers mix up

Trace vs span: a trace is the whole request, a span is one step. Generation vs span: a generation is a span for a model call with usage. Offline vs online evaluation: fixed dataset before release vs live traffic after. Feedback vs score: user reaction vs any attached quality value (feedback is one source of scores).

त्वरित जाँच: Which pairing of term and meaning is correct?

• Session: a single model call
• Generation: the root of every trace
• Trace: one end-to-end request made of nested spans
• Dataset: a live stream of production requests