LLM Observability (Langfuse / LangSmith)

What you'll learn

• Explain why LLM applications need tracing beyond classic logs, metrics and APM.
• Model requests as traces, spans, generations and sessions with useful tags and metadata.
• Instrument Python apps, RAG pipelines and agents with Langfuse, LangSmith and OpenTelemetry.
• Track token usage, cost, latency percentiles and time to first token, and build useful dashboards.
• Evaluate quality with user feedback, LLM judges, datasets and experiments, and roll out prompt versions safely.
• Run observability in production with masking, retention, sampling and correct context propagation.

Syllabus

Why LLM Observability Matters

1. Why LLM Apps Need Their Own Observability
2. How It Differs from Traditional APM
3. What to Capture: The Core Signals

The Core Data Model

1. Traces, Spans and Observations
2. Generations: Recording Model Calls
3. Sessions, Users, Tags and Metadata

Langfuse Fundamentals

1. Langfuse: Cloud or Self-Hosted Setup
2. Langfuse: The @observe Decorator
3. Langfuse: Manual Spans, Scores and Flushing

LangSmith Fundamentals

1. LangSmith: Projects and Setup
2. LangSmith: @traceable and Client Wrappers
3. Automatic Tracing for LangChain and LangGraph

Instrumenting Real Applications

1. Tracing a RAG Pipeline
2. Tracing Agents and Tool Calls
3. OpenTelemetry and GenAI Semantic Conventions

Cost, Latency and Analytics

1. Tracking Token Usage and Cost
2. Latency, Streaming and Time to First Token
3. Dashboards, Filters and Alerts

Evaluation and Feedback

1. User Feedback and Scores
2. LLM-as-a-Judge Evaluators
3. Datasets, Experiments and Regression Tests
4. Online vs Offline Evaluation

Prompt Management and Experiments

1. Prompt Management and Versioning
2. Comparing Prompt and Model Versions

Production: Privacy, Performance and Pitfalls

1. Privacy, Masking and Data Retention
2. Performance Overhead and Sampling
3. Common Pitfalls in Production
4. Choosing Between Langfuse, LangSmith and Others

Revision and Interview Prep

1. Cheat Sheet: LLM Observability in One Page
2. Common Interview Questions
3. Mixed Self-Check

Related courses

• Dify (25 lessons)
• LLM Engineering (29 lessons)
• Prompt Scoring (25 lessons)
• Agent Builder (26 lessons)