</>SkillByAI
🇬🇧 English
Test our app

LLM Observability (Langfuse / LangSmith) interview questions

31 questions from the LLM Observability (Langfuse / LangSmith) course, each with the correct answer and an explanation. Try to answer before you open the answer.

Why LLM Observability Matters

1. Why do ordinary error-rate dashboards miss many LLM failures?

  1. LLM providers hide error codes from clients
  2. A wrong or unhelpful answer usually still arrives as a successful response
  3. LLM calls are too fast to be logged
  4. Error dashboards cannot count HTTP requests to external APIs
Show answer

Answer: A wrong or unhelpful answer usually still arrives as a successful response

Hallucinations, refusals and off-topic answers come back as normal successful responses, so you need to look at content and quality, not only status codes.

Lesson: Why LLM Apps Need Their Own Observability

2. Which field is specific to LLM observability rather than classic APM?

  1. Request duration in milliseconds
  2. HTTP status code
  3. Service name
  4. Input and output token counts for a model call
Show answer

Answer: Input and output token counts for a model call

Duration, status and service name are standard APM fields. Token usage is LLM-specific and drives both cost and context-window limits.

Lesson: How It Differs from Traditional APM

3. A team sees answer quality drop after a deploy but cannot tell what changed. Which missing signal hurts most?

  1. The prompt version and model configuration on each trace
  2. The CPU usage of the web server
  3. The number of HTTP requests per minute
  4. The size of the Docker image
Show answer

Answer: The prompt version and model configuration on each trace

Configuration signals (prompt version, model, parameters) let you group traces before and after a change and pinpoint the cause.

Lesson: What to Capture: The Core Signals

The Core Data Model

4. In a trace tree, what is the relationship between a retrieval span and an LLM span that both run directly inside the root span?

  1. The LLM span is the parent of the retrieval span
  2. They belong to two different traces
  3. They are siblings: both are children of the root span
  4. The retrieval span is the root
Show answer

Answer: They are siblings: both are children of the root span

Both were opened while the root was the current span, so both record the root as their parent, making them siblings.

Lesson: Traces, Spans and Observations

5. Why should model calls be recorded as generations (or llm runs) instead of plain spans?

  1. Plain spans cannot have inputs or outputs
  2. Only generations can be nested inside a trace
  3. Generations are stored for free while spans are billed
  4. Token usage and cost analytics aggregate over model-call records with model and usage fields
Show answer

Answer: Token usage and cost analytics aggregate over model-call records with model and usage fields

Cost and token dashboards rely on the model and usage fields that the generation type carries. Plain spans still support input and output, but lack those semantics.

Lesson: Generations: Recording Model Calls

6. Which value is the best choice for a trace's user ID?

  1. The user's email address
  2. A stable internal user identifier
  3. The user's phone number
  4. A new random value for every request
Show answer

Answer: A stable internal user identifier

A stable internal ID lets you group a user's traces without sending personal data to the observability backend. A random per-request value defeats grouping.

Lesson: Sessions, Users, Tags and Metadata

Langfuse Fundamentals

7. Why might a company choose to self-host Langfuse rather than use the cloud service?

  1. Self-hosting is the only way to use the Python SDK
  2. The cloud version cannot record token usage
  3. Trace data containing sensitive information must stay inside its own infrastructure
  4. Self-hosting removes the need for API keys
Show answer

Answer: Trace data containing sensitive information must stay inside its own infrastructure

Data residency and privacy are the usual reasons. The SDK, token tracking and API keys work the same way with both options.

Lesson: Langfuse: Cloud or Self-Hosted Setup

8. Function `answer()` and function `retrieve()` both have `@observe()`, and `answer()` calls `retrieve()`. What does the trace look like?

  1. One trace with `answer` as the parent and `retrieve` nested inside it
  2. Two separate traces, one per function
  3. One flat list with no parent-child relationship
  4. Only `answer` is recorded because nested decorators are ignored
Show answer

Answer: One trace with `answer` as the parent and `retrieve` nested inside it

The SDK tracks the current observation in context, so the inner decorated call becomes a child of the outer one, and the outermost call defines the trace.

Lesson: Langfuse: The @observe Decorator

9. A nightly batch script uses Langfuse but only some traces show up. What is the most likely cause?

  1. The script uses too many decorators
  2. Langfuse only records traces during business hours
  3. Batch scripts must use LangSmith instead
  4. The process exits before the background queue is flushed
Show answer

Answer: The process exits before the background queue is flushed

The SDK batches events in the background; a short-lived process must call `flush()` before it exits or the final batches are lost.

Lesson: Langfuse: Manual Spans, Scores and Flushing

LangSmith Fundamentals

10. Which statement about LangSmith is correct?

  1. It can only trace applications built with LangChain
  2. It can trace plain Python or JS/TS code through its SDK, not only LangChain apps
  3. It requires every run to be of type `llm`
  4. It stores traces only on the developer's laptop
Show answer

Answer: It can trace plain Python or JS/TS code through its SDK, not only LangChain apps

The `langsmith` SDK's `@traceable` decorator and client wrappers work with any code; LangChain integration is a convenience, not a requirement.

Lesson: LangSmith: Projects and Setup

11. You want LangSmith to display retrieved documents nicely for a vector-search function. Which run type fits?

  1. llm
  2. parser
  3. retriever
  4. prompt
Show answer

Answer: retriever

The `retriever` run type tells LangSmith the output is a list of documents, so it renders their content and metadata.

Lesson: LangSmith: @traceable and Client Wrappers

12. Your LangGraph agent is traced automatically, but a direct database query inside a helper function does not appear. What should you do?

  1. Disable LangGraph tracing and log everything manually
  2. Switch the project name
  3. Increase the trace sampling rate
  4. Wrap the helper with `@traceable` so it joins the current trace
Show answer

Answer: Wrap the helper with `@traceable` so it joins the current trace

Auto tracing only covers framework components. Decorating your own function makes it a child run of the currently active trace.

Lesson: Automatic Tracing for LangChain and LangGraph

Instrumenting Real Applications

13. A trace shows the correct policy chunk in the retrieval span, yet the answer contradicts it. Where should you focus first?

  1. Generation: the prompt, context ordering or model choice
  2. Retrieval: the embedding model
  3. Chunking: the chunk size is too large
  4. The vector database's index type
Show answer

Answer: Generation: the prompt, context ordering or model choice

Retrieval did its job because the right chunk was present. The failure happened when the model used the context, so look at the prompt and generation step.

Lesson: Tracing a RAG Pipeline

14. An agent's traces show the same search tool called eight times with nearly identical arguments. What does this most likely indicate?

  1. The tracing SDK is duplicating spans
  2. The agent is looping without making progress; check tool results and prompt instructions
  3. The tool is working perfectly
  4. Token counting is broken
Show answer

Answer: The agent is looping without making progress; check tool results and prompt instructions

Repeated near-identical tool calls are a classic agent loop. Look at what the tool returned and whether the model is told how to proceed when results are empty or unhelpful.

Lesson: Tracing Agents and Tool Calls

15. What is the main benefit of instrumenting with OpenTelemetry GenAI conventions?

  1. It makes model calls faster
  2. It removes the need to store prompts
  3. It guarantees that attribute names will never change
  4. The same instrumentation can export to different observability backends
Show answer

Answer: The same instrumentation can export to different observability backends

OTel decouples instrumentation from the backend. The GenAI conventions are still evolving, so names can change, and they do not affect model speed.

Lesson: OpenTelemetry and GenAI Semantic Conventions

Cost, Latency and Analytics

16. A model call uses 2,000 input tokens and 500 output tokens. Input costs 1 unit per 1M tokens and output 4 units per 1M tokens. What is the cost?

  1. 0.002 units
  2. 0.0025 units
  3. 0.004 units
  4. 0.010 units
Show answer

Answer: 0.004 units

2,000 x 1 / 1,000,000 = 0.002, plus 500 x 4 / 1,000,000 = 0.002, giving 0.004 units.

Lesson: Tracking Token Usage and Cost

17. Why is p95 latency more useful than the mean for an LLM endpoint?

  1. It is cheaper to compute
  2. It shows the slow tail that a meaningful share of users actually experience
  3. It is always lower than the mean
  4. It ignores streaming responses
Show answer

Answer: It shows the slow tail that a meaningful share of users actually experience

A few slow outliers distort the mean, while p95 directly states that 5% of requests are slower than this value, which is what users complain about.

Lesson: Latency, Streaming and Time to First Token

18. Which alert is most useful for an LLM support bot?

  1. Alert whenever any single answer gets a thumbs-down
  2. Alert when the number of tags changes
  3. Alert every time a new trace is created
  4. Alert when average user-feedback score drops sharply compared with the previous week
Show answer

Answer: Alert when average user-feedback score drops sharply compared with the previous week

Trend-based alerts on user-visible symptoms catch real regressions. Single thumbs-downs are normal noise and belong in a review queue, not a pager.

Lesson: Dashboards, Filters and Alerts

Evaluation and Feedback

19. What must a feedback event include so it can be attached to the right answer?

  1. The ID of the trace or run that produced the answer
  2. The user's full chat history
  3. The model's API key
  4. The time the frontend was deployed
Show answer

Answer: The ID of the trace or run that produced the answer

Feedback is useful only when linked to the exact trace or run, which is why apps pass the trace or run ID to the frontend.

Lesson: User Feedback and Scores

20. Which practice makes an LLM judge more reliable?

  1. Asking for a free-form opinion with no scale
  2. Always using the same model that generated the answer, at high temperature
  3. Hiding the retrieved context from a faithfulness judge
  4. A specific rubric with a small discrete scale, validated against human labels
Show answer

Answer: A specific rubric with a small discrete scale, validated against human labels

Clear rubrics, discrete scales and agreement checks with humans reduce noise. A faithfulness judge needs the context to check claims against it.

Lesson: LLM-as-a-Judge Evaluators

21. What is the main purpose of running an experiment on a fixed dataset before deploying a new prompt?

  1. To reduce the token price charged by the provider
  2. To catch regressions by comparing scores with the previous version on the same examples
  3. To create user sessions
  4. To replace production tracing
Show answer

Answer: To catch regressions by comparing scores with the previous version on the same examples

Running both versions on the same examples gives a fair comparison and shows which cases improved or broke.

Lesson: Datasets, Experiments and Regression Tests

22. Which is a strength of online evaluation that offline evaluation lacks?

  1. It gives a perfectly controlled comparison between two versions
  2. It is free to run
  3. It reveals how the system handles real, previously unseen user inputs
  4. It works without any traces
Show answer

Answer: It reveals how the system handles real, previously unseen user inputs

Online evaluation scores real traffic, including new question types and drift. Controlled comparisons are the strength of offline experiments.

Lesson: Online vs Offline Evaluation

Prompt Management and Experiments

23. In Langfuse prompt management, what does moving the `production` label to version 7 do?

  1. Deletes versions 1 to 6
  2. Requires redeploying the application
  3. Makes apps that fetch the production label start using version 7, without a code change
  4. Renames the prompt
Show answer

Answer: Makes apps that fetch the production label start using version 7, without a code change

Labels are pointers to versions. Apps fetching by label pick up the new version (after their cache refreshes) while old versions remain available for rollback.

Lesson: Prompt Management and Versioning

24. Why should canary variant assignment be sticky per session?

  1. So a single conversation does not switch prompts mid-way, keeping behaviour and comparisons consistent
  2. Because observability tools reject random assignment
  3. To reduce token cost
  4. So that every user receives the canary
Show answer

Answer: So a single conversation does not switch prompts mid-way, keeping behaviour and comparisons consistent

Switching variants within a conversation confuses users and mixes two versions inside one session's traces, making the comparison unreliable.

Lesson: Comparing Prompt and Model Versions

Production: Privacy, Performance and Pitfalls

25. Where is the safest place to remove phone numbers from trace payloads?

  1. In the observability UI, by hiding the column
  2. In a weekly cleanup job on the backend database
  3. In the SDK's masking function, before data leaves the application
  4. In the model's system prompt
Show answer

Answer: In the SDK's masking function, before data leaves the application

Masking in the SDK stops the raw values from ever being sent or stored. The other options leave the data stored somewhere first.

Lesson: Privacy, Masking and Data Retention

26. Why should the sampling decision be made per trace rather than per span?

  1. Spans cannot be sampled at all
  2. Per-span sampling increases token cost
  3. Per-trace sampling is required by the HTTP specification
  4. Per-span sampling produces incomplete trees where some steps of a request are missing
Show answer

Answer: Per-span sampling produces incomplete trees where some steps of a request are missing

If each span is sampled independently, kept traces lose random steps, which makes them misleading for debugging. Decide once per trace and apply it to every span.

Lesson: Performance Overhead and Sampling

27. Spans from a background worker show up as separate traces instead of under the original request. What is the likely cause?

  1. Trace context was not propagated to the worker
  2. The model returned too many tokens
  3. The project has too many tags
  4. The masking function removed the spans
Show answer

Answer: Trace context was not propagated to the worker

Spans attach to whatever trace context is current. In a separate worker or service that context is absent unless you pass and re-attach it.

Lesson: Common Pitfalls in Production

28. A hospital startup must keep all patient conversations inside its own cloud account. Which factor should weigh most in choosing an LLM observability tool?

  1. The number of chart types in the dashboard
  2. Whether the tool can be self-hosted within its own infrastructure
  3. Whether the tool has a dark mode
  4. Which tool is mentioned most on social media
Show answer

Answer: Whether the tool can be self-hosted within its own infrastructure

Data residency is a hard requirement here, so a self-hostable option (or a contract that guarantees it) comes first. The other factors are preferences.

Lesson: Choosing Between Langfuse, LangSmith and Others

Revision and Interview Prep

29. Which pairing of term and meaning is correct?

  1. Session: a single model call
  2. Generation: the root of every trace
  3. Trace: one end-to-end request made of nested spans
  4. Dataset: a live stream of production requests
Show answer

Answer: Trace: one end-to-end request made of nested spans

A trace is one request's tree of spans. A session groups many traces, a generation is a model-call span, and a dataset is a fixed set of examples.

Lesson: Cheat Sheet: LLM Observability in One Page

30. An interviewer asks how you would reduce the cost of a chat feature. What is the best first step?

  1. Switch every request to the smallest model available
  2. Use traces to attribute cost by feature, step and token type to find where money actually goes
  3. Turn off tracing to save money
  4. Shorten every answer to one sentence
Show answer

Answer: Use traces to attribute cost by feature, step and token type to find where money actually goes

Measure before optimising. Traces show whether cost comes from long prompts, history, retries or agent loops, and that decides the right fix.

Lesson: Common Interview Questions

31. A serverless function traced with Langfuse shows spans nested correctly in local tests, but in production no traces arrive and no errors are raised. Which fix should you try first?

  1. Switch from @observe to manual spans
  2. Increase the model's max tokens
  3. Remove all tags and metadata
  4. Ensure tracing keys are set in production and call flush() before the handler returns
Show answer

Answer: Ensure tracing keys are set in production and call flush() before the handler returns

Silent loss in production usually means missing configuration or unflushed background batches when the runtime freezes the process.

Lesson: Mixed Self-Check

← Back to the course