# Mixed Self-Check — LLM Observability (Langfuse / LangSmith)

Source: https://www.skillbyai.com/en/llm-observability/lo-self-check

> Test yourself across tracing, tools, evaluation and production practice, and find topics to revise.

## Ten questions to answer aloud

Try answering each in two or three sentences before checking the relevant section. 1) Why can an LLM endpoint with a 0% error rate still be broken? 2) What is the difference between a span and a generation? 3) What do sessions and user IDs let you do? 4) What does `@observe()` capture, and how does nesting happen? 5) Why must a Lambda function call `flush()`? 6) Which LangSmith run type renders documents, and which shows token counts? 7) Why is p95 more informative than mean latency, and what is TTFT? 8) Name two biases of LLM judges and one mitigation. 9) How do prompt labels let you roll back without a deploy? 10) Why sample per trace, and why mask in the SDK? If any answer felt vague, revisit that topic and rerun its code example; explaining it to a friend is the fastest test of understanding.

## Revise by doing

Create a free account or a local self-hosted instance, trace a tiny script with two nested functions and one model call, add a score, and build one chart. Hands-on traces make every concept in this course concrete.

## A mini lab script

A local exercise without any SDK: record a tiny trace as nested dictionaries and add up its tokens, so the data model sticks.

```python
trace = {
    "name": "chat-request",
    "children": [
        {"name": "retrieve-docs", "children": []},
        {"name": "llm-call", "usage": {"input": 900, "output": 120}, "children": []},
        {"name": "llm-call-retry", "usage": {"input": 900, "output": 110}, "children": []},
    ],
}

def total_tokens(node):
    own = sum(node.get("usage", {}).values())
    return own + sum(total_tokens(c) for c in node["children"])

print("spans:", 1 + len(trace["children"]))
print("tokens:", total_tokens(trace))
```

Output:

```
spans: 4
tokens: 2030
```

**Quiz:** A serverless function traced with Langfuse shows spans nested correctly in local tests, but in production no traces arrive and no errors are raised. Which fix should you try first?

- [ ] Switch from @observe to manual spans
- [ ] Increase the model's max tokens
- [ ] Remove all tags and metadata
- [x] Ensure tracing keys are set in production and call flush() before the handler returns

*Answer:* Ensure tracing keys are set in production and call flush() before the handler returns. Silent loss in production usually means missing configuration or unflushed background batches when the runtime freezes the process.