# Common Pitfalls in Production — LLM Observability (Langfuse / LangSmith)

Source: https://www.skillbyai.com/en/llm-observability/lo-pitfalls

> Recognise the mistakes that make LLM traces missing, misleading or risky, and how to avoid each.

## Mistakes teams repeat

A short list of problems seen again and again. **Lost traces**: no flush in serverless or scripts, or tracing disabled by a missing environment variable in production. **Broken trees**: work started in a new thread, task queue or another service loses the trace context, so spans appear as separate traces; propagate context (OpenTelemetry headers or passing the trace ID) explicitly. **No version info**: traces lack prompt version, model or release, so regressions cannot be pinned. **Secrets and personal data** in payloads because whole request objects were logged. **Noise**: every helper decorated, so traces are thousands of spans deep. **Mixed environments**: dev and prod in one project, corrupting metrics. **Unvalidated judges** whose scores drive decisions. **Ignoring cost of observability**: tracing large payloads such as full documents or images at 100%. Most are fixed by a short checklist reviewed whenever a new LLM feature ships.

## Propagate context across async boundaries

Python context variables are copied into new `asyncio` tasks (and `asyncio.to_thread`), but not into threads started with `ThreadPoolExecutor.submit` or `loop.run_in_executor`, and never into a Celery job or another process. Pass the trace ID along and re-attach it, or use OpenTelemetry propagation, so background work joins the right trace.

## A pre-launch checklist as config

A team checklist kept in the repository so every new LLM feature is reviewed the same way.

```yaml
llm_observability_checklist:
  tracing_enabled_in_prod: true
  flush_on_shutdown: true          # serverless, cron, CLI
  separate_project_per_env: true
  trace_attributes: [user_id, session_id, release, prompt_version]
  masking_function: enabled
  retention_days: 30
  sampling:
    default_rate: 0.2
    always_keep: [errors, negative_feedback]
  online_evaluators: [faithfulness, policy_check]
  alerts: [p95_latency, daily_cost, feedback_drop]
```

**Quiz:** Spans from a background worker show up as separate traces instead of under the original request. What is the likely cause?

- [x] Trace context was not propagated to the worker
- [ ] The model returned too many tokens
- [ ] The project has too many tags
- [ ] The masking function removed the spans

*Answer:* Trace context was not propagated to the worker. Spans attach to whatever trace context is current. In a separate worker or service that context is absent unless you pass and re-attach it.