Common Pitfalls in Production

Mistakes teams repeat

A short list of problems seen again and again. Lost traces: no flush in serverless or scripts, or tracing disabled by a missing environment variable in production. Broken trees: work started in a new thread, task queue or another service loses the trace context, so spans appear as separate traces; propagate context (OpenTelemetry headers or passing the trace ID) explicitly. No version info: traces lack prompt version, model or release, so regressions cannot be pinned. Secrets and personal data in payloads because whole request objects were logged. Noise: every helper decorated, so traces are thousands of spans deep. Mixed environments: dev and prod in one project, corrupting metrics. Unvalidated judges whose scores drive decisions. Ignoring cost of observability: tracing large payloads such as full documents or images at 100%. Most are fixed by a short checklist reviewed whenever a new LLM feature ships.

Propagate context across async boundaries

Python context variables are copied into new asyncio tasks (and asyncio.to_thread), but not into threads started with ThreadPoolExecutor.submit or loop.run_in_executor, and never into a Celery job or another process. Pass the trace ID along and re-attach it, or use OpenTelemetry propagation, so background work joins the right trace.

A pre-launch checklist as config

A team checklist kept in the repository so every new LLM feature is reviewed the same way.

llm_observability_checklist:
  tracing_enabled_in_prod: true
  flush_on_shutdown: true          # serverless, cron, CLI
  separate_project_per_env: true
  trace_attributes: [user_id, session_id, release, prompt_version]
  masking_function: enabled
  retention_days: 30
  sampling:
    default_rate: 0.2
    always_keep: [errors, negative_feedback]
  online_evaluators: [faithfulness, policy_check]
  alerts: [p95_latency, daily_cost, feedback_drop]

Quick check: Spans from a background worker show up as separate traces instead of under the original request. What is the likely cause?

• Trace context was not propagated to the worker
• The model returned too many tokens
• The project has too many tags
• The masking function removed the spans