How It Differs from Traditional APM

Same roots, different payload

Classic APM (application performance monitoring) tools track requests, latency, error rates and resource usage, and distributed tracing links spans across services. LLM observability reuses the same idea of traces made of nested spans, but the interesting data is different. In APM a span is mostly timing plus a few small attributes; in an LLM trace a span carries large text payloads: the full prompt, the system message, retrieved chunks, tool arguments and the model's reply. It also needs LLM-specific fields: model name, parameters such as temperature, input and output token counts, computed cost, and quality scores from users or automated evaluators. Because payloads are big and may contain personal data, storage, masking and retention matter more. The two are complementary: keep your APM for infrastructure health, and add an LLM tool (such as Langfuse or LangSmith) for prompt-level debugging and evaluation.

Do not throw away your existing monitoring

Students sometimes think an LLM observability tool replaces Prometheus, Grafana or Datadog. It does not. CPU, memory, database latency and 5xx rates still belong in your infrastructure monitoring. Use OpenTelemetry trace IDs or shared request IDs so you can jump from an infrastructure alert to the matching LLM trace.

What one LLM span records

An illustrative record (field names vary by tool) showing data that a plain APM span would not normally hold.

{
  "name": "answer-question",
  "type": "generation",
  "model": "my-chat-model",
  "modelParameters": { "temperature": 0.2, "max_tokens": 400 },
  "input": [
    { "role": "system", "content": "Answer only from the context." },
    { "role": "user", "content": "Can I get a refund after 10 days?" }
  ],
  "output": "No. Refunds are allowed within 7 days of purchase.",
  "usage": { "input": 182, "output": 14 },
  "latencyMs": 940,
  "scores": { "faithfulness": 1 }
}

Quick check: Which field is specific to LLM observability rather than classic APM?

• Request duration in milliseconds
• HTTP status code
• Service name
• Input and output token counts for a model call