What to Capture: The Core Signals

Five families of signals

Think of the data you capture in five families. Content: prompts, messages, retrieved context, tool calls and final output; you need these to debug a single bad answer. Configuration: model name, temperature, max tokens and the prompt version used; without them you cannot tell which change caused a regression. Usage and cost: input and output tokens per call, turned into money using a price table. Performance: total latency, time to first token for streaming, and per-step timing so you can see whether retrieval or generation is slow. Quality: user feedback (thumbs up or down), automated evaluator scores and labels from human review. Finally, attach context such as user ID, session ID, environment and release tag so you can filter and group. Each signal should answer a real question; capturing everything with no plan creates cost and privacy risk without insight.

A doctor's case file

A good doctor's file holds symptoms (content), prescriptions and dosage (configuration), bills (cost), how long recovery took (performance) and whether the patient felt better (quality). Missing any one makes the next visit guesswork.

Always record the prompt version

The most common blind spot is not knowing which prompt produced an answer. If prompts live in code, log a version string or commit hash in trace metadata. If you use a prompt-management feature, link the prompt object to the generation so the tool records it for you.

Quick check: A team sees answer quality drop after a deploy but cannot tell what changed. Which missing signal hurts most?

• The prompt version and model configuration on each trace
• The CPU usage of the web server
• The number of HTTP requests per minute
• The size of the Docker image