Generations: Recording Model Calls

The special span for a model call

Most steps in a trace are generic spans, but a call to a language model deserves a richer record. Langfuse names this observation type a generation; LangSmith uses a run with run_type="llm". A generation stores the model identifier, model parameters (temperature, max tokens, top_p), the input as a list of chat messages, the output message, and usage: input tokens, output tokens and sometimes cached or reasoning tokens. From usage and a price table the tool computes cost. For streaming calls a generation may also record when the first token arrived, which gives you time to first token. Why separate it from a plain span? Because analytics depend on it: cost per user, tokens per model and latency per model all aggregate over generations only. If you log a model call as a plain span with the text in a string, the tool cannot add up tokens or money.

A generation record in Langfuse (manual style)

Uses the Langfuse Python SDK (v3 style). Replace the placeholder reply with your real model call; the drop-in OpenAI wrapper shown later fills these fields automatically.

from langfuse import get_client

langfuse = get_client()  # reads LANGFUSE_PUBLIC_KEY / SECRET_KEY / HOST from env

with langfuse.start_as_current_generation(
    name="draft-reply",
    model="my-chat-model",
    model_parameters={"temperature": 0.2},
    input=[{"role": "user", "content": "Namaste! Where is my order?"}],
) as gen:
    reply = "Your order is out for delivery."   # call your model here
    gen.update(output=reply, usage_details={"input": 14, "output": 8})

langfuse.flush()

Let the tool count tokens when it can

Prefer the usage numbers returned by the provider's API over your own estimate; tokenizers differ by model and your count will drift. If you self-host a model that returns no usage, record your best estimate and label it as such in metadata.

Quick check: Why should model calls be recorded as generations (or llm runs) instead of plain spans?

• Plain spans cannot have inputs or outputs
• Only generations can be nested inside a trace
• Generations are stored for free while spans are billed
• Token usage and cost analytics aggregate over model-call records with model and usage fields