Tracking Token Usage and Cost

Where the money goes

Most hosted model APIs charge per token, with separate prices for input (prompt) and output (completion) tokens; output tokens usually cost more. Some providers also price cached input tokens or reasoning tokens differently. Cost per call is therefore input_tokens x input_price + output_tokens x output_price. Observability tools compute this for you from a built-in or custom model price table: Langfuse and LangSmith both infer cost for many known models and let you define prices for custom or self-hosted ones, or you can send the cost yourself. The value comes from attribution: cost per trace, per user, per feature (via tags) and per model. Typical surprises it reveals: a long system prompt sent on every turn, a chat history that grows without trimming, an agent that loops, or a retry storm. Prices change often, so treat any table as configuration to review, not a constant in code.

Computing cost per call

Prices below are invented for the example; always use your provider's current price list.

# Illustrative prices in USD per 1M tokens -- NOT real list prices.
PRICES = {
    "model-small": {"input": 0.15, "output": 0.60},
    "model-large": {"input": 2.50, "output": 10.00},
}

def call_cost(model, input_tokens, output_tokens):
    p = PRICES[model]
    return (input_tokens * p["input"] + output_tokens * p["output"]) / 1_000_000

calls = [("model-large", 1200, 300), ("model-small", 800, 150), ("model-large", 3000, 500)]
for model, inp, out in calls:
    print(f"{model:<12} {call_cost(model, inp, out):.6f}")
total = sum(call_cost(*c) for c in calls)
print(f"total USD    {total:.6f}")
print(f"per 10k requests: {total / len(calls) * 10_000:.2f}")

Output:

model-large  0.006000
model-small  0.000210
model-large  0.012500
total USD    0.018710
per 10k requests: 62.37

From tokens to a cost dashboard

Usage is recorded per generation, multiplied by a price table, rolled up per trace, and finally grouped by user, feature or model.

Input tokens are often the bigger bill

Students focus on long answers, but in chat and RAG apps the prompt is often several times larger than the reply: system prompt, history and retrieved chunks are re-sent every turn. Check the input/output split before optimising.

त्वरित जाँच: A model call uses 2,000 input tokens and 500 output tokens. Input costs 1 unit per 1M tokens and output 4 units per 1M tokens. What is the cost?

• 0.002 units
• 0.0025 units
• 0.004 units
• 0.010 units