Latency, Streaming and Time to First Token

Measure what the user feels

LLM latency has two numbers that matter. Total latency is from request to the last token. Time to first token (TTFT) is from request until the first piece of the reply appears; with streaming, the user starts reading at TTFT, so it shapes perceived speed more than total time. Output length drives total time because tokens are generated one after another, so a prompt that asks for shorter answers is also a speed fix. Never judge latency by the average: one slow call hides among many fast ones and the mean is pulled by outliers. Use percentiles: p50 (the median) shows the typical experience and p95 or p99 shows the slow tail that generates complaints. Traces then explain the tail: open a p99 trace and the span timeline shows whether time went to retrieval, a slow tool, a retry after a rate-limit error, or a very long generation.

Percentiles beat the mean

Ten latencies with one outlier. Python's statistics module computes the cut points.

import statistics

latencies_ms = [820, 910, 760, 1500, 880, 940, 4200, 870, 990, 1020]
cuts = statistics.quantiles(latencies_ms, n=100, method="inclusive")
print("mean", round(statistics.mean(latencies_ms)))
print("p50 ", round(statistics.median(latencies_ms)))
print("p95 ", round(cuts[94]))

Output:

mean 1289
p50  925
p95  2985

Measuring TTFT on a stream

A simulated stream: the gap before the first token is the time to first token.

import time

def fake_stream():
    time.sleep(0.30)                      # model "thinks" before the first token
    for tok in ["Namaste", ",", " how", " can", " I", " help", "?"]:
        time.sleep(0.02)
        yield tok

start = time.perf_counter()
first = None
parts = []
for tok in fake_stream():
    if first is None:
        first = time.perf_counter() - start
    parts.append(tok)
total = time.perf_counter() - start
print("reply:", "".join(parts))
print(f"TTFT  ~{first:.1f} s")
print(f"total ~{total:.1f} s")

Output:

reply: Namaste, how can I help?
TTFT  ~0.3 s
total ~0.4 s

त्वरित जाँच: Why is p95 latency more useful than the mean for an LLM endpoint?

• It is cheaper to compute
• It shows the slow tail that a meaningful share of users actually experience
• It is always lower than the mean
• It ignores streaming responses