# Latency, Streaming and Time to First Token — LLM Observability (Langfuse / LangSmith)

Source: https://www.skillbyai.com/en/llm-observability/lo-latency

> Measure total latency, time to first token and percentiles, and find which step is slow.

## Measure what the user feels

LLM latency has two numbers that matter. **Total latency** is from request to the last token. **Time to first token (TTFT)** is from request until the first piece of the reply appears; with streaming, the user starts reading at TTFT, so it shapes perceived speed more than total time. Output length drives total time because tokens are generated one after another, so a prompt that asks for shorter answers is also a speed fix. Never judge latency by the **average**: one slow call hides among many fast ones and the mean is pulled by outliers. Use **percentiles**: p50 (the median) shows the typical experience and **p95 or p99** shows the slow tail that generates complaints. Traces then explain the tail: open a p99 trace and the span timeline shows whether time went to retrieval, a slow tool, a retry after a rate-limit error, or a very long generation.

## Percentiles beat the mean

Ten latencies with one outlier. Python's statistics module computes the cut points.

```python
import statistics

latencies_ms = [820, 910, 760, 1500, 880, 940, 4200, 870, 990, 1020]
cuts = statistics.quantiles(latencies_ms, n=100, method="inclusive")
print("mean", round(statistics.mean(latencies_ms)))
print("p50 ", round(statistics.median(latencies_ms)))
print("p95 ", round(cuts[94]))
```

Output:

```
mean 1289
p50  925
p95  2985
```

## Measuring TTFT on a stream

A simulated stream: the gap before the first token is the time to first token.

```python
import time

def fake_stream():
    time.sleep(0.30)                      # model "thinks" before the first token
    for tok in ["Namaste", ",", " how", " can", " I", " help", "?"]:
        time.sleep(0.02)
        yield tok

start = time.perf_counter()
first = None
parts = []
for tok in fake_stream():
    if first is None:
        first = time.perf_counter() - start
    parts.append(tok)
total = time.perf_counter() - start
print("reply:", "".join(parts))
print(f"TTFT  ~{first:.1f} s")
print(f"total ~{total:.1f} s")
```

Output:

```
reply: Namaste, how can I help?
TTFT  ~0.3 s
total ~0.4 s
```

**Quiz:** Why is p95 latency more useful than the mean for an LLM endpoint?

- [ ] It is cheaper to compute
- [x] It shows the slow tail that a meaningful share of users actually experience
- [ ] It is always lower than the mean
- [ ] It ignores streaming responses

*Answer:* It shows the slow tail that a meaningful share of users actually experience. A few slow outliers distort the mean, while p95 directly states that 5% of requests are slower than this value, which is what users complain about.