# Making Inference Faster — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-inference-speed

> Apply the main levers for faster inference: precision, batching, attention kernels and compilation.

## The levers, in order of effort

Start with the cheap wins. Run under `torch.inference_mode()` and keep the model on a GPU if you have one. Load in **bfloat16 or float16** with `dtype`, which halves memory traffic. **Batch** requests so the GPU processes several inputs per forward pass, and sort or bucket inputs by length to reduce padding waste. Choose an efficient **attention implementation** with `attn_implementation`: `"sdpa"` uses PyTorch's fused scaled-dot-product attention and is the default for many architectures, while `"flash_attention_2"` needs the separate flash-attn package and a supported GPU. **Smaller models** are the biggest win of all: a distilled or smaller checkpoint fine-tuned on your task can beat a large general model on speed and cost. Then consider heavier tools: `torch.compile` can speed up repeated forward passes after a warm-up, quantization saves memory, and exporting to ONNX via Hugging Face Optimum suits CPU deployments. Always measure latency and throughput with realistic inputs; a change that helps one batch size can hurt another.

## A request through a serving stack

Most production latency is spent in the middle two stages, so that is where optimisation pays off.

![Four connected stages: an incoming request, a batching queue, the model running on a GPU, and the response returned to the client.](assets/figures/hugging-face/section-9-map.svg) — Figure 9.1: request -> batching queue -> model on GPU -> response

## Fast loading options for a causal LM

Each option is independent; measure after enabling each one.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
tok.padding_side = "left"                     # batched generation

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)

prompts = ["Define recursion.", "What is a mutex?", "Explain DNS briefly."]
inputs = tok(prompts, return_tensors="pt", padding=True).to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=40)
print(tok.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True))
```

## Measure tokens per second, not just total time

For generation, report time to first token and tokens per second separately. A long answer can take more total time while the system is actually faster per token.

**Quiz:** Which change is usually the largest and cheapest speed win for a fixed classification task?

- [ ] Raising max_new_tokens
- [ ] Switching from bfloat16 to float32
- [x] Using a smaller model fine-tuned for the task
- [ ] Turning off batching

*Answer:* Using a smaller model fine-tuned for the task. A small task-specific model does far less computation per request. float32 is slower than bfloat16, and turning off batching reduces throughput.