Making Inference Faster

The levers, in order of effort

Start with the cheap wins. Run under torch.inference_mode() and keep the model on a GPU if you have one. Load in bfloat16 or float16 with dtype, which halves memory traffic. Batch requests so the GPU processes several inputs per forward pass, and sort or bucket inputs by length to reduce padding waste. Choose an efficient attention implementation with attn_implementation: "sdpa" uses PyTorch's fused scaled-dot-product attention and is the default for many architectures, while "flash_attention_2" needs the separate flash-attn package and a supported GPU. Smaller models are the biggest win of all: a distilled or smaller checkpoint fine-tuned on your task can beat a large general model on speed and cost. Then consider heavier tools: torch.compile can speed up repeated forward passes after a warm-up, quantization saves memory, and exporting to ONNX via Hugging Face Optimum suits CPU deployments. Always measure latency and throughput with realistic inputs; a change that helps one batch size can hurt another.

A request through a serving stack

Most production latency is spent in the middle two stages, so that is where optimisation pays off.

Fast loading options for a causal LM

Each option is independent; measure after enabling each one.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-0.5B-Instruct"
tok = AutoTokenizer.from_pretrained(model_id)
tok.padding_side = "left"                     # batched generation

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    dtype=torch.bfloat16,
    device_map="auto",
    attn_implementation="sdpa",
)

prompts = ["Define recursion.", "What is a mutex?", "Explain DNS briefly."]
inputs = tok(prompts, return_tensors="pt", padding=True).to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=40)
print(tok.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Measure tokens per second, not just total time

For generation, report time to first token and tokens per second separately. A long answer can take more total time while the system is actually faster per token.

त्वरित जाँच: Which change is usually the largest and cheapest speed win for a fixed classification task?

• Raising max_new_tokens
• Switching from bfloat16 to float32
• Using a smaller model fine-tuned for the task
• Turning off batching