# Serving Models in Production — Hugging Face Transformers

Source: https://www.skillbyai.com/en/hugging-face/hf-serving

> Wrap a model in an API correctly and know when to move to a dedicated inference server.

## From a script to a service

The simplest service wraps a pipeline in a web framework such as **FastAPI**. The golden rule: **load the model once** at startup, never inside the request handler, because loading takes seconds and gigabytes of memory. A single Python process handles GPU work one request at a time, so for low traffic this is fine, but under load requests queue up. Dedicated inference servers solve that. Servers such as **vLLM** and Hugging Face's **Text Generation Inference** focus on LLMs and implement **continuous batching**, where new requests join a running batch as soon as earlier ones finish, plus memory-efficient KV-cache management, which together raise throughput a lot. Many expose an OpenAI-compatible HTTP API, so client code barely changes. For encoder models such as embeddings and classifiers, Text Embeddings Inference or an ONNX export are common choices. Managed options, such as Hugging Face Inference Endpoints or your cloud provider's ML platform, trade some cost for not managing GPUs yourself.

## A minimal FastAPI wrapper

The pipeline is created at import time, once per worker process. Run with `uvicorn app:app`.

```python
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import pipeline

app = FastAPI()
classifier = pipeline(
    "text-classification",
    model="distilbert/distilbert-base-uncased-finetuned-sst-2-english",
)

class Review(BaseModel):
    text: str

@app.post("/predict")
def predict(review: Review):
    result = classifier(review.text, truncation=True)[0]
    return {"label": result["label"], "score": round(result["score"], 4)}
```

## One model copy per worker

Each uvicorn or gunicorn worker process loads its own copy of the model. Four workers on one GPU means four copies in GPU memory. Scale with a batching inference server instead of simply adding workers.

**Quiz:** What is continuous batching, as used by LLM inference servers?

- [x] Adding new requests to a running batch as soon as other sequences finish
- [ ] Training the model continuously on user requests
- [ ] Waiting until exactly 32 requests arrive before running
- [ ] Splitting one request across many GPUs

*Answer:* Adding new requests to a running batch as soon as other sequences finish. Continuous batching keeps the GPU busy by swapping finished sequences out and new requests in at each generation step.