Saving, Loading and Serving Models

Inference is different from training

Save model weights with torch.save(model.state_dict(), path) and load them into the same architecture with load_state_dict. For inference, always call model.eval() (turns off dropout and makes batch norm use its stored statistics) and wrap predictions in torch.no_grad() (or torch.inference_mode()) to save memory and time. Record the preprocessing, library versions and class labels with the weights. For production, consider exporting (for example to ONNX or TorchScript-style formats) or using a serving framework, and monitor inputs for drift.

Why eval mode matters, run

I ran this on CPU with Python 3, PyTorch 2.14.1, numpy 2.5.3 and scikit-learn 1.9.1, with fixed seeds and one thread. After saving and reloading a model with dropout, two calls on the same input give different outputs in training mode, and identical outputs after model.eval().

import torch, os, tempfile
torch.manual_seed(0)
model = torch.nn.Sequential(torch.nn.Linear(4, 8), torch.nn.ReLU(), torch.nn.Dropout(0.5), torch.nn.Linear(8, 2))
path = os.path.join(tempfile.mkdtemp(), "model.pt")
torch.save(model.state_dict(), path)
restored = torch.nn.Sequential(torch.nn.Linear(4, 8), torch.nn.ReLU(), torch.nn.Dropout(0.5), torch.nn.Linear(8, 2))
restored.load_state_dict(torch.load(path))
x = torch.randn(1, 4)
restored.train()
a, b = restored(x), restored(x)
print("train mode, two calls equal:", torch.allclose(a, b))
restored.eval()
with torch.no_grad():
    a, b = restored(x), restored(x)
print("eval mode,  two calls equal:", torch.allclose(a, b))

Output:

train mode, two calls equal: False
eval mode,  two calls equal: True

Save a checkpoint bundle

Store weights, optimiser state, epoch, metrics, preprocessing settings and the code version together so training can resume and results can be reproduced.

त्वरित जाँच: Why call model.eval() before inference?

• It saves the model to disk
• It trains the model faster
• It disables dropout and uses stored batch-norm statistics for consistent outputs
• It increases the learning rate