Reproducibility and randomness

Where randomness comes from

A training run is random in several places: weight initialisation, data shuffling, dropout masks, data augmentation and, on GPUs, some kernels themselves. To repeat a run, control each one.

torch.manual_seed(seed) seeds PyTorch's generator on the CPU and all CUDA devices. If your code also uses Python's random or NumPy, seed those too. DataLoader shuffling can take its own generator argument, and worker processes can be seeded with a worker_init_fn.

Some GPU operations are nondeterministic because they add numbers in a varying order using atomic operations, and floating-point addition is not associative. torch.use_deterministic_algorithms(True) makes PyTorch pick deterministic implementations, and raise an error for operations that have none, so you know exactly where the problem is. Also set torch.backends.cudnn.benchmark = False, since benchmarking can pick different algorithms per run.

Even with all this, results can differ across PyTorch versions, GPUs and drivers. Do not chase bit-identical numbers across machines; run several seeds and report the mean and spread.

A seed_everything helper

Call it once at the start of a script and pass the returned generator to DataLoader(..., generator=g) so shuffling is repeatable too.

import random

import numpy as np
import torch

def seed_everything(seed: int = 42) -> torch.Generator:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)                    # CPU and all CUDA devices
    torch.backends.cudnn.benchmark = False
    torch.use_deterministic_algorithms(True)   # error if an op is nondeterministic
    g = torch.Generator()
    g.manual_seed(seed)
    return g

The cuBLAS environment variable

With deterministic algorithms on, some CUDA matrix operations also require the environment variable CUBLAS_WORKSPACE_CONFIG=:4096:8 (or :16:8) to be set before the program starts; PyTorch's error message tells you when. Determinism can cost speed, so many teams enable it only for debugging and final reported runs.

Quick check: What does `torch.use_deterministic_algorithms(True)` do when an operation has no deterministic implementation?

• Silently uses the nondeterministic version
• Falls back to running that operation on the CPU
• Raises an error naming the operation
• Skips the operation