# Reproducibility and randomness — PyTorch

Source: https://www.skillbyai.com/en/pytorch/pt-reproducibility

> Seed every source of randomness, enable deterministic algorithms, and report results honestly across seeds.

## Where randomness comes from

A training run is random in several places: weight initialisation, data shuffling, dropout masks, data augmentation and, on GPUs, some kernels themselves. To repeat a run, control each one.

`torch.manual_seed(seed)` seeds PyTorch's generator on the CPU and all CUDA devices. If your code also uses Python's `random` or NumPy, seed those too. DataLoader shuffling can take its own `generator` argument, and worker processes can be seeded with a `worker_init_fn`.

Some GPU operations are **nondeterministic** because they add numbers in a varying order using atomic operations, and floating-point addition is not associative. `torch.use_deterministic_algorithms(True)` makes PyTorch pick deterministic implementations, and **raise an error** for operations that have none, so you know exactly where the problem is. Also set `torch.backends.cudnn.benchmark = False`, since benchmarking can pick different algorithms per run.

Even with all this, results can differ across PyTorch versions, GPUs and drivers. Do not chase bit-identical numbers across machines; run several seeds and report the mean and spread.

## A seed_everything helper

Call it once at the start of a script and pass the returned generator to `DataLoader(..., generator=g)` so shuffling is repeatable too.

```python
import random

import numpy as np
import torch

def seed_everything(seed: int = 42) -> torch.Generator:
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)                    # CPU and all CUDA devices
    torch.backends.cudnn.benchmark = False
    torch.use_deterministic_algorithms(True)   # error if an op is nondeterministic
    g = torch.Generator()
    g.manual_seed(seed)
    return g
```

## The cuBLAS environment variable

With deterministic algorithms on, some CUDA matrix operations also require the environment variable `CUBLAS_WORKSPACE_CONFIG=:4096:8` (or `:16:8`) to be set before the program starts; PyTorch's error message tells you when. Determinism can cost speed, so many teams enable it only for debugging and final reported runs.

**Quiz:** What does `torch.use_deterministic_algorithms(True)` do when an operation has no deterministic implementation?

- [ ] Silently uses the nondeterministic version
- [ ] Falls back to running that operation on the CPU
- [x] Raises an error naming the operation
- [ ] Skips the operation

*Answer:* Raises an error naming the operation. It raises a RuntimeError so you know reproducibility cannot be guaranteed, instead of quietly running nondeterministic code.