Core ML

Reproducibility in Machine Learning

If you compile a Java app twice, you get the same binary. If you train a neural network twice with the exact same code and data, you might get two slightly different models. Reproducibility is the battle to make ML training deterministic.

True reproducibility requires three locked pillars: Versioned Data, Pinned Environments, and fixed Random Seeds. But even with all three, hardware-level parallel reductions on GPUs can still introduce microscopic non-determinism.

Why Does This Exist?

In traditional software engineering, determinism is assumed. $2 + 2$ always equals $4$ .

In Machine Learning, non-determinism is the default state. If you train a PyTorch model today, and your colleague trains the exact same code on the exact same dataset tomorrow, your models might end up with different weights. This is a massive problem. If you cannot reproduce a model, you cannot debug it when it breaks in production. You also cannot scientifically prove that your new feature actually improved the model, because the difference in accuracy might just be random variance.

Reproducibility is the engineering discipline of locking down every source of randomness in an ML pipeline until $f(\text{code}, \text{data})$ always produces the exact same $M$ .

Think of It Like This

Imagine baking a cake in two different kitchens. You have the exact same recipe (Code) and the exact same ingredients (Data).

But in Kitchen A, the oven is slightly hotter on the left side. In Kitchen B, the chef stirs the batter clockwise instead of counter-clockwise. The cakes will taste slightly different. Reproducibility means not just sharing the recipe and ingredients, but shipping a locked, identical kitchen (Docker) and specifying the exact direction and speed of the mixing spoon (Random Seeds).

How It Actually Works

To achieve reproducibility, you have to systematically eliminate three sources of variance.

1. Environmental Variance (The Kitchen)

If you train a model on scikit-learn 1.2 and your colleague runs it on scikit-learn 1.3, the underlying math might have changed. The Fix: You must pin your environment. This means using a requirements.txt with exact versions (e.g., pandas==2.0.1), or better yet, running all training inside a version-controlled Docker container.

2. Algorithmic Randomness (The Spoon)

Machine learning is built on randomness. Weights are initialized randomly. Data batches are shuffled randomly. Dropout randomly turns off neurons. The Fix: You must lock the Random Number Generator (RNG) by setting a strict Seed at the very top of your script. If the seed is 42, the "random" numbers generated will be identical every time the script runs.

3. Hardware Non-Determinism (The Oven)

This is the hardest one. When you run a neural network on a GPU, operations like atomicAdd are executed in parallel by thousands of tiny cores. Because of microscopic differences in hardware timing, Core 1 might finish before Core 2 on Monday, but Core 2 finishes before Core 1 on Tuesday. Floating-point addition is not associative (i.e., $(A + B) + C \neq A + (B + C)$ at high precision). This causes tiny rounding differences that compound over 100 epochs. The Fix: You have to explicitly tell CUDA (NVIDIA's software layer) to use deterministic algorithms, which often forces it to run slower, sequential operations instead of fast, parallel ones.

Show Me the Code

Here is the standard "Seed Everything" block that you will see at the top of robust PyTorch training scripts.

1import torch2import numpy as np3import random4import os5
6def seed_everything(seed: int = 42):7    # 1. Python's built-in random module8    random.seed(seed)9    10    # 2. OS-level hash randomization (affects dictionary ordering)11    os.environ['PYTHONHASHSEED'] = str(seed)12    13    # 3. NumPy's random number generator14    np.random.seed(seed)15    16    # 4. PyTorch's random number generator17    torch.manual_seed(seed)18    torch.cuda.manual_seed(seed)19    torch.cuda.manual_seed_all(seed) # If using multi-GPU20    21    # 5. Force CUDA to use deterministic algorithms (WARNING: Slows down training)22    torch.backends.cudnn.deterministic = True23    torch.backends.cudnn.benchmark = False24
25# Call this before you define your model or load your data!26seed_everything(42)

Watch Out For

The "It Runs on My Machine" GPU Trap. Even if you seed everything and use deterministic CUDA algorithms, your PyTorch model trained on an RTX 3090 might not be bit-for-bit identical to the same code run on an A100. Different GPU architectures execute instructions differently. True bit-level reproducibility across different hardware generations is incredibly difficult, which is why large teams often allocate a specific, homogenous cluster of GPUs strictly for final production training.

The Quick Version

Reproducibility ensures that training an ML model twice yields the exact same model weights.
You must eliminate Environmental Variance by pinning dependencies (e.g., Docker).
You must eliminate Algorithmic Randomness by setting a global random seed for Python, NumPy, and PyTorch.
You must mitigate Hardware Non-Determinism by forcing the GPU to use deterministic (but often slower) kernel operations.