MLOps & Production ML

A million float32 numbers summed in two different orders differ by 0.00012. Nothing is wrong with either sum; floating-point addition is not associative, and parallel hardware changes the order. Even the size of the gap depends on the machine: run the code below in the browser and it comes out at 0.00018, because that build of NumPy adds the numbers in a different order. The same seed does give the same random numbers (True), but bit-identical training also needs the same order of operations.

• Fixing seeds is necessary, not sufficient: library versions, hardware and non-deterministic GPU kernels change results too.
• Exact reproduction is not always the goal; reporting the spread over seeds shows whether a result is robust.
• Unversioned data makes every other control useless.

x = np.random.default_rng(6).normal(0, 1, 1_000_000).astype(np.float32)
s1 = x.sum()
s2 = x[np.random.default_rng(7).permutation(len(x))].sum()   # same numbers, different order
order_diff = abs(float(s1) - float(s2))
same_seed = np.random.default_rng(42).random(3).tolist() == np.random.default_rng(42).random(3).tolist()

Assumes import numpy as np and import pandas as pd. Each snippet carries its own example numbers; the comments say which are made up or simulated.

# Python — reproducibility essentials
import random, numpy as np, torch

# Lock all seeds
SEED = 42
random.seed(SEED)
np.random.seed(SEED)
torch.manual_seed(SEED)
torch.cuda.manual_seed_all(SEED)
torch.backends.cudnn.deterministic = True

# Pin deps: pip freeze > requirements.txt
# Version data: dvc add data/train.csv
# Container: docker build -t experiment:v1 .