ML Math

A model that is right on 76 of 100 cases. A single random 80/20 split measures its accuracy on 20 cases, and across 2,000 such splits the result ranges from 0.60 to 0.90 (5th to 95th percentile). Five-fold cross-validation tests every case once and gives 0.76. One split is a noisy measurement; k folds average the noise away.

• Folds must respect the structure of the data: groups, time order, duplicates. Otherwise the score leaks (see Cross-Validation Done Right in The Toolkit).
• Using the CV score to choose hyperparameters and to report performance double-counts it; use nested CV.
• The fold scores are not independent, so their standard deviation understates the true uncertainty.

rng = np.random.default_rng(5)
correct = rng.random(100) < 0.8           # a model that is right on 80 of 100 cases, fixed
single = []
for _ in range(2000):                      # one random 80/20 split each time
    test = rng.permutation(100)[:20]
    single.append(correct[test].mean())
lo, hi = np.percentile(single, [5, 95])
five_fold = np.mean([correct[k::5].mean() for k in range(5)])   # every case tested once

Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.