The Toolkit

Two groups of 100,000 whose means differ by 0.75 points on a standard deviation of 15. The p-value is 1.6×10⁻³⁴ — overwhelming evidence that a difference exists. Cohen’s d is 0.055, a tenth of what is usually called small. With enough data, significance only tells you the effect is not exactly zero.

• Cohen’s benchmarks (0.2, 0.5, 0.8) were meant as a last resort; what counts as meaningful depends on the field and the cost of the change.
• Standardised effect sizes depend on the spread of the sample; the same effect looks bigger in a homogeneous group.
• Effects reported from small significant studies are inflated on average (see Hypothesis Testing).

from scipy.stats import ttest_ind
rng = np.random.default_rng(17)
a = rng.normal(100, 15, 100_000)
b = rng.normal(100.75, 15, 100_000)                  # a 0.75-point difference on an SD of 15
p = ttest_ind(a, b).pvalue
d = (b.mean() - a.mean()) / np.sqrt((a.var(ddof=1) + b.var(ddof=1)) / 2)

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

# Python — effect size
import numpy as np

def cohens_d(group1, group2):
    n1, n2 = len(group1), len(group2)
    var1, var2 = group1.var(), group2.var()
    pooled_std = np.sqrt(((n1-1)*var1 + (n2-1)*var2) / (n1+n2-2))
    return (group1.mean() - group2.mean()) / pooled_std

d = cohens_d(model_a_scores, model_b_scores)
print(f"Cohen's d = {d:.3f}")