Which statistical test should I use?
Answer a few questions about your data and get the test, the SciPy call, and what to report with it.
One group’s average against a fixed value
Yes
scipy.stats.ttest_1samp(x, popmean)
Compares the sample mean with the fixed value, scaled by its standard error.
No — small and skewed
scipy.stats.wilcoxon(x - value)
Uses ranks of the differences, so a few extreme values cannot dominate.
Two groups
Paired (before/after, same users, same folds)
Yes
scipy.stats.ttest_rel(after, before)
Tests the mean of the paired differences; far more sensitive than treating the groups as independent.
No
scipy.stats.wilcoxon(after, before)
The rank-based version of the paired test.
Independent groups
A number, roughly normal or with large samples
scipy.stats.ttest_ind(a, b, equal_var=False)
The safe default for two means: it does not assume equal variances.
A number with heavy tails, outliers, or ranks
scipy.stats.mannwhitneyu(a, b)
Tests whether one group tends to have larger values; with heavy tails it can be far more powerful than the t-test.
Yes/no — a proportion (conversion, defect rate)
Yes
scipy.stats.chi2_contingency(table)
Compares observed counts with those expected if the groups did not differ. For A/B tests, a Bayesian view is in Bayesian A/B.
No — small counts
scipy.stats.fisher_exact(table)
Exact for small samples, where the chi-square approximation breaks down.
Three or more groups
A number, roughly normal
scipy.stats.f_oneway(a, b, c) # then statsmodels pairwise_tukeyhsd
ANOVA says whether any group differs; the post-hoc test says which, while controlling for the extra comparisons. Use Welch’s ANOVA if the spreads differ a lot.
A number, skewed or with outliers
scipy.stats.kruskal(a, b, c)
The rank-based version of ANOVA; follow up with pairwise tests and a multiple-testing correction.
A category
scipy.stats.chi2_contingency(table)
Tests whether the category mix differs between groups.
Whether two variables are related
Both numeric, roughly linear
scipy.stats.pearsonr(x, y)
Measures straight-line association. Plot it first — r = 0 can hide a strong curved relationship.
Numeric but curved, ranked, or with outliers
scipy.stats.spearmanr(x, y)
Captures any monotonic relationship; mutual information captures non-monotonic ones too.
Both categories
scipy.stats.chi2_contingency(table)
Tests whether the two categorisations are associated.
Whether data follow a distribution, or two samples share one
scipy.stats.ks_2samp(a, b)
With large samples the test flags differences too small to matter; judge the size of the difference on the plot.
Last reviewed: 2 October 2026 · How this site checks its content · Report a mistake