The Toolkit

Model B beats model A by about one point on each of five CV folds, while the folds themselves vary from 0.78 to 0.90. An unpaired t-test sees the fold-to-fold variation and gives p = 0.736. A paired test compares the models fold by fold and gives p = 0.00015. When two models are scored on the same splits, compare them on the same splits.

• Folds share training data, so even the paired test is too optimistic; the corrected resampled t-test adjusts for it (Nadeau & Bengio 2003).
• Across many datasets, use rank-based tests such as Wilcoxon or Friedman (Demšar 2006).
• A significant difference can still be too small to matter; report the size of the gain.

from scipy.stats import ttest_ind, ttest_rel
a = np.array([0.81, 0.86, 0.78, 0.90, 0.84])        # model A, five CV folds
b = a + np.array([0.012, 0.009, 0.011, 0.008, 0.010])   # B is a little better on every fold
p_unpaired = ttest_ind(b, a).pvalue
p_paired = ttest_rel(b, a).pvalue

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

# Python — comparing two models
from scipy.stats import ttest_rel, wilcoxon

# Paired t-test on k-fold scores
scores_a = cross_val_score(model_a, X, y, cv=10)
scores_b = cross_val_score(model_b, X, y, cv=10)
t_stat, p_val = ttest_rel(scores_a, scores_b)

# Non-parametric alternative
w_stat, p_val = wilcoxon(scores_a, scores_b)
print(f"p = {p_val:.4f}")