The Toolkit

y depends on x₁ alone. Shuffling x₁ in a model that uses only x₁ raises the error by 7.95. Add x₂, an almost exact copy, and let the model split the weight: shuffling x₁ now raises the error by just 1.95, because x₂ still carries the information. Correlated features make each other look unimportant.

• Permuting a feature that is correlated with others creates impossible rows, and the model is judged on data it never saw (Hooker, Mentch & Zhou 2021).
• Importance on the training set reflects memorisation; compute it on held-out data.
• Low importance does not mean a feature is irrelevant to the outcome — only that this model does not need it.

rng = np.random.default_rng(4)
n = 5_000
x1 = rng.normal(size=n); x2 = x1 + rng.normal(0, 0.05, n)    # x2 almost a copy of x1
y = 2 * x1 + rng.normal(0, 0.5, n)

def importance(X, w, col):
    base = ((X @ w - y) ** 2).mean()
    Xp = X.copy(); Xp[:, col] = rng.permutation(Xp[:, col])
    return ((Xp @ w - y) ** 2).mean() - base

alone = importance(x1[:, None], np.array([2.0]), 0)
both = importance(np.c_[x1, x2], np.array([1.0, 1.0]), 0)     # a model that split the weight

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

# Python — permutation importance
from sklearn.inspection import permutation_importance

result = permutation_importance(
    model, X_test, y_test,
    n_repeats=30, random_state=42,
    scoring='accuracy'
)

# Sorted importance
for i in result.importances_mean.argsort()[::-1]:
    print(f"{features[i]}: {result.importances_mean[i]:.3f}")