Credit card default — case study — Pattern is Everything

Cases · Classification · 30,000 clients · Taiwan, 2005

Credit card default: 82% accurate, most defaults missed

Which credit card clients will miss next month’s payment? A model that is right four times in five sounds useful — this case asks what it actually catches, and what the data dictionary did not say.

DataDefault of Credit Card Clients — One row per client of a Taiwanese bank, April–September 2005: credit limit, sex, education, marital status, age, six months of repayment status, bill amounts and payments — and whether the client defaulted the following month.

SourceUCI Machine Learning Repository, donated by I-Cheng Yeh

LicenceCC BY 4.0; here as CSV, with the ID column dropped and the target renamed default

Filecredit.csv (2635 KB), as used below

01The number to beat

Before any model: how often do clients default, and how accurate is a model that never predicts a default?

import numpy as np
import pandas as pd

df = pd.read_csv('credit.csv')
default_rate = df['default'].mean()
print(len(df), 'clients | default rate:', round(default_rate, 4))
print('accuracy of always predicting "no default":', round(1 - default_rate, 4))

Python, in your browser — edit the code and run it again

30000 clients | default rate: 0.2212
accuracy of always predicting "no default": 0.7788

About one client in 4.5 defaults. Predicting “no default” for everyone is right 77.9% of the time and catches nobody. Any accuracy figure for this data has to be read against that.

02The data dictionary and the data disagree

The documentation says repayment status is −1 for “paid duly” and 1–9 for months of delay, education is 1–4 and marital status 1–3. Count what is actually there.

pay = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']
print('PAY_0 values:', df.PAY_0.value_counts().sort_index().to_dict())
undocumented_pay = df[pay].isin([-2, 0]).any(axis=1).mean()
odd_education = (~df.EDUCATION.isin([1, 2, 3, 4])).sum()
odd_marriage = (~df.MARRIAGE.isin([1, 2, 3])).sum()
print('clients with a -2 or 0 status in some month:', round(undocumented_pay, 3))
print('education codes outside 1-4:', odd_education, '| marriage outside 1-3:', odd_marriage)
print('a status of 1 in PAY_0 vs PAY_2:', (df.PAY_0 == 1).sum(), (df.PAY_2 == 1).sum())

Python, in your browser — runs step 1 first — edit the code and run it again

PAY_0 values: {-2: 2759, -1: 5686, 0: 14737, 1: 3688, 2: 2667, 3: 322, 4: 76, 5: 26, 6: 11, 7: 9, 8: 19}
clients with a -2 or 0 status in some month: 0.865
education codes outside 1-4: 345 | marriage outside 1-3: 54
a status of 1 in PAY_0 vs PAY_2: 3688 28

The most common repayment status, 0, is not in the documentation, nor is −2; 86% of clients have one of them in some month. (Discussions of the data commonly read 0 as “revolving credit” and −2 as “no consumption”, but the source does not say.) Status 1 — one month late — appears 3,688 times in the latest month and only 28 times in the month before, which suggests the months were not coded the same way. A model will use these columns either way; the question is whether you can explain what it learned.

03Two models, and what 0.5 hides

Hold out 30% of the clients (stratified, so both parts have the same default rate). Fit a logistic regression and gradient-boosted trees, and score them at the default threshold of 0.5.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import make_pipeline
from sklearn.ensemble import HistGradientBoostingClassifier
from sklearn.metrics import accuracy_score, roc_auc_score, recall_score, precision_score

X, y = df.drop(columns='default'), df['default']
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)

logit = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)
boost = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)
for name, m in [('logistic', logit), ('boosted trees', boost)]:
    p = m.predict_proba(X_te)[:, 1]
    flag = p >= 0.5
    print(f'{name:14} accuracy {accuracy_score(y_te, flag):.3f}  AUC {roc_auc_score(y_te, p):.3f}  '
          f'recall {recall_score(y_te, flag):.3f}  precision {precision_score(y_te, flag):.3f}')
p = boost.predict_proba(X_te)[:, 1]

Python, in your browser — runs steps 1–2 first — edit the code and run it again

logistic       accuracy 0.811  AUC 0.715  recall 0.237  precision 0.724
boosted trees  accuracy 0.823  AUC 0.775  recall 0.364  precision 0.687

The boosted model is 82.3% accurate — 4.4 points better than never predicting a default — and at a threshold of 0.5 it catches 36% of the clients who default. Most defaults go unflagged. The AUC of 0.77 says the ranking is reasonable; the threshold is what throws the catches away.

04One column against the model

A rule anyone could write: flag every client who is at least a month late right now.

late = X_te.PAY_0 >= 1
print('rule PAY_0 >= 1: flags', round(late.mean(), 3), 'of clients',
      '| recall', round(recall_score(y_te, late), 3),
      '| precision', round(precision_score(y_te, late), 3))
print('AUC of PAY_0 alone, as a score:', round(roc_auc_score(y_te, X_te.PAY_0), 3))

Python, in your browser — runs steps 1–3 first — edit the code and run it again

rule PAY_0 >= 1: flags 0.222 of clients | recall 0.51 | precision 0.507
AUC of PAY_0 alone, as a score: 0.688

The one-line rule catches 51% of the defaults — more than the model at 0.5 — with precision 0.51. Last month’s repayment status carries most of the signal; the other 22 columns lift the AUC from 0.69 (that column alone) to 0.77. A baseline like this belongs in every model report, because it tells you what the model adds.

05Choose the threshold from the costs

Suppose a missed default costs five times as much as flagging a client who would have paid. If the probabilities are calibrated, the cost-minimising rule is to flag when the probability exceeds 1 / (1 + 5) ≈ 0.17. Check the calibration, then compare thresholds.

bins = pd.qcut(p, 10, labels=False)                # ten groups by predicted probability
calib = pd.DataFrame({'predicted': p, 'actual': y_te.values}).groupby(bins).mean().round(3)
print(calib.T.to_string())

costs = {}
for th in (0.5, 0.3, 0.2, 1 / 6, 0.1):
    flag = p >= th
    missed = (~flag & (y_te == 1)).sum()
    false_alarm = (flag & (y_te == 0)).sum()
    cost = 5 * missed + false_alarm
    costs[round(th, 3)] = cost
    print(f'threshold {th:.3f}: flags {flag.mean():.3f}  recall {recall_score(y_te, flag):.3f}  cost {cost}')

Python, in your browser — runs steps 1–4 first — edit the code and run it again

               0      1      2      3      4      5      6      7      8      9
predicted  0.051  0.072  0.094  0.117  0.139  0.161  0.200  0.270  0.409  0.694
actual     0.049  0.079  0.083  0.120  0.151  0.176  0.172  0.263  0.412  0.707
threshold 0.500: flags 0.117  recall 0.364  cost 6665
threshold 0.300: flags 0.220  recall 0.536  cost 5532
threshold 0.200: flags 0.348  recall 0.662  cost 5170
threshold 0.167: flags 0.429  recall 0.722  cost 5193
threshold 0.100: flags 0.725  recall 0.914  cost 5564

Predicted and actual default rates agree within a few points in every decile, so the probabilities can be taken at face value. At the cost-based threshold the model flags 43% of clients and catches 72% of the defaults, and the total cost falls by 22% against 0.5. Nothing about the model changed — only the decision drawn from it. The cost curve is flat near its minimum: 0.2 does about as well as 1/6, while 0.5 and 0.1 are clearly worse.

Total cost on the 9,000 test clients: 5 per missed default plus 1 per false alarm.

What this case shows

Report accuracy next to the accuracy of the majority class. On imbalanced data, recall and precision at the threshold you will use say more than accuracy or AUC.
0.5 is a convention, not a decision. With calibrated probabilities, the threshold that minimises cost is cost of a false alarm ÷ (cost of a false alarm + cost of a miss).
Compare the model with the simplest rule a domain expert would write. If one column does most of the work, say so — and check what that column really encodes.
Read the data, not only the documentation: count the values that occur and look for codes no one explained.

The patterns behind it

Sampling & Class Imbalance — why the majority class sets the bar
Confusion Matrix & Classification Metrics — recall, precision and the threshold
ROC & AUC Curves — ranking quality, independent of the threshold
The Threshold — a threshold is a decision about costs
Missing Data Strategies — codes that stand in for “unknown”

Run it yourself

Each step above has a Run button: Python runs in this page, on the data file above.
Download the notebook — the same steps, for Jupyter on your own machine
Open it in JupyterLite — Jupyter in the browser, nothing to install

Sources

I-C. Yeh & C-H. Lien, “The comparisons of data mining techniques for the predictive accuracy of probability of default of credit card clients”, Expert Systems with Applications 36(2), 2009 — the data set
C. Elkan, “The Foundations of Cost-Sensitive Learning”, Proceedings of IJCAI, 2001 — the cost-based threshold
A. Niculescu-Mizil & R. Caruana, “Predicting Good Probabilities with Supervised Learning”, ICML, 2005 — calibration

Last reviewed: 2 October 2026 · How this site checks its content · Report a mistake