{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Credit card default: 82% accurate, most defaults missed\n",
"\n",
"Which credit card clients will miss next month’s payment? A model that is right four times in five sounds useful — this case asks what it actually catches, and what the data dictionary did not say.\n",
"\n",
"*From [Pattern is Everything](https://patterniseverything.com/cases/credit-default/). Data: Default of Credit Card Clients — UCI Machine Learning Repository, donated by I-Cheng Yeh.*"
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"# Fetch the data file if it is not next to this notebook (in JupyterLite it already is)\n",
"import os, urllib.request\n",
"if not os.path.exists('credit.csv'):\n",
" urllib.request.urlretrieve('https://patterniseverything.com/cases/data/credit.csv', 'credit.csv')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. The number to beat\n",
"\n",
"Before any model: how often do clients default, and how accurate is a model that never predicts a default?"
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"import numpy as np\n",
"import pandas as pd\n",
"\n",
"df = pd.read_csv('credit.csv')\n",
"default_rate = df['default'].mean()\n",
"print(len(df), 'clients | default rate:', round(default_rate, 4))\n",
"print('accuracy of always predicting \"no default\":', round(1 - default_rate, 4))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"About one client in 4.5 defaults. Predicting “no default” for everyone is right 77.9% of the time and catches nobody. Any accuracy figure for this data has to be read against that."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. The data dictionary and the data disagree\n",
"\n",
"The documentation says repayment status is −1 for “paid duly” and 1–9 for months of delay, education is 1–4 and marital status 1–3. Count what is actually there."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"pay = ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']\n",
"print('PAY_0 values:', df.PAY_0.value_counts().sort_index().to_dict())\n",
"undocumented_pay = df[pay].isin([-2, 0]).any(axis=1).mean()\n",
"odd_education = (~df.EDUCATION.isin([1, 2, 3, 4])).sum()\n",
"odd_marriage = (~df.MARRIAGE.isin([1, 2, 3])).sum()\n",
"print('clients with a -2 or 0 status in some month:', round(undocumented_pay, 3))\n",
"print('education codes outside 1-4:', odd_education, '| marriage outside 1-3:', odd_marriage)\n",
"print('a status of 1 in PAY_0 vs PAY_2:', (df.PAY_0 == 1).sum(), (df.PAY_2 == 1).sum())"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The most common repayment status, 0, is not in the documentation, nor is −2; 86% of clients have one of them in some month. (Discussions of the data commonly read 0 as “revolving credit” and −2 as “no consumption”, but the source does not say.) Status 1 — one month late — appears 3,688 times in the latest month and only 28 times in the month before, which suggests the months were not coded the same way. A model will use these columns either way; the question is whether you can explain what it learned."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Two models, and what 0.5 hides\n",
"\n",
"Hold out 30% of the clients (stratified, so both parts have the same default rate). Fit a logistic regression and gradient-boosted trees, and score them at the default threshold of 0.5."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"from sklearn.model_selection import train_test_split\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.preprocessing import StandardScaler\n",
"from sklearn.pipeline import make_pipeline\n",
"from sklearn.ensemble import HistGradientBoostingClassifier\n",
"from sklearn.metrics import accuracy_score, roc_auc_score, recall_score, precision_score\n",
"\n",
"X, y = df.drop(columns='default'), df['default']\n",
"X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=0.3, stratify=y, random_state=0)\n",
"\n",
"logit = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)).fit(X_tr, y_tr)\n",
"boost = HistGradientBoostingClassifier(random_state=0).fit(X_tr, y_tr)\n",
"for name, m in [('logistic', logit), ('boosted trees', boost)]:\n",
" p = m.predict_proba(X_te)[:, 1]\n",
" flag = p >= 0.5\n",
" print(f'{name:14} accuracy {accuracy_score(y_te, flag):.3f} AUC {roc_auc_score(y_te, p):.3f} '\n",
" f'recall {recall_score(y_te, flag):.3f} precision {precision_score(y_te, flag):.3f}')\n",
"p = boost.predict_proba(X_te)[:, 1]"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The boosted model is 82.3% accurate — 4.4 points better than never predicting a default — and at a threshold of 0.5 it catches 36% of the clients who default. Most defaults go unflagged. The AUC of 0.77 says the ranking is reasonable; the threshold is what throws the catches away."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. One column against the model\n",
"\n",
"A rule anyone could write: flag every client who is at least a month late right now."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"late = X_te.PAY_0 >= 1\n",
"print('rule PAY_0 >= 1: flags', round(late.mean(), 3), 'of clients',\n",
" '| recall', round(recall_score(y_te, late), 3),\n",
" '| precision', round(precision_score(y_te, late), 3))\n",
"print('AUC of PAY_0 alone, as a score:', round(roc_auc_score(y_te, X_te.PAY_0), 3))"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The one-line rule catches 51% of the defaults — more than the model at 0.5 — with precision 0.51. Last month’s repayment status carries most of the signal; the other 22 columns lift the AUC from 0.69 (that column alone) to 0.77. A baseline like this belongs in every model report, because it tells you what the model adds."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Choose the threshold from the costs\n",
"\n",
"Suppose a missed default costs five times as much as flagging a client who would have paid. If the probabilities are calibrated, the cost-minimising rule is to flag when the probability exceeds 1 / (1 + 5) ≈ 0.17. Check the calibration, then compare thresholds."
]
},
{
"cell_type": "code",
"metadata": {},
"source": [
"bins = pd.qcut(p, 10, labels=False) # ten groups by predicted probability\n",
"calib = pd.DataFrame({'predicted': p, 'actual': y_te.values}).groupby(bins).mean().round(3)\n",
"print(calib.T.to_string())\n",
"\n",
"costs = {}\n",
"for th in (0.5, 0.3, 0.2, 1 / 6, 0.1):\n",
" flag = p >= th\n",
" missed = (~flag & (y_te == 1)).sum()\n",
" false_alarm = (flag & (y_te == 0)).sum()\n",
" cost = 5 * missed + false_alarm\n",
" costs[round(th, 3)] = cost\n",
" print(f'threshold {th:.3f}: flags {flag.mean():.3f} recall {recall_score(y_te, flag):.3f} cost {cost}')"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Predicted and actual default rates agree within a few points in every decile, so the probabilities can be taken at face value. At the cost-based threshold the model flags 43% of clients and catches 72% of the defaults, and the total cost falls by 22% against 0.5. Nothing about the model changed — only the decision drawn from it. The cost curve is flat near its minimum: 0.2 does about as well as 1/6, while 0.5 and 0.1 are clearly worse."
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 5
}