Classification✓ Mathematical
1 min read
◆ The PatternThe foundation of classification evaluation — TP, FP, TN, FN and the metrics built on them
Every classifier's performance starts here. The confusion matrix gives you four counts: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Every classification metric is built from these four numbers.
Precision = TP / (TP + FP)
Precision = of all predicted positives, how many were correct? High when you can't afford false alarms.
Recall = TP / (TP + FN)
Recall = of all actual positives, how many did we catch? High when you can't afford to miss cases.
F1 = 2 · (P · R) / (P + R)
F1 = harmonic mean of precision and recall. Balanced score when both matter equally.
// Interactive — adjust threshold, see all metrics update
Threshold0.50
Precision—
Recall—
F1—
| Metric | Formula | When to use |
|---|---|---|
| Accuracy | (TP+TN)/(TP+TN+FP+FN) | Balanced classes only |
| Precision | TP/(TP+FP) | Cost of false positives is high (spam) |
| Recall | TP/(TP+FN) | Cost of misses is high (cancer) |
| F1 | 2PR/(P+R) | Balance precision & recall |
| MCC | (TP·TN−FP·FN)/√(...) | Imbalanced datasets |
# Python — classification metrics from sklearn.metrics import ( confusion_matrix, classification_report, precision_score, recall_score, f1_score ) y_true = [1,0,1,1,0,1,0,0,1,0] y_pred = [1,0,1,0,0,1,1,0,1,0] # Full report print(classification_report(y_true, y_pred)) # Individual metrics p = precision_score(y_true, y_pred) r = recall_score(y_true, y_pred) f = f1_score(y_true, y_pred)
Threshold matters: Most classifiers output probabilities. Changing the threshold trades precision for recall. Don't just use 0.5 — tune it for your problem.
Pattern bridge: Precision vs recall is the same trade-off you see in ROC curves and in market fear/greed — cautious vs aggressive, and the cost of being wrong in each direction.
How to use this in practice
- Train your model and get
y_predprobabilities on your validation set (never train set) - Plot the confusion matrix at threshold 0.5 — check if FP or FN is the bigger problem
- If FP is costly (spam filter, fraud alert) → optimize for precision — raise threshold
- If FN is costly (cancer screening, security) → optimize for recall — lower threshold
- Use
classification_report()to get all metrics at once — report this in your model card
Common pitfall — data leakage: If you tune the threshold on your test set, you're overfitting to it. Use a separate validation split or nested cross-validation to select the threshold, then evaluate once on the held-out test set.
Performance in practice
In production fraud detection at scale (millions of transactions/day), the precision-recall trade-off has real dollar costs:
- Low precision (many false positives) → customer friction, blocked legitimate purchases, support costs ~$5-15 per case
- Low recall (missed fraud) → direct financial loss, average $150+ per missed case
- Most production systems operate at 95%+ precision with 60-80% recall — the cost asymmetry drives the threshold
- At Stripe/PayPal scale, moving the threshold by 0.01 can shift millions of dollars annually
When to use this
✓ Use when: Any binary classification task. Always your first evaluation step. Essential for imbalanced datasets where accuracy is misleading (99% accuracy on 1% fraud rate = useless).
✗ Skip when: Regression tasks (use MSE/MAE instead), ranking problems (use NDCG/MAP), or when you only care about ordering (use ROC-AUC instead of fixed-threshold metrics).
Try it on real data
Kaggle: Credit Card Fraud Detection (284K transactions, 492 frauds)
UCI: Breast Cancer Wisconsin (569 samples, binary diagnosis)
The credit card dataset is extremely imbalanced (0.17% positive) — perfect for seeing why accuracy fails and precision/recall matters.
Quick start — copy to notebook
pip install scikit-learn pandas matplotlib seaborn
# ────────────────────────────────────────
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
import matplotlib.pyplot as plt
df = pd.read_csv('creditcard.csv')
X, y = df.drop('Class', axis=1), df['Class']
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print(classification_report(y_test, y_pred=model.predict(X_test)))
ConfusionMatrixDisplay.from_estimator(model, X_test, y_test)
plt.show()