ClassificationStatistical
1 min read
◆ The PatternThe foundation of classification evaluation — TP, FP, TN, FN and the metrics built on them
Every classifier's performance starts here. The confusion matrix gives you four counts: true positives (TP), false positives (FP), true negatives (TN), and false negatives (FN). Every classification metric is built from these four numbers.
Precision = TP / (TP + FP)
Precision = of all predicted positives, how many were correct? High when you can't afford false alarms.
Recall = TP / (TP + FN)
Recall = of all actual positives, how many did we catch? High when you can't afford to miss cases.
F1 = 2 · (P · R) / (P + R)
F1 = harmonic mean of precision and recall. Balanced score when both matter equally.
// Interactive — adjust threshold, see all metrics update
Threshold0.50
Precision—
Recall—
F1—
| Metric | Formula | When to use |
|---|---|---|
| Accuracy | (TP+TN)/(TP+TN+FP+FN) | Balanced classes only |
| Precision | TP/(TP+FP) | Cost of false positives is high (spam) |
| Recall | TP/(TP+FN) | Cost of misses is high (cancer) |
| F1 | 2PR/(P+R) | Balance precision & recall |
| MCC | (TP·TN−FP·FN)/√(...) | Imbalanced datasets |
Threshold matters: Most classifiers output probabilities. Changing the threshold trades precision for recall. Don't just use 0.5 — tune it for your problem.
Pattern bridge: Precision vs recall is the same trade-off you see in ROC curves and in market fear/greed — cautious vs aggressive, and the cost of being wrong in each direction.
How to use this in practice
- Train your model and get
y_predprobabilities on your validation set (never train set) - Plot the confusion matrix at threshold 0.5 — check if FP or FN is the bigger problem
- If FP is costly (spam filter, fraud alert) → optimize for precision — raise threshold
- If FN is costly (cancer screening, security) → optimize for recall — lower threshold
- Use
classification_report()to get all metrics at once — report this in your model card
Common pitfall — data leakage: If you tune the threshold on your test set, you're overfitting to it. Use a separate validation split or nested cross-validation to select the threshold, then evaluate once on the held-out test set.
Performance in practice
In production fraud detection at scale (millions of transactions/day), the precision-recall trade-off has real dollar costs:
- Low precision (many false positives) → customer friction, blocked legitimate purchases, support costs ~$5-15 per case
- Low recall (missed fraud) → direct financial loss, average $150+ per missed case
- Where an operating point lands — high precision, or high recall — depends on what each kind of error costs
- At Stripe/PayPal scale, moving the threshold by 0.01 can shift millions of dollars annually
When to use this
✓ Use when: Any binary classification task. Always your first evaluation step. Essential for imbalanced datasets where accuracy is misleading (99% accuracy on 1% fraud rate = useless).
✗ Skip when: Regression tasks (use MSE/MAE instead), ranking problems (use NDCG/MAP), or when you only care about ordering (use ROC-AUC instead of fixed-threshold metrics).
Try it on real data
Kaggle: Credit Card Fraud Detection (284K transactions, 492 frauds)
UCI: Breast Cancer Wisconsin (569 samples, binary diagnosis)
The credit card dataset is extremely imbalanced (0.17% positive) — perfect for seeing why accuracy fails and precision/recall matters.
Quick start — copy to notebook
pip install scikit-learn pandas matplotlib seaborn
# ────────────────────────────────────────
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, ConfusionMatrixDisplay
import matplotlib.pyplot as plt
df = pd.read_csv('creditcard.csv')
X, y = df.drop('Class', axis=1), df['Class']
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y)
model = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print(classification_report(y_test, y_pred=model.predict(X_test)))
ConfusionMatrixDisplay.from_estimator(model, X_test, y_test)
plt.show()Worked example
2,000 cases, 20% positive, and one model’s scores. At a threshold of 0.3 the model catches 97% of positives with 30% precision; at 0.5, 79% recall and 51% precision; at 0.7, 39% recall and 78% precision. One model, three confusion matrices: the threshold is a business decision, not a property of the model.
Where it misleads
- The default 0.5 threshold is only sensible if the scores are calibrated probabilities and both errors cost the same.
- Precision depends on how common positives are; the same matrix means something different in another population.
- Aggregated matrices hide which subgroups bear the errors; break them down by segment.
In code
rng = np.random.default_rng(0)
y = rng.random(2_000) < 0.2 # 20% positives
score = np.clip(rng.normal(np.where(y, 0.65, 0.35), 0.18), 0, 1)
table = {}
for th in (0.3, 0.5, 0.7):
pred = score >= th
tp, fp, fn = (pred & y).sum(), (pred & ~y).sum(), (~pred & y).sum()
table[th] = (round(tp / (tp + fp), 2), round(tp / (tp + fn), 2)) # (precision, recall)Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Python — classification metrics from sklearn.metrics import ( confusion_matrix, classification_report, precision_score, recall_score, f1_score ) y_true = [1,0,1,1,0,1,0,0,1,0] y_pred = [1,0,1,0,0,1,1,0,1,0] # Full report print(classification_report(y_true, y_pred)) # Individual metrics p = precision_score(y_true, y_pred) r = recall_score(y_true, y_pred) f = f1_score(y_true, y_pred)
Sources
- D. M. W. Powers, “Evaluation: From Precision, Recall and F-Measure to ROC, Informedness, Markedness and Correlation”, Journal of Machine Learning Technologies 2(1), 2011
- C. Elkan, “The Foundations of Cost-Sensitive Learning”, IJCAI, 2001
Last reviewed October 2026 · How this site checks its content · Report a mistake
Linked from 1