02 — Evaluate Your Model

ROC & AUC Curves#

ClassificationStatistical 1 min read
◆ The PatternVisualising model performance across all thresholds

The ROC curve plots True Positive Rate vs False Positive Rate at every threshold. The AUC (area under curve) collapses this into a single number: 1.0 = perfect, 0.5 = random. It's threshold-independent, making it ideal for comparing models.

TPR = TP / (TP + FN)    FPR = FP / (FP + TN)
TPR = sensitivity/recall  |  FPR = 1 − specificity  |  plotted as (FPR, TPR)
AUC = ∫ TPR(FPR) dFPR ∈ [0, 1]
AUC = probability that the model ranks a random positive higher than a random negative
// Interactive ROC — adjust model separability
Separability0.65
AUC—
Multi-class: Use roc_auc_score(y, y_prob, multi_class='ovr') with one-vs-rest for multi-class problems.
Pattern bridge: AUC measures discrimination — can the model separate classes? In RSI, you're doing the same thing: separating overbought from oversold regimes across different threshold levels.
Worked example

300 positives and 700 negatives. The area under the ROC curve computed from ranks is 0.7531; the share of all positive–negative pairs in which the positive scores higher is also 0.7531. That is what AUC means: the probability that a random positive outranks a random negative — a ranking measure, blind to the threshold you will actually use.

Where it misleads
  • A good AUC can hide poor performance in the region you operate in; look at the curve near your threshold, or at partial AUC.
  • AUC is insensitive to calibration: multiplying all scores by 0.1 leaves it unchanged.
  • With rare positives, the ROC curve looks good while precision is poor; check the precision–recall curve too.
In code
from scipy.stats import rankdata
rng = np.random.default_rng(1)
pos, neg = rng.normal(1.0, 1, 300), rng.normal(0, 1, 700)
ranks = rankdata(np.r_[pos, neg])
auc_rank = (ranks[:300].sum() - 300 * 301 / 2) / (300 * 700)      # Mann-Whitney U, scaled
auc_pairs = (pos[:, None] > neg[None, :]).mean()                   # P(random positive outranks random negative)
Python, in your browser — edit the code and run it again

Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.

The same idea with a library
# Python — ROC & AUC
from sklearn.metrics import roc_curve, roc_auc_score
import matplotlib.pyplot as plt

fpr, tpr, thresholds = roc_curve(y_true, y_prob)
auc = roc_auc_score(y_true, y_prob)

plt.plot(fpr, tpr, label=f'AUC = {auc:.3f}')
plt.plot([0,1],[0,1], '--', color='gray')
plt.xlabel('FPR'); plt.ylabel('TPR')
plt.legend(); plt.show()
Sources
  • J. A. Hanley & B. J. McNeil, “The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve”, Radiology 143(1), 1982
  • T. Fawcett, “An Introduction to ROC Analysis”, Pattern Recognition Letters 27(8), 2006

Last reviewed October 2026 · How this site checks its content · Report a mistake

Check yourself

1. A classifier has AUC = 0.80. What does that mean?

Show answer

A. AUC measures ranking. Accuracy and recall depend on the threshold you choose, which AUC averages over.

2. Your model’s AUC on the test set is 0.30. What is the quickest fix?

Show answer

A. AUC below 0.5 usually means a sign or label flip somewhere. 0.5 is the uninformative point, not 0.

3. Which of these does AUC not tell you?

Show answer

C. AUC is unchanged by any order-preserving transformation of the scores, so a model can rank well and still give badly calibrated probabilities.

Linked from 1
← Previous
Confusion Matrix & Classification Metrics
Open in the full reader, with the topic sidebar