The ROC curve plots True Positive Rate vs False Positive Rate at every threshold. The AUC (area under curve) collapses this into a single number: 1.0 = perfect, 0.5 = random. It's threshold-independent, making it ideal for comparing models.
roc_auc_score(y, y_prob, multi_class='ovr') with one-vs-rest for multi-class problems.300 positives and 700 negatives. The area under the ROC curve computed from ranks is 0.7531; the share of all positive–negative pairs in which the positive scores higher is also 0.7531. That is what AUC means: the probability that a random positive outranks a random negative — a ranking measure, blind to the threshold you will actually use.
- A good AUC can hide poor performance in the region you operate in; look at the curve near your threshold, or at partial AUC.
- AUC is insensitive to calibration: multiplying all scores by 0.1 leaves it unchanged.
- With rare positives, the ROC curve looks good while precision is poor; check the precision–recall curve too.
from scipy.stats import rankdata rng = np.random.default_rng(1) pos, neg = rng.normal(1.0, 1, 300), rng.normal(0, 1, 700) ranks = rankdata(np.r_[pos, neg]) auc_rank = (ranks[:300].sum() - 300 * 301 / 2) / (300 * 700) # Mann-Whitney U, scaled auc_pairs = (pos[:, None] > neg[None, :]).mean() # P(random positive outranks random negative)
Assumes import numpy as np and import pandas as pd (and SciPy where imported). Each snippet simulates or makes up its own data, as the comments say.
The same idea with a library
# Python — ROC & AUC from sklearn.metrics import roc_curve, roc_auc_score import matplotlib.pyplot as plt fpr, tpr, thresholds = roc_curve(y_true, y_prob) auc = roc_auc_score(y_true, y_prob) plt.plot(fpr, tpr, label=f'AUC = {auc:.3f}') plt.plot([0,1],[0,1], '--', color='gray') plt.xlabel('FPR'); plt.ylabel('TPR') plt.legend(); plt.show()
- J. A. Hanley & B. J. McNeil, “The Meaning and Use of the Area under a Receiver Operating Characteristic (ROC) Curve”, Radiology 143(1), 1982
- T. Fawcett, “An Introduction to ROC Analysis”, Pattern Recognition Letters 27(8), 2006
Last reviewed October 2026 · How this site checks its content · Report a mistake
1. A classifier has AUC = 0.80. What does that mean?
Show answer
A. AUC measures ranking. Accuracy and recall depend on the threshold you choose, which AUC averages over.
2. Your model’s AUC on the test set is 0.30. What is the quickest fix?
Show answer
A. AUC below 0.5 usually means a sign or label flip somewhere. 0.5 is the uninformative point, not 0.
3. Which of these does AUC not tell you?
Show answer
C. AUC is unchanged by any order-preserving transformation of the scores, so a model can rank well and still give badly calibrated probabilities.