PCA finds the axes along which data varies the most. These are the eigenvectors of the covariance matrix.
5,000 points from a correlated 2-d distribution. The covariance matrix has eigenvalues 4.53 and 0.43, so the first principal component explains 91.3% of the variance. One number per point, its position along that axis, keeps almost all the spread of two.
- PCA finds directions of large variance, not directions that matter for a prediction; a small component can carry the signal.
- It is sensitive to scale; standardize features measured in different units.
- It finds only linear structure; curved data needs other methods.
rng = np.random.default_rng(7) X = rng.multivariate_normal([0, 0], [[3, 2], [2, 2]], size=5_000) X = X - X.mean(0) vals, vecs = np.linalg.eigh(np.cov(X.T)) vals = vals[::-1] explained = vals / vals.sum()
Assumes import numpy as np. Each snippet carries its own example numbers; the comments say which are made up or simulated.
- K. Pearson, “On Lines and Planes of Closest Fit to Systems of Points in Space”, Philosophical Magazine 2(11), 1901
- H. Hotelling, “Analysis of a Complex of Statistical Variables into Principal Components”, Journal of Educational Psychology 24(6), 1933
- I. T. Jolliffe & J. Cadima, “Principal Component Analysis: A Review and Recent Developments”, Philosophical Transactions of the Royal Society A 374, 2016
Last reviewed October 2026 · How this site checks its content · Report a mistake
1. Why standardise variables before PCA?
Show answer
A. Income in kroner would swamp age in years. Standardising lets each variable count by its correlation, not its units.
2. What is the first principal component?
Show answer
B. Each next component is the direction of largest remaining variance, orthogonal to the earlier ones.
3. PCA keeps components covering 95% of the variance. Does that keep 95% of the information for predicting y?
Show answer
B. Unsupervised compression is not supervised selection. Partial least squares, or validating the downstream model, guards against this.