Dimensionality Reduction with PCA

What you will learn in this lesson:

• What "dimensionality reduction" means and why you'd want it
• The core idea of PCA: finding new axes that capture the most variance in the data
• How to read explained variance ratios to decide how many components to keep
• The tradeoff PCA introduces: less noise and redundancy, but less interpretability
• How to fit PCA in Python and inspect explained variance

Why Reduce Dimensions at All

Real datasets often have far more features than you can usefully visualize or reason about directly. A genomics dataset might have thousands of gene-expression measurements per sample. A sensor array might log hundreds of correlated readings per second. Many of those features carry overlapping, redundant information, similar to the multicollinearity problem from the regression lessons, just at a much larger scale.

Dimensionality reduction compresses many correlated features down into a much smaller number of new features, while trying to preserve as much of the original information as possible. Principal Component Analysis (PCA) is the most widely used technique for this, and unlike K-means, it isn't trying to group data points. It's trying to re-describe them more efficiently.

The Core Idea: Directions of Maximum Variance

Picture a scatter of points that are spread out diagonally, more variation along one diagonal direction than along the direction perpendicular to it. The original x and y axes don't capture this well: both axes mix together "the direction the data actually spreads out in" and "the direction it barely varies in at all."

PCA finds a new set of axes, called principal components, that are rotated to align with the actual directions of spread in the data. The first principal component is the single direction along which the data varies the most. The second principal component is the direction of next-most variance, constrained to be perpendicular to the first. This continues for as many components as there are original features.

A Worked Example: Variance Along Different Axes

Take four points that fall exactly on a diagonal line: (1, 1), (2, 2), (3, 3), (4, 4). Along the original x1 axis, the values are 1, 2, 3, 4, with mean 2.5 and variance (1.5² + 0.5² + 0.5² + 1.5²) / 4 = 5/4 = 1.25. By symmetry, the x2 axis has the same variance, 1.25, so the two original axes split a total variance of 2.5 between them.

Now project each point onto the diagonal direction (the line x1 = x2) instead. The projected values become 1.414, 2.828, 4.243, and 5.657 (each original value multiplied by √2), with variance 2.5, the entire total variance, captured by this single direction. Projecting onto the perpendicular direction instead gives the same value, 0, for every point, meaning that direction captures zero variance.

This is PCA's whole idea in miniature: since these four points are perfectly correlated, all of their spread lives along one direction. A real PCA fit on this exact data would find that diagonal as the first principal component, capturing 100% of the variance, and the perpendicular direction as the second component, capturing none of it, meaning you could drop it entirely and lose no information at all.

Once you have these new axes, you can keep only the first few, the ones capturing the most variance, and drop the rest. If the first two principal components capture 95% of the total variance in a 50-feature dataset, you've compressed 50 numbers down to 2 while keeping the large majority of the actual information.

Reading Explained Variance

Every principal component comes with an explained variance ratio: the fraction of the dataset's total variance that this one component captures on its own. These ratios are sorted from largest to smallest by construction, since each component is defined to capture the most variance remaining after the previous ones.

A common way to choose how many components to keep is to look at the cumulative explained variance and stop once it crosses a threshold you're comfortable with, often somewhere around 90 to 95%. There's no universally correct threshold. It depends on how much information loss is acceptable for whatever you're doing with the reduced data afterward, whether that's visualization, feeding a downstream model, or just exploring the data's structure.

The Tradeoff: Compression vs. Interpretability

PCA's components are linear combinations of all the original features, not a clean subset of them. A principal component might be something like "0.4 times square footage, plus 0.3 times number of rooms, minus 0.1 times age," with contributions from every original feature folded together. This makes the compressed representation much harder to interpret directly than the original, named columns.

This is the opposite tradeoff from lasso regression, which keeps the original features but drops some of them entirely, preserving interpretability while losing some information. PCA keeps essentially all of the information in a compressed form, but loses the ability to point at a single original feature and say what it contributed. Choose between them based on whether you need to explain individual features or only need a faithful, compact representation of the overall data.

PCA in Python

import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
print("Original number of features:", X.shape[1])

# Always scale before PCA: it operates on variance, and an unscaled
# feature with a huge numeric range would dominate the components
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

pca = PCA()
pca.fit(X_scaled)

print("\nExplained variance ratio per component (first 10):")
print(np.round(pca.explained_variance_ratio_[:10], 3))

cumulative = np.cumsum(pca.explained_variance_ratio_)
n_for_95pct = np.argmax(cumulative >= 0.95) + 1
print(f"\nComponents needed to reach 95% cumulative explained variance: {n_for_95pct}")
print(f"(compressed from {X.shape[1]} original features)")

This dataset has 30 original features. You should find that a much smaller number of principal components, typically under 10, already captures 95% of the total variance, which tells you the original 30 features contained substantial redundancy.