Ridge and Lasso Regression

What you will learn in this lesson:

• A concrete scenario showing why unregularized regression overfits with many or correlated features
• How ridge's L2 penalty reshapes the loss surface and smoothly shrinks coefficients
• How lasso's L1 penalty can zero out coefficients exactly, and the geometric reason why (diamond vs. circle constraint regions)
• How to choose between ridge and lasso in practice
• Runnable code comparing both across several alpha values on the same dataset

Why Plain Regression Can Overfit: a Concrete Scenario

Ordinary linear regression finds the coefficients that minimize squared error on the training data, full stop, with no regard for how large or extreme those coefficients become. Consider a concrete case: you're predicting a patient's health outcome from 40 measured biomarkers, but you only have 50 patients in your training set.

With almost as many features as data points, the model has enormous freedom. It can essentially "memorize" the 50 patients by assigning large positive and negative coefficients that cancel out in just the right way to match every training outcome almost exactly, including the random noise specific to those 50 patients.

The fitted coefficients can swing to extreme values, with some biomarkers pushing predictions sharply upward and others pushing sharply downward, mostly explaining noise rather than real biology. The training error looks fantastic. The moment you apply this model to a new patient, the predictions are often far worse than a much simpler model would have given.

This is a direct manifestation of the bias-variance tradeoff from an earlier lesson: a model with many features and no constraint on coefficient size has low bias (it can fit almost anything in training) but high variance (it changes dramatically if you retrain it on a slightly different sample of patients). Regularization addresses this by deliberately introducing a small amount of bias, by constraining what the coefficients are allowed to be, in exchange for a large reduction in variance.

Ridge Regression: The L2 Penalty

Ridge regression modifies the loss function itself. Instead of only minimizing the sum of squared errors, it minimizes the sum of squared errors plus a penalty term proportional to the sum of the squared coefficients:

Loss = Sum of Squared Errors + α · (b1² + b2² + ... + bn²)

The term α (alpha) controls how strongly the penalty is enforced. When α is zero, ridge regression is identical to ordinary linear regression. As α increases, the optimizer is pushed to keep coefficients small, because large coefficients now cost more in the loss function, even if they would have reduced the squared-error term slightly.

Geometrically, adding the squared-coefficient penalty reshapes the loss surface. Instead of a flat valley with a long, shallow trough along directions where coefficients trade off against each other (which is what happens with correlated features), the squared penalty adds a bowl-shaped term centered at the origin, pulling the minimum away from any extreme corner of that valley and toward the center, where coefficients are smaller and more evenly distributed.

The practical effect: ridge shrinks all coefficients toward zero, proportionally, but it essentially never sets any of them to exactly zero, no matter how large α gets (short of being infinite). Every feature stays in the model, just with a dampened, more conservative influence.

This stabilizes predictions and reduces variance, particularly when features are correlated with each other. Instead of arbitrarily assigning most of the credit to one of two correlated features (the multicollinearity problem from the previous lesson), ridge tends to split the credit between them more evenly, because having two moderately sized coefficients is cheaper under the squared penalty than having one large one and one near zero.

Lasso Regression: The L1 Penalty, and Why It Zeroes Coefficients

Lasso regression uses the same idea but penalizes the absolute value of the coefficients instead of their square:

Loss = Sum of Squared Errors + α · (|b1| + |b2| + ... + |bn|)

This looks like a small change, but it has a dramatic practical consequence: lasso can shrink coefficients all the way to exactly zero, effectively removing those features from the model entirely. Understanding why requires the actual geometry, not just a label.

A Worked Comparison: Same Total, Different Penalty

Compare two candidate coefficient pairs that both sum their values to 4: (b1, b2) = (3, 1) and (b1, b2) = (2, 2). Computing each penalty by hand:

Under the L2 penalty, (2, 2) is strictly cheaper than (3, 1), which is strictly cheaper than (4, 0). Ridge has a real, numeric preference for spreading weight out evenly.

Under the L1 penalty, all three options cost exactly the same, 4. Lasso has no preference at all between spreading the weight out and dumping all of it onto a single coefficient, which is exactly why the optimizer is free to land on a corner solution like (4, 0), where one coefficient is exactly zero, without that solution costing anything extra under the L1 penalty.

The Geometric Reason: Diamonds vs. Circles

Minimizing squared error subject to a penalty is mathematically equivalent to minimizing squared error subject to a hard constraint on the total "size" of the coefficients (this equivalence is a standard result from constrained-optimization theory, called Lagrangian duality). For ridge, the constraint region, the set of coefficient values allowed within a given budget, is defined by b1² + b2² ≤ budget, which is the equation of a circle (or sphere, in higher dimensions) centered at the origin.

For lasso, the constraint region is |b1| + |b2| ≤ budget, which is the equation of a diamond (a square rotated 45°) centered at the origin.

Now picture the unconstrained least-squares solution sitting somewhere outside this region, surrounded by elliptical contour lines of equal squared error (like rings around a target, getting worse the further you move from the unconstrained optimum). The regularized solution is the point where the smallest possible error ellipse just touches the constraint region's boundary.

Here is the crux. A circle (ridge's constraint region) has no corners, it's smooth everywhere. An error ellipse touching a smooth circle will, in general, touch it at some interior point of the circle's edge where neither coordinate is exactly zero, because there's nothing geometrically special about the axes.

A diamond (lasso's constraint region), on the other hand, has sharp corners exactly on the coordinate axes, at points like (budget, 0) and (0, budget). Because the diamond's corners stick out further toward the axes relative to its flat edges, an error ellipse expanding outward from the unconstrained optimum is disproportionately likely to first touch the diamond exactly at one of those corners, a point where one coefficient is exactly zero.

This is not a coincidence or an approximation. It's a direct geometric consequence of the constraint region having corners on the axes versus being perfectly round. This is the actual reason lasso performs feature selection and ridge does not. It has nothing to do with one penalty being "more aggressive," and everything to do with the shape of the constraint region the penalty defines.

This gives lasso a genuinely useful property: automatic feature selection. If you have a dataset with fifty candidate features but suspect only a handful actually matter, lasso will often zero out the irrelevant ones for you, leaving a sparse, more interpretable model, valuable when you care about understanding which features matter, not just maximizing predictive accuracy.

Choosing Between Them

As a rule of thumb: use ridge when you believe most or all of your features are at least somewhat relevant and you mainly want to stabilize the model and reduce variance, especially in the presence of correlated features. Use lasso when you suspect many features are irrelevant or redundant and you want the model to actively select a smaller subset.

One important caveat with lasso: when features are highly correlated with each other, lasso's geometric corner-picking tends to arbitrarily select one of the correlated group and zero out the rest. Which one gets picked can be unstable across slightly different training samples. The next lesson introduces Elastic Net, which blends both penalties to get the best of both worlds.

Comparing Ridge and Lasso Across Alpha Values

import numpy as np
from sklearn.linear_model import Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# Synthetic example: 8 features, but only 3 are actually informative
np.random.seed(0)
X = np.random.randn(200, 8)
true_coef = np.array([5, -3, 2, 0, 0, 0, 0, 0])
y = X @ true_coef + np.random.randn(200) * 0.5

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

print(f"{'alpha':>8} | {'model':>6} | {'test R2':>8} | coefficients (rounded)")
print("-" * 70)
for alpha in [0.01, 0.1, 1.0, 10.0]:
    ridge = Ridge(alpha=alpha).fit(X_train_scaled, y_train)
    lasso = Lasso(alpha=alpha).fit(X_train_scaled, y_train)

    r2_ridge = ridge.score(X_test_scaled, y_test)
    r2_lasso = lasso.score(X_test_scaled, y_test)

    print(f"{alpha:>8} | {'ridge':>6} | {r2_ridge:>8.3f} | {np.round(ridge.coef_, 2)}")
    print(f"{alpha:>8} | {'lasso':>6} | {r2_lasso:>8.3f} | {np.round(lasso.coef_, 2)}")

Run this and watch two things as alpha grows: ridge's coefficients shrink smoothly and proportionally, but every one of the 8 stays nonzero even at alpha=10. Lasso's coefficients for the 5 truly irrelevant features (positions 4 through 8 in true_coef) get driven to exactly 0.0 even at moderate alpha values, recovering something close to the true sparse structure used to generate the data, while ridge never gets there, by design.