Elastic Net Regularization

What you will learn in this lesson:

• How Elastic Net combines ridge's L2 penalty and lasso's L1 penalty into one loss function
• What the l1_ratio parameter controls, and how to think about tuning it
• Exactly how correlated features break pure lasso's feature selection, with a concrete mechanism, not just a warning
• How Elastic Net's L2 component fixes that instability
• Runnable code comparing Elastic Net against pure lasso on a dataset with correlated feature groups

Blending Two Penalties

The previous lesson covered two regularization strategies. Ridge shrinks all coefficients smoothly toward zero using a squared (L2) penalty, whose circular constraint region rarely produces an exact zero. Lasso can zero out coefficients entirely using an absolute-value (L1) penalty, whose diamond-shaped constraint region has corners exactly on the coordinate axes, which is geometrically why it performs feature selection.

Each has a real weakness in practice. Ridge never eliminates any feature, even ones that are genuinely useless, which hurts interpretability when you have many candidate features. Lasso, when faced with a group of correlated features, tends to arbitrarily pick one of them and zero out the rest, which makes its feature selection unstable from one training run to the next.

Elastic Net resolves this by combining both penalties into a single loss function:

Loss = Sum of Squared Errors + α · [ l1_ratio · (|b1| + ... + |bn|) + (1 − l1_ratio) · (b1² + ... + bn²) ]

This gives you the sparsity-inducing behavior of lasso (the ability to drop irrelevant features entirely) alongside the stabilizing, "share the credit" behavior of ridge when features are correlated with each other.

The l1_ratio Parameter

The mix between the two penalties is controlled by a parameter usually called l1_ratio, ranging from 0 to 1:

• l1_ratio = 1.0 makes Elastic Net behave exactly like pure lasso (the L2 term's weight drops to zero).
• l1_ratio = 0.0 makes Elastic Net behave exactly like pure ridge (the L1 term's weight drops to zero).
• Any value in between (commonly 0.5 as a starting point) blends both penalties, gaining some feature selection while keeping correlated features more stable than lasso alone.

In practice, you typically tune both alpha (overall penalty strength) and l1_ratio (the blend) together using cross-validation, since the best combination depends heavily on your specific dataset: how many features are genuinely redundant versus genuinely irrelevant, and how strongly the redundant ones are correlated with each other.

Exactly How Correlated Features Break Pure Lasso

Recall the geometric picture from the previous lesson: lasso's diamond-shaped constraint region has corners on the coordinate axes, and the regularized solution tends to land on one of those corners, zeroing out whichever coefficients correspond to flat, non-corner directions. Now consider two features, x1 and x2, that are almost perfectly correlated with each other. Say, two slightly different lab measurements of essentially the same underlying biological quantity, with correlation 0.98.

Because x1 and x2 carry almost identical information, the model can achieve essentially the same prediction quality whether it assigns weight (3, 0), (0, 3), or (1.5, 1.5) to the pair. Any combination that sums to roughly the same total influence produces nearly the same predictions, since the two features move together.

With lasso's diamond constraint, several of these combinations sit equally close to the corner of the diamond, and which exact corner the optimizer lands on becomes sensitive to tiny numerical details: the specific noise in this particular training sample, the exact correlation in this particular batch of data, even floating-point rounding in the solver.

Retrain lasso on a slightly different sample of the same underlying population, and it might now assign all the weight to x2 and zero out x1 instead, the opposite choice, even though the underlying relationship in the world hasn't changed at all. This is the concrete mechanism behind the often-repeated warning that "lasso is unstable with correlated features." It isn't a vague caveat. It's a direct consequence of the diamond having many nearly-equally-good corners to choose between when features are redundant.

Elastic Net's ridge component breaks this tie in a principled way. Adding even a modest squared penalty makes the combination (1.5, 1.5) strictly cheaper than (3, 0) or (0, 3), because 1.5² + 1.5² = 4.5 is less than 3² + 0² = 9. So the optimizer is now actively rewarded for splitting the weight evenly across the correlated group rather than arbitrarily picking one.

The result: correlated features that are jointly useful tend to be kept together with similar, stable coefficients, while features that are genuinely irrelevant (uncorrelated with the target and with everything else) still get pushed to exactly zero by the L1 component.

When to Prefer Elastic Net

Elastic Net is particularly useful in exactly the situation that breaks lasso: many features, some of them highly correlated with each other. A genomics dataset with thousands of gene-expression measurements is the textbook case. Groups of genes that participate in the same biological pathway tend to be correlated, since they're co-regulated.

Pure lasso would arbitrarily select one gene from each correlated group and discard the rest, which makes the result hard to trust or reproduce. A slightly different sample of patients might lead lasso to pick a completely different gene from the same pathway. Elastic Net's ridge component encourages correlated features to share the weight more evenly, producing a more stable, more reproducible selection, while its lasso component still drops the features that are genuinely irrelevant to the outcome.

As a general guideline: if you have more features than data points, or features that come in naturally correlated groups, Elastic Net is usually a safer default starting point than lasso alone.

Comparing Elastic Net Against Pure Lasso on Correlated Data

import numpy as np
from sklearn.linear_model import ElasticNet, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

np.random.seed(0)
n_samples = 200

# Build two correlated feature pairs (each pair shares an underlying signal)
# plus six pure-noise features that are genuinely irrelevant
base_a = np.random.randn(n_samples)
base_b = np.random.randn(n_samples)
X = np.column_stack([
    base_a + np.random.randn(n_samples) * 0.1,   # x0: correlated with x1
    base_a + np.random.randn(n_samples) * 0.1,   # x1: correlated with x0
    base_b + np.random.randn(n_samples) * 0.1,   # x2: correlated with x3
    base_b + np.random.randn(n_samples) * 0.1,   # x3: correlated with x2
    np.random.randn(n_samples, 6),                # x4-x9: pure noise, irrelevant
])
true_coef = np.array([3, 3, -2, -2, 0, 0, 0, 0, 0, 0])
y = X @ true_coef + np.random.randn(n_samples) * 0.5

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

lasso = Lasso(alpha=0.1).fit(X_train_scaled, y_train)
elastic = ElasticNet(alpha=0.1, l1_ratio=0.5).fit(X_train_scaled, y_train)

print("True coefficients:        ", true_coef)
print("Lasso coefficients:       ", np.round(lasso.coef_, 2))
print("Elastic Net coefficients: ", np.round(elastic.coef_, 2))
print()
print("Lasso test R2:       ", round(lasso.score(X_test_scaled, y_test), 3))
print("Elastic Net test R2: ", round(elastic.score(X_test_scaled, y_test), 3))

print()
print("Look at the first correlated pair (x0, x1), true contribution split evenly as (3, 3):")
print(f"  Lasso:       x0={lasso.coef_[0]:.2f}, x1={lasso.coef_[1]:.2f}")
print(f"  Elastic Net: x0={elastic.coef_[0]:.2f}, x1={elastic.coef_[1]:.2f}")

Run this a few times with different random_state values in train_test_split. You'll typically see lasso assign weight unevenly within each correlated pair, sometimes giving most of the credit to x0 and almost none to x1, sometimes the reverse, while Elastic Net spreads the weight more evenly and consistently between the two correlated features, even as it still drives the six genuinely irrelevant noise features close to zero. That contrast is the entire practical case for Elastic Net in one comparison.

Where This Leaves You

Across this course's regularization arc, you've now seen the same underlying idea applied three ways: ridge trades bias for variance by shrinking everything smoothly, lasso trades bias for variance and sparsity by shrinking some coefficients all the way to zero, and Elastic Net blends the two to get sparsity without lasso's instability under correlated features.

The right choice is never automatic. It depends on whether you believe most features are relevant, whether you expect strong correlation among them, and whether you care more about raw predictive accuracy or about a clean, interpretable, reproducible set of selected features.