R-Squared Explained

What you will learn in this lesson:

• What "variance explained" means intuitively, using a naive baseline model
• The R² formula and how to read values between 0 and 1 (and occasionally below 0)
• Why R² always increases when you add features, even useless ones
• What adjusted R² fixes, with a worked numeric example
• How to compute R² and adjusted R² directly from a fitted scikit-learn model

What "Variance Explained" Means

Suppose you want to predict house prices and you know nothing about any individual house, no size, no location, no age. Your best strategy with no other information is to guess the average price every time.

This naive "always guess the mean" approach will still be wrong for almost every house, but the total amount of that "wrongness" across your whole dataset gives you a natural baseline against which any real model should be judged. If your model can't beat simply guessing the average, it isn't adding any value at all.

R², the coefficient of determination, measures how much better your model does compared to that naive baseline. Specifically, it answers: what fraction of the variation in the target variable can be explained by the model's inputs, rather than left over as unexplained randomness or noise?

An R² of 0.75 means your model explains 75% of the variation in house prices using the features you gave it; the remaining 25% is variation your model can't account for, whether due to missing features, genuine randomness, or model limitations. R² is unitless and bounded above by 1, which makes it convenient for comparing across very different regression problems in a way that raw error metrics like RMSE, which stay in the original units, can't.

The Formula and How to Read It

R² is computed as:

R² = 1 - (Sum of Squared Residuals / Total Sum of Squares)

The numerator, sum of squared residuals (sometimes written SSR or SSE), is the total squared error your actual model makes: sum over every data point of (actual − predicted)². The denominator, total sum of squares (SST), is the total squared error the naive "always guess the mean" baseline would make: sum over every point of (actual − mean)².

If your model is no better than always guessing the mean, the two sums are equal, the ratio is 1, and R² is 0. If your model predicts every value perfectly, the numerator is 0 and R² is 1. Everything in between is some fraction of variance your model managed to "capture" relative to that naive baseline.

In practice, R² typically falls somewhere between 0 and 1, though it technically can go negative if a model performs worse than the naive mean baseline. This sounds strange but does happen: a badly misspecified model, or a model evaluated on data that looks very different from what it was trained on (a classic sign of distribution shift), can produce predictions worse than simply guessing the average every time.

A negative R² is a strong red flag that something in your modeling pipeline is broken, not just imperfect.

What counts as a "good" R² is entirely domain-dependent, and there's no universal threshold. In physics or engineering with controlled experiments and clean measurement, an R² below 0.95 might be considered poor, the underlying relationships are often near-deterministic.

In social science or behavioral prediction, where human behavior is inherently noisy and influenced by countless unmeasured factors, an R² of 0.3 might be considered a genuinely useful and publishable result. Always interpret R² relative to what's typically achievable for similar problems in your specific field, never against some abstract universal bar.

The Problem: R² Always Goes Up

Here's a subtle but important trap. If you add any additional feature to a linear regression model, even one that's completely unrelated to the target (literally random noise), R² on the training data will never decrease, it will either stay exactly the same or increase slightly.

This happens because the model gains more flexibility to fit the training data exactly, and with enough random features, the optimizer can always find some small accidental correlation in the training sample to exploit, purely by chance.

This makes plain R² dangerous for comparing models with different numbers of features. A model with twenty features will almost always show a higher training R² than a model with five features, even if fifteen of those extra features are pure noise that will actively hurt the model's performance on new, unseen data.

If you used R² alone to decide whether to add features, you'd be systematically tempted to keep adding more and more, overfitting along the way while your training-set R² climbs reassuringly toward 1, all the while your model's real-world usefulness quietly degrades.

Adjusted R²: Penalizing Extra Features

Adjusted R² fixes this by explicitly penalizing the score for each additional feature added to the model, unless that feature improves the fit by more than what would be expected from chance alone:

Adjusted R² = 1 - [ (1 - R²) * (n - 1) / (n - p - 1) ]

where n is the number of data points and p is the number of features. As you add features that don't meaningfully improve the fit, adjusted R² will actually decrease, even while plain R² ticks upward, giving you a much more honest signal about whether a new feature is genuinely earning its place in the model or just exploiting noise.

As a worked example: suppose a model with 5 features on 100 data points achieves R² = 0.82. Plugging into the formula:

Adjusted R² = 1 − [(1 − 0.82) × 99 / 94] = 1 − [0.18 × 1.053] = 1 − 0.190 = 0.810

Now suppose you add five more, essentially useless, features and R² ticks up slightly to 0.83, but now p = 10:

Adjusted R² = 1 − [(1 − 0.83) × 99 / 89] = 1 − [0.17 × 1.112] = 1 − 0.189 = 0.811

The improvement is negligible once you account for the extra complexity, a strong hint that those five extra features weren't worth adding. If instead the new features had pushed R² up to, say, 0.90 with the same ten features, adjusted R² would climb meaningfully too, telling you the added complexity was genuinely earning its keep.

Always prefer adjusted R² over plain R² whenever you're comparing models with different numbers of features, and treat a rising plain R² with suspicion whenever it isn't accompanied by a similarly rising adjusted R².

Computing Both in Practice

scikit-learn's fitted regressors expose .score(), which returns plain R² directly. Adjusted R² isn't built in, but it's a one-line calculation once you have R², n, and p. The example below fits two models on the same data, one with a handful of genuinely useful features, one with several additional noise features bolted on, and compares how plain R² and adjusted R² react differently:

import numpy as np
from sklearn.linear_model import LinearRegression

def adjusted_r2(r2, n, p):
    return 1 - (1 - r2) * (n - 1) / (n - p - 1)

np.random.seed(0)
n = 100

# Two genuinely useful features driving the target
X_useful = np.random.randn(n, 2)
y = 3 * X_useful[:, 0] - 2 * X_useful[:, 1] + np.random.randn(n) * 0.5

# Model A: just the useful features
model_a = LinearRegression().fit(X_useful, y)
r2_a = model_a.score(X_useful, y)
adj_r2_a = adjusted_r2(r2_a, n, p=2)

# Model B: the same data, plus 10 columns of pure random noise
X_noise = np.random.randn(n, 10)
X_padded = np.hstack([X_useful, X_noise])
model_b = LinearRegression().fit(X_padded, y)
r2_b = model_b.score(X_padded, y)
adj_r2_b = adjusted_r2(r2_b, n, p=12)

print("Model A (2 useful features):  R2 = {:.4f}  Adjusted R2 = {:.4f}".format(r2_a, adj_r2_a))
print("Model B (+10 noise features): R2 = {:.4f}  Adjusted R2 = {:.4f}".format(r2_b, adj_r2_b))

Run this and you'll typically see plain R² for Model B come out slightly higher than Model A, purely from the extra flexibility of fitting ten more (useless) coefficients to the training noise, while adjusted R² for Model B comes out lower, correctly flagging that the additional complexity wasn't worth it. This is exactly the failure mode adjusted R² exists to catch, and it's worth running this snippet a few times with different random seeds to see how consistently the pattern holds.