Multiple Linear Regression

What you will learn in this lesson:

• How to extend a single-feature line into a multi-feature linear model
• How feature scaling changes what a coefficient actually means, with a concrete before/after comparison
• What multicollinearity is, demonstrated with a concrete correlated-feature example showing coefficients swing wildly
• How to detect multicollinearity with a correlation matrix and the Variance Inflation Factor
• Runnable code fitting a three-feature model with scikit-learn

From One Feature to Many

In the previous lesson, we predicted exam scores from a single input: hours studied. Real prediction problems almost always involve several relevant inputs at once. A house's price doesn't just depend on its size. It also depends on the number of bedrooms, the neighborhood, the age of the building, and more.

Multiple linear regression generalizes the line equation by adding more terms, one per feature:

y = b0 + b1·x1 + b2·x2 + ... + bn·xn

Here, b0 is the intercept (the baseline prediction when every feature is zero), and each b1, b2, ..., bn is a coefficient telling you how much the prediction changes for a one-unit increase in that specific feature, holding all other features constant. That last phrase is the crux of multiple regression: the coefficient for "size in square meters" tells you the effect of size specifically, after accounting for whatever effect bedrooms, location, and age already had.

The model is still "linear" because the prediction is a weighted sum of the inputs, even though it's no longer a single 2D line. Geometrically, with two features it's a flat plane cutting through 3D space, and with more features it's a hyperplane in higher dimensions you can no longer visualize directly, though the math works identically.

How Feature Scaling Changes Coefficient Interpretation

Suppose you're predicting house prices using square footage (ranging roughly 50 to 500) and number of bedrooms (ranging 1 to 6). Fit ordinary least squares directly on these raw units, and you'll get a coefficient for square footage that's small (maybe around 2,000 dollars per extra square meter) and a coefficient for bedrooms that looks huge (maybe 15,000 dollars per extra bedroom).

It's tempting to read this as "bedrooms matter way more than size," but that comparison is meaningless across features measured in different units. A one-unit change in square meters is a tiny nudge; a one-unit change in bedroom count is a substantial jump. You're comparing apples to oranges.

Standardization fixes this comparison problem: subtract each feature's mean and divide by its standard deviation, so every feature ends up with mean 0 and standard deviation 1. After standardizing, a coefficient now answers a directly comparable question: "if this feature increases by one standard deviation, holding others constant, how much does the prediction change?"

Now coefficients across features are on the same footing, and you can legitimately say which feature has the larger standardized effect. This doesn't change the model's predictions at all. It's the same plane fit through the same data, just re-expressed in different units, but it changes what the coefficient numbers mean and whether they're comparable.

Beyond interpretability, scaling matters for practical reasons too. Many optimization algorithms used to fit models (like gradient descent) converge much faster and more reliably when features are on comparable scales. Regularized models (the next two lessons) penalize coefficient size directly, so an unscaled feature on a larger numeric range gets penalized unfairly differently than the same relationship expressed on a smaller scale.

Distance-based algorithms like K-Nearest Neighbors, covered later in this course, are extremely sensitive to scale for a related reason: a feature ranging in the thousands will dominate a feature ranging from 0 to 1 in any distance calculation.

Multicollinearity: When Features Overlap

Multicollinearity occurs when two or more input features are highly correlated with each other, meaning they carry overlapping, redundant information. Consider a concrete case: predicting house price using both "total square footage" and "number of rooms." Bigger houses almost always have more rooms, so these two features move together strongly, say with a correlation of 0.95.

Here is precisely why that breaks coefficient interpretation. The model is trying to answer "how much does an extra unit of square footage add to price, holding rooms constant" and separately "how much does an extra room add, holding square footage constant."

But in the data, square footage and rooms almost never vary independently of each other. When one goes up, the other almost always goes up too. The model has very little information about what happens when one changes while the other stays fixed, because that combination barely appears in the data.

As a result, the optimizer can distribute the "credit" for predicting price between the two correlated features almost arbitrarily. One possible solution gives most of the credit to square footage and a small, perhaps even negative, coefficient to rooms. Another equally valid-looking solution gives most of the credit to rooms instead. Both solutions predict house prices almost equally well overall, but they tell wildly different, contradictory stories about which feature "matters."

Worse, a tiny change in the training sample, adding or removing a handful of houses, can flip which feature gets the larger coefficient, or even flip a coefficient's sign entirely, even though the model's overall predictions barely change.

This is the key distinction to hold onto: multicollinearity rarely hurts the model's overall predictive accuracy on data similar to what it was trained on, but it makes the individual coefficients unstable and untrustworthy for interpretation.

This becomes a real problem the moment you want to say something like "each extra bedroom adds $15,000 to price." With correlated features, that specific number could have easily come out as $2,000 or $40,000 with a slightly different training sample, even though the model's predictions were equally good in both cases.

Detecting Multicollinearity

The simplest check is to compute the correlation matrix of your features and look for pairs with a correlation coefficient above roughly 0.8 to 0.9 in magnitude. A more rigorous tool is the Variance Inflation Factor (VIF), computed for each feature by regressing it against all the other features and seeing how well they predict it. A VIF above about 5 to 10 is typically considered a warning sign that a feature is largely redundant with the others.

Common fixes include dropping one of the redundant features, combining correlated features into a single composite feature (for example, "price per square foot" instead of both raw inputs), or using regularized regression, covered in the next two lessons, which naturally stabilizes coefficients in the presence of correlated inputs rather than letting the optimizer pick an arbitrary split of credit.

Fitting a Three-Feature Model in Python

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import StandardScaler

# Features: size_sqm, bedrooms, age_years
X = np.array([
    [50,  1, 10],
    [80,  2, 5],
    [120, 3, 15],
    [60,  2, 20],
    [150, 4, 2],
    [200, 4, 8],
    [95,  3, 12],
    [175, 5, 3],
])
y = np.array([150000, 220000, 300000, 200000, 400000, 480000, 260000, 430000])

feature_names = ["size_sqm", "bedrooms", "age_years"]
df = pd.DataFrame(X, columns=feature_names)
print("Correlation matrix (check for multicollinearity):")
print(df.corr().round(2))

# Scale features before fitting -- good habit, and essential once
# you compare coefficient magnitudes across features
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

model = LinearRegression()
model.fit(X_scaled, y)

print("\nStandardized coefficients:")
for name, coef in zip(feature_names, model.coef_):
    print(f"  {name}: {coef:.1f}")
print("Intercept:", round(model.intercept_, 1))

# Predict price for a new house: 100 sqm, 3 bedrooms, 7 years old
new_house = scaler.transform([[100, 3, 7]])
print("\nPredicted price:", round(model.predict(new_house)[0], 2))

Run this and inspect the correlation matrix first. If size_sqm and bedrooms show a high correlation in your own dataset, treat the individual coefficients with suspicion even if the overall predictions look reasonable.

Looking Ahead

With multiple features in play, always sanity-check your coefficients against domain knowledge, and check for multicollinearity before trusting any individual coefficient's interpretation. The next two lessons introduce ridge and lasso regularization, which directly address the instability that comes from too many or too correlated features, not by removing the correlation in your data, but by changing what the optimizer is allowed to do with it.