Overfitting and Generalization

What you will learn in this lesson:

• What overfitting and underfitting actually look like in train vs. validation error curves, and mechanically why the crossover happens
• Why a single train/test split isn't enough, and what the validation set is really for
• How k-fold cross-validation works, traced through a concrete 5-fold example on 20 samples
• The practical fixes for overfitting, and the specific reason each one works

What Overfitting and Underfitting Actually Look Like

Picture an experiment: train a model, and after every step of training (or at every level of model complexity), measure its error twice, once on the data it was trained on (training error) and once on a separate batch of data it has never seen (validation error). Plot both curves against complexity or training time on the x-axis. The shape of those two curves tells you almost everything you need to know about how a model is behaving.

The underfitting regime

At low complexity, both curves sit high on the error axis and stay close together. A straight line trying to fit a clearly curved relationship is the textbook case: it does a mediocre job on the training data, and an equally mediocre job on validation data, because the model's hypothesis space simply doesn't contain anything close to the true relationship.

There is no "memorization" happening yet. The model is too constrained to memorize anything beyond the broad strokes. Underfitting is recognizable by this signature: training error and validation error are both high, and the gap between them is small.

The crossover, and why it happens

As complexity increases (more polynomial terms, deeper trees, more training epochs, less regularization), training error falls steadily, often toward zero, because a more flexible model can bend itself around more and more of the quirks of the specific training points it sees. Validation error falls too, for a while. That initial joint improvement happens because some of the complexity the model is adding really does capture genuine, generalizable structure in the data.

But past a certain point, the validation curve does something the training curve never will: it turns around and starts rising, even while training error keeps dropping. This happens because, once a model has enough capacity to capture all the real signal, any additional capacity has nothing left to do except fit noise, the specific random fluctuations, measurement error, and accidental patterns of exactly those training rows.

Noise by definition does not repeat in a new sample, so a model that has started memorizing it will do worse, not better, on validation data, even as it looks like it's improving by every metric computed on the training set. The widening gap between a still-falling training error and a rising validation error is the signature of overfitting.

The model you actually want to ship sits at the bottom of the validation curve, the complexity level right before that curve turns upward, not at the bottom of the training curve, which keeps falling long after it has stopped being a useful guide. This is the single most important practical consequence of this whole lesson: training error is not a trustworthy signal for choosing model complexity, regularization strength, or training duration. Validation error is.

Why You Need Three Separate Splits, Not Two

A single train/test split is the bare minimum for evaluating a model honestly, but it is not enough once you start making choices about the model: which value of K in KNN, how many trees in a forest, how much regularization to apply. In practice you need three distinct subsets, each with a strictly different job:

• Training set. Used to actually fit the model's parameters: coefficients, tree splits, support vectors, whatever the algorithm learns directly from data.
• Validation set. Used to make decisions about the model, comparing K=3 vs. K=15 in KNN, or alpha=0.1 vs. alpha=10 in ridge regression, by checking which setting scores best on data the model wasn't fit on.
• Test set. Held back untouched until the very end, used exactly once, to report an honest estimate of how the final, already-chosen model will perform on genuinely new data.

The reason validation and test must stay separate is subtle but real. If you tune hyperparameters by checking performance on a set, and then report your final number on that same set, you've let the model's selection process, which is driven by your own iterative choices, implicitly adapt to that data's particular quirks.

You'll tend to land on whichever hyperparameter setting happens to look best on that one validation draw, which is a mildly optimistic, overstated estimate of true future performance. A test set that nothing ever got to influence is the only thing that protects against this. It's a one-shot check, not a tuning knob.

K-Fold Cross-Validation, Traced Through a Worked Example

Carving out a fixed validation set is wasteful when data is scarce, since that's data your model never trains on at all. K-fold cross-validation reuses the same data for both training and validation, just never on the same iteration, which gives a far more data-efficient and more stable estimate.

The mechanics, step by step

Suppose you have 20 samples and choose k=5. Here is exactly what happens:

1. Shuffle the 20 samples and split them into 5 folds of 4 samples each: Fold 1 (samples 1-4), Fold 2 (samples 5-8), Fold 3 (samples 9-12), Fold 4 (samples 13-16), Fold 5 (samples 17-20).
2. Round 1: train on Folds 2-5 (16 samples), validate on Fold 1 (4 samples). Record the validation score.
3. Round 2: train on Folds 1, 3, 4, 5 (16 samples), validate on Fold 2 (4 samples). Record the score.
4. Repeat for Rounds 3, 4, and 5, each time leaving out a different fold as the validation set.
5. You now have 5 validation scores, one per round. Average them into a single number. This is your cross-validated estimate of model performance.

Notice the key property: every one of the 20 samples gets used for training in 4 of the 5 rounds, and for validation in exactly 1 round. No sample is ever used for both training and validation at the same time, which is what keeps the estimate honest, but every sample still contributes to both roles across the full process, which is what makes it data-efficient.

Averaging across 5 rounds also smooths out the luck of any single split: a model that happens to score unusually well or badly on one particular train/validation partition gets balanced out by the other four.

This is especially valuable when comparing several candidate models or hyperparameter settings against each other, since a single train/validation split might accidentally favor one setting just because of how the data happened to fall. A held-out test set is still kept completely outside this entire cross-validation loop, for the final, once-only performance check.

import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.linear_model import Ridge
from sklearn.model_selection import cross_val_score

# Use a small slice of the diabetes dataset to keep folds easy to reason about
X, y = load_diabetes(return_X_y=True)
X_small, y_small = X[:50], y[:50]

model = Ridge(alpha=1.0)
scores = cross_val_score(model, X_small, y_small, cv=5, scoring="r2")

print("Per-fold R^2 scores:", np.round(scores, 3))
print("Mean R^2 across folds: {:.3f}".format(scores.mean()))
print("Std across folds:      {:.3f}".format(scores.std()))

Run it and look at the spread between folds, not just the mean. A small standard deviation across folds means the model's performance estimate is stable; a large one means your evaluation is noisier than the single mean number suggests, and you should be skeptical of small differences when comparing models.

Practical Warning Signs and Fixes

The clearest practical sign of overfitting is a large gap between training and validation performance, say, 98% training accuracy against 80% validation accuracy. A smaller gap that grows as you add complexity is also a warning sign worth acting on before it becomes a large one.

• Get more training data. More genuine examples make any single noise pattern a smaller fraction of the dataset, so a model has less to gain, relatively, from memorizing it.
• Add regularization (ridge, lasso, Elastic Net). These penalties explicitly constrain how large or extreme the model's parameters can get, which removes some of the flexibility a model would otherwise use to chase noise.
• Simplify the model. Fewer features, shallower trees, or a less flexible algorithm directly reduces capacity, leaving less room to fit anything beyond the real signal.
• Use early stopping. For models trained iteratively (gradient boosting, neural networks), stop once validation performance plateaus or starts rising, instead of training for a fixed number of rounds regardless of what validation error is doing.
• Use more folds, or cross-validation generally, when tuning. This doesn't fix an overfit model directly, but it stops you from picking hyperparameters that merely happen to overfit one particular validation split.

Generalization is the entire point of supervised learning. A model earns its keep only on data it has never seen. Every idea that follows in this course, regularization, ensembling, careful metric selection, exists in service of that one goal: making the validation curve, not the training curve, look as good as possible.