Random Forests and an Introduction to Boosting

What you will learn in this lesson:

• The bagging intuition behind random forests, and the extra trick that makes them work better than plain bagging
• Why averaging many decorrelated trees reduces variance, the actual statistical reasoning, with a worked numeric example
• Feature importance as a useful side-output of tree ensembles, and how to read it
• A first look at boosting as a contrasting, sequential ensemble idea

The Problem With a Single Deep Tree

In the previous lesson, you saw that a single, deep decision tree is prone to overfitting: it has low bias (it can capture very specific patterns) but high variance (small changes in the training data lead to a very different tree). One way to tame this is to constrain the tree's depth, accepting more bias in exchange for less variance. Random forests take an entirely different approach: instead of constraining a single tree, they build many deep, flexible trees and average their predictions together.

Bagging: Many Trees, Averaged Together

The core technique behind random forests is called bagging, short for bootstrap aggregating. Here's the recipe:

• Create many random subsamples of your training data, drawn with replacement (this is called bootstrapping. Some data points will appear multiple times in a given subsample, others not at all).
• Train a separate, usually unconstrained, decision tree on each subsample.
• For classification, combine all the trees' predictions by majority vote; for regression, by averaging.

Random forests add one more twist on top of plain bagging: at each split within each tree, instead of considering every available feature, the algorithm only considers a random subset of features (commonly the square root of the total feature count).

This might seem like it would hurt performance, since each individual tree now has less information to work with, but it serves a crucial purpose: it forces the trees to be different from one another, rather than all converging on the same dominant feature for their first split.

Why Averaging Decorrelated Trees Reduces Variance

Here's the key statistical insight: if you average many noisy, high-variance estimates that are largely independent of each other, the noise tends to cancel out while the genuine signal, which all the trees roughly agree on, reinforces itself. A small worked example makes this concrete. Suppose a single tree's prediction has variance σ². If you average n independent trees, the variance of the average drops to σ²/n. Average 100 fully independent trees and you cut variance by a factor of 100.

But trees trained on the same dataset are never fully independent. They're correlated by some amount ρ. The actual formula for the variance of the average becomes ρσ² + (1−ρ)σ²/n. As n grows large, the second term shrinks toward zero, but the first term, ρσ², doesn't go away. It's a floor set entirely by how correlated the trees are.

This is precisely why the feature-subsampling trick matters: it directly lowers ρ, which lowers the variance floor that no amount of additional trees can get you past. If every tree were allowed to consider all features at every split, they'd likely all make very similar decisions at the top of the tree (everyone picks the single most predictive feature first), keeping ρ high and limiting how much averaging can help, no matter how many trees you add.

Feature Importance: A Useful Side-Output

Beyond just making predictions, random forests naturally produce a ranking of which features mattered most across all the trees in the ensemble. This is typically computed by measuring, on average across all trees, how much each feature's splits reduced impurity (Gini impurity or entropy, from the previous lesson), weighted by how many samples passed through that split. Features that consistently provide strong, useful splits across many trees will have high importance scores.

This makes random forests not just a strong predictive model but also a useful diagnostic tool: even before worrying about final model performance, you can train a quick random forest on your data and inspect feature importances to get a sense of which variables are actually carrying predictive signal. This is useful for guiding further feature engineering or simply understanding your dataset.

One caveat worth knowing: importance scores can be misleading when features are highly correlated with each other (similar to the multicollinearity problem from the regression lessons). The "credit" for a good split can get arbitrarily split between two correlated features, making both look less important individually than the underlying signal actually is.

Random Forest in Python

Run the cell below to fit a random forest on a small synthetic dataset and inspect both its accuracy and its feature importances.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split

np.random.seed(0)
X = np.random.randn(300, 5)
y = ((X[:, 0] + X[:, 1] * 0.5 - X[:, 2] > 0)).astype(int)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

forest = RandomForestClassifier(n_estimators=200, max_depth=None, random_state=0)
forest.fit(X_train, y_train)

print("Test accuracy:", forest.score(X_test, y_test))
print("Feature importances:", np.round(forest.feature_importances_, 3))

Notice that the individual trees inside the forest (accessible via forest.estimators_) are typically grown deep and unconstrained. Each one individually overfits, but the forest as a whole generalizes well because of the averaging effect. You should also see that features 0, 1, and 2 (the ones actually used to construct y) get noticeably higher importance scores than features 3 and 4, which are pure noise. That's a quick sanity check that the model is picking up on real signal rather than memorizing noise.

A Contrasting Idea: Boosting

Random forests build many trees in parallel, independently of each other, and combine them by averaging. Boosting takes a fundamentally different, sequential approach: it builds trees one at a time, where each new tree is specifically trained to correct the mistakes made by the ensemble built so far.

The process roughly works like this: train a first, usually very simple, tree. Look at where it made errors. Train a second tree that focuses specifically on those errors, essentially trying to fix what the first tree got wrong. Add the second tree's corrections to the first tree's predictions. Repeat, each new tree targeting whatever mistakes remain after all the previous trees' combined predictions. Over many rounds, the ensemble's combined predictions get progressively more accurate.

Where random forests reduce variance by averaging many independent, high-variance trees, boosting reduces bias by sequentially layering simple models that each chip away at the remaining error. Both are powerful ensemble strategies built on the same humble decision tree, just combined in opposite ways. The next lesson dives into gradient boosting, the mechanism behind some of the most effective tabular-data models used in practice.