Classifiers Head-to-Head: KNN vs. Tree vs. Forest vs. Gradient Boosting

Objective: Train KNN, a single decision tree, a random forest, and a gradient boosting model on the same classification dataset with comparable settings, then compare their performance and discuss why the "winner" might not generalize to other problems.

Setup

Every code block below runs live in your browser using Pyodide. Click Run to execute it. We'll use the load_breast_cancer dataset bundled with scikit-learn: a binary classification problem with 30 numeric features and 569 samples. (Pyodide doesn't have the standalone xgboost package available, so the fourth model below uses scikit-learn's own GradientBoostingClassifier, the same gradient-boosting algorithm you learned in the previous lesson, just a different implementation.)

Task 1: Load the Data and Train All Four Models

Each code block is self-contained, so this one loads the data, splits it, trains all four classifiers, and times each one.

import time
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import accuracy_score, f1_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

def evaluate_model(model, name):
    start = time.time()
    model.fit(X_train, y_train)
    elapsed = time.time() - start
    y_pred = model.predict(X_test)
    return {
        "model": name,
        "accuracy": accuracy_score(y_test, y_pred),
        "f1": f1_score(y_test, y_pred),
        "train_time_s": round(elapsed, 4),
    }

models = [
    (KNeighborsClassifier(n_neighbors=5), "KNN"),
    (DecisionTreeClassifier(max_depth=5, random_state=42), "Decision Tree"),
    (RandomForestClassifier(n_estimators=100, random_state=42), "Random Forest"),
    (GradientBoostingClassifier(n_estimators=100, random_state=42), "Gradient Boosting"),
]

results = [evaluate_model(model, name) for model, name in models]
results_df = pd.DataFrame(results)
print(results_df)

Using stratify=y keeps the class balance roughly the same in both the train and test splits, which matters here since breast cancer datasets are typically a bit imbalanced toward the benign class. KNN with default settings does no real "training" beyond storing the data, so its training time will look near-zero. Most of KNN's cost actually happens at prediction time, when it compares every test point against every training point. Keep that distinction in mind when interpreting the training-time column.

Task 2: Visualize the Comparison

Re-run the setup and plot a bar chart comparing accuracy across the four models.

import time
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import accuracy_score, f1_score

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

def evaluate_model(model, name):
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    return {"model": name, "accuracy": accuracy_score(y_test, y_pred), "f1": f1_score(y_test, y_pred)}

models = [
    (KNeighborsClassifier(n_neighbors=5), "KNN"),
    (DecisionTreeClassifier(max_depth=5, random_state=42), "Decision Tree"),
    (RandomForestClassifier(n_estimators=100, random_state=42), "Random Forest"),
    (GradientBoostingClassifier(n_estimators=100, random_state=42), "Gradient Boosting"),
]

results_df = pd.DataFrame([evaluate_model(m, n) for m, n in models])

plt.bar(results_df["model"], results_df["accuracy"])
plt.ylabel("Test accuracy")
plt.ylim(0.85, 1.0)
plt.title("Accuracy by model")
plt.xticks(rotation=15)
show_plot()

Setting the y-axis to start above zero makes small differences between models visible. On this dataset all four models typically land within a few percentage points of each other, often all above 93-95% accuracy, so an axis starting at 0 would make them look identical when they aren't.

Task 3: Reflection

On load_breast_cancer, the ensemble methods (random forest and gradient boosting) usually edge out the single decision tree and KNN slightly, though the gap is often small because this dataset is relatively clean and well-separated by a handful of features. This is a dataset where even a single shallow tree does reasonably well.

The central point to take away is that this ranking is dataset-dependent: ensembles tend to win when there's more noise or more complex feature interactions for many trees to average over or correct, while KNN tends to struggle as dimensionality grows (the "curse of dimensionality" makes distance less meaningful in high-dimensional spaces) and tends to do better on datasets with fewer, well-scaled features.

A single decision tree's relative competitiveness here is partly because this dataset has a small number of genuinely strong, individually predictive features, so there isn't always a large gain from averaging or boosting over many trees. That gain shows up much more clearly on noisier or larger datasets.

The lesson isn't "the fanciest model always wins," it's that you should benchmark multiple model families on your specific data rather than assuming a more sophisticated algorithm automatically performs better.