Objective: Train Logistic Regression, KNN, a Decision Tree, and a Random Forest on the same imbalanced classification dataset, compare them on precision, recall, and F1 rather than accuracy, and write a short justified recommendation for which one to ship.
Setup
Every code block below runs live in your browser using Pyodide. Click Run to execute it. The dataset is synthetic and imbalanced on purpose, about 10% positive class, simulating something like fraud detection or rare-disease screening, the exact scenario the metrics and imbalanced-classification lessons warned accuracy would mislead you on.
Task 1: Build the Dataset and Establish a Naive Baseline
Before comparing real models, compute what a trivial "always predict the majority class" baseline would score on accuracy. This is the number every real model needs to meaningfully beat, not just on accuracy, but on recall and F1.
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, f1_score, recall_score
np.random.seed(0)
n = 1200
X = np.random.randn(n, 6)
weights = np.array([1.4, -1.0, 0.8, 0.0, 0.0, 0.5])
score = X @ weights + np.random.randn(n) * 1.6
y = (score > np.quantile(score, 0.90)).astype(int) # ~10% positive
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=0, stratify=y)
print("Positive rate in test set:", round(y_test.mean(), 3))
naive_pred = np.zeros_like(y_test) # always predict the majority class
print("Naive 'always negative' baseline:")
print(" Accuracy:", round(accuracy_score(y_test, naive_pred), 3))
print(" Recall: ", round(recall_score(y_test, naive_pred, zero_division=0), 3))
print(" F1: ", round(f1_score(y_test, naive_pred, zero_division=0), 3))
The naive baseline's accuracy will look deceptively high, while its recall and F1 collapse to zero, since it never predicts the positive class at all. Any real model needs to beat zero recall to be worth anything, regardless of how its accuracy compares.
Task 2: Train Four Classifiers
Train Logistic Regression, KNN, a Decision Tree, and a Random Forest, each with class_weight="balanced"
where the model supports it (KNN has no such parameter, since it has no loss function to reweight), and compare
all four on the same metrics.
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score
np.random.seed(0)
n = 1200
X = np.random.randn(n, 6)
weights = np.array([1.4, -1.0, 0.8, 0.0, 0.0, 0.5])
score = X @ weights + np.random.randn(n) * 1.6
y = (score > np.quantile(score, 0.90)).astype(int)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=0, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
models = {
"Logistic Regression": LogisticRegression(class_weight="balanced", max_iter=5000),
"KNN": KNeighborsClassifier(n_neighbors=7),
"Decision Tree": DecisionTreeClassifier(class_weight="balanced", max_depth=5, random_state=0),
"Random Forest": RandomForestClassifier(class_weight="balanced", n_estimators=200, random_state=0),
}
results = []
for name, model in models.items():
model.fit(X_train_scaled, y_train)
pred = model.predict(X_test_scaled)
results.append({
"model": name,
"accuracy": round(accuracy_score(y_test, pred), 3),
"precision": round(precision_score(y_test, pred, zero_division=0), 3),
"recall": round(recall_score(y_test, pred, zero_division=0), 3),
"f1": round(f1_score(y_test, pred, zero_division=0), 3),
})
results_df = pd.DataFrame(results)
print(results_df)
Task 3: Compare With and Without Imbalance Handling
Re-run the Random Forest without class_weight="balanced" and compare its recall and F1 to the
weighted version above, isolating exactly how much that one parameter changes.
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import recall_score, f1_score
np.random.seed(0)
n = 1200
X = np.random.randn(n, 6)
weights = np.array([1.4, -1.0, 0.8, 0.0, 0.0, 0.5])
score = X @ weights + np.random.randn(n) * 1.6
y = (score > np.quantile(score, 0.90)).astype(int)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0, stratify=y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
rf_default = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train_scaled, y_train)
rf_balanced = RandomForestClassifier(class_weight="balanced", n_estimators=200, random_state=0).fit(X_train_scaled, y_train)
pred_default = rf_default.predict(X_test_scaled)
pred_balanced = rf_balanced.predict(X_test_scaled)
print("Default Random Forest -> recall: {:.3f}, F1: {:.3f}".format(
recall_score(y_test, pred_default), f1_score(y_test, pred_default)))
print("Balanced Random Forest -> recall: {:.3f}, F1: {:.3f}".format(
recall_score(y_test, pred_balanced), f1_score(y_test, pred_balanced)))
Task 4: Write a Justified Recommendation
Based on your Task 2 results table, write 3-4 sentences (as a comment in a code cell, or just as your own notes) recommending one model to ship. A strong recommendation does more than name the highest F1 score.
It states what the use case implicitly cares about (recall, if missing a positive case is costly, like the medical screening and fraud scenarios from the metrics lessons, or precision, if false alarms are the bigger cost) and picks accordingly, even if that means not picking the single highest-F1 model.
There's no single correct answer graded here. The skill being practiced is connecting a metrics table back to a real decision, the same judgment call every one of the earlier lessons on precision, recall, and imbalance has been building toward.