Objective: Practice computing RMSE, MAE, and R² for a regression task and accuracy, precision, recall, F1, and a confusion matrix for a classification task, then articulate what each metric reveals that the others don't.
Setup
Every code block below runs live in your browser using Pyodide. Click Run to execute it, no
installation required. We'll use two of scikit-learn's bundled datasets: load_diabetes for the
regression half of the lab, and load_breast_cancer for the classification half.
Task 1: Regression Metrics on the Diabetes Dataset
Fit a plain LinearRegression model and compute RMSE, MAE, and R² on the held-out test set.
import numpy as np
from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
X, y = load_diabetes(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
reg = LinearRegression()
reg.fit(X_train, y_train)
y_pred = reg.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)
print(f"RMSE: {rmse:.2f}")
print(f"MAE: {mae:.2f}")
print(f"R2: {r2:.3f}")
You should see an R² somewhere in the 0.4-0.5 range on this dataset with a plain linear model. This dataset is noisy, and 0.45 is a perfectly reasonable result, not a failure.
Task 2: Classification Metrics on the Breast Cancer Dataset
The workflow is structurally similar, but the metrics differ because the target is categorical rather than continuous.
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score, confusion_matrix
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
clf = LogisticRegression(max_iter=5000)
clf.fit(X_train_scaled, y_train)
y_pred = clf.predict(X_test_scaled)
print(f"Accuracy: {accuracy_score(y_test, y_pred):.3f}")
print(f"Precision: {precision_score(y_test, y_pred):.3f}")
print(f"Recall: {recall_score(y_test, y_pred):.3f}")
print(f"F1: {f1_score(y_test, y_pred):.3f}")
print("Confusion matrix:")
print(confusion_matrix(y_test, y_pred))
Scaling features first (with StandardScaler) helps logistic regression's solver converge cleanly,
since the breast cancer dataset's features live on very different scales. The confusion matrix prints as a 2x2 grid;
rows are true classes, columns are predicted classes, so the off-diagonal entries are your false positives and
false negatives. Look at which off-diagonal cell is larger to see whether your model leans toward
over-predicting or under-predicting the positive class.
Task 3: Reflection, What Each Metric Tells You
No single metric tells the whole story. RMSE and MAE both summarize prediction error in regression, but RMSE penalizes large errors more heavily (because it squares them before averaging), so a model with one huge outlier error will look worse under RMSE than under MAE even if most predictions are accurate.
R², by contrast, doesn't tell you about error magnitude at all. It tells you what fraction of the target's variance your model explains relative to a naive "always predict the mean" baseline, which makes it useful for judging whether your model is doing meaningfully better than nothing.
On the classification side, accuracy can be dangerously misleading on imbalanced datasets, since a model that always predicts the majority class can still score high accuracy. Precision answers "of the cases I flagged as positive, how many actually were?" while recall answers "of all the actual positives, how many did I catch?" These two often trade off against each other, and F1 balances them into one number when you don't want to favor one over the other.
In a medical screening context like this dataset, you'd typically prioritize recall (catching every malignant case matters more than avoiding a false alarm), whereas in a spam filter you might prioritize precision (you don't want to bury real email in the spam folder). The right metric always depends on the cost of different kinds of mistakes in your specific application, not on some universal "best" metric.