Model Interpretability and Feature Importance

What you will learn in this lesson:

• Why interpretability is a practical job skill, not an academic afterthought
• Built-in feature importance, revisited, and its main limitation
• Permutation importance, a model-agnostic alternative that works for any model
• Partial dependence, for seeing a feature's effect directly rather than just its importance ranking
• Runnable code computing and comparing all three on the same model

Why Interpretability Is a Practical Skill

A model that performs well but can't be explained is often not deployable at all. A loan-approval model needs to justify its decisions to regulators. A medical screening model needs to tell a doctor which symptoms drove a prediction, not just produce a number. Even when explanation isn't a legal requirement, being able to say "the model relies heavily on these three features" is what lets you sanity-check a model against domain knowledge before trusting it with real decisions.

Built-In Feature Importance, Revisited

The random forest lesson introduced feature_importances_, computed from how much each feature's splits reduced impurity across the ensemble. This is fast and built directly into tree-based models, but it has a real limitation worth naming explicitly: it can be misleading when features are highly correlated with each other, the same multicollinearity problem from the regression lessons, since the "credit" for a good split can get arbitrarily divided between two redundant features, making both look less important than the underlying signal actually is.

It's also only available for tree-based models. Linear models expose coefficients instead (which carry their own scaling caveats from the multiple regression lesson), and some models, like KNN, don't have anything built-in to inspect at all.

Permutation Importance: A Model-Agnostic Alternative

Permutation importance works around both limitations. The idea: take a trained model and a validation set, shuffle one feature's column randomly, breaking its relationship with the target while keeping every other feature intact, and measure how much the model's performance drops. A feature the model relies on heavily will cause a big performance drop when scrambled. A feature the model barely uses will barely move the score at all.

This works for any model, not just trees, since it only needs the ability to call .predict() or .score() repeatedly. It's more computationally expensive than built-in importance (the model has to be re-evaluated once per feature, sometimes repeated multiple times for stability), but it directly measures what you actually care about: how much does the model's real performance depend on this feature.

import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.inspection import permutation_importance
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
feature_names = load_breast_cancer().feature_names
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

model = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)

# Built-in importance
builtin_top5 = np.argsort(model.feature_importances_)[::-1][:5]
print("Top 5 by built-in feature_importances_:")
for i in builtin_top5:
    print(f"  {feature_names[i]}: {model.feature_importances_[i]:.4f}")

# Permutation importance on the held-out test set
perm = permutation_importance(model, X_test, y_test, n_repeats=10, random_state=0)
perm_top5 = np.argsort(perm.importances_mean)[::-1][:5]
print("\nTop 5 by permutation importance:")
for i in perm_top5:
    print(f"  {feature_names[i]}: {perm.importances_mean[i]:.4f}")

Run this and compare the two top-5 lists. They usually overlap substantially but rarely match exactly, since the two methods are measuring genuinely different things: one measures how much a feature contributed to reducing impurity during training, the other measures how much real predictive performance depends on it.

Partial Dependence: Seeing the Effect, Not Just the Ranking

Importance scores rank features but don't show how a feature affects predictions. Partial dependence fills that gap: hold every other feature fixed, vary one feature across its range, and plot how the model's average predicted probability (or value, for regression) changes. This shows the shape of the relationship the model has learned, whether it's roughly linear, has a clear threshold, or is non-monotonic.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
feature_names = list(load_breast_cancer().feature_names)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

model = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_train, y_train)
most_important = np.argmax(model.feature_importances_)

# Compute partial dependence manually: for each candidate value of the
# chosen feature, set every row in the training set to that value, predict,
# and average. This is exactly what partial dependence means.
grid = np.linspace(X_train[:, most_important].min(), X_train[:, most_important].max(), 30)
avg_predictions = []
X_modified = X_train.copy()
for value in grid:
    X_modified[:, most_important] = value
    avg_predictions.append(model.predict_proba(X_modified)[:, 1].mean())

plt.plot(grid, avg_predictions)
plt.xlabel(feature_names[most_important])
plt.ylabel("Average predicted probability")
plt.title(f"Partial dependence: {feature_names[most_important]}")
show_plot()

The resulting curve shows the model's average predicted probability of malignancy as that single feature varies, with everything else held at its observed values. A steadily rising or falling curve confirms the intuitive direction of the relationship; a flat curve over part of the range shows where the feature stops mattering; a non-monotonic curve would reveal a relationship more complex than "more is always worse" or "more is always better."