Practical ML Pitfalls: A Debugging Review

What you will learn in this lesson:

• A consolidated checklist of the pitfalls covered separately throughout this course
• Silent NaN propagation, and why a model that "ran without errors" doesn't mean it ran correctly
• Train/test mismatch, when production data quietly stops looking like training data
• A practical debugging order to work through when a model that worked in the notebook stops working

Why This Lesson Is Just a Review

Every pitfall in this lesson has already been covered somewhere earlier in this course. This lesson doesn't introduce new concepts; it gathers them into one list, in the order you should actually check them when something goes wrong, the way an experienced practitioner would. Treat it as a checklist to return to, not a new set of facts to memorize.

The Consolidated Pitfall Checklist

• Train/test contamination. Did any transformer get fit on data outside the training split? Covered in the data leakage lesson. The fix: use a Pipeline and never call fit or fit_transform on test data.
• Target leakage. Does any feature encode information that wouldn't exist yet at prediction time? Covered in the data leakage lesson. The fix: for any suspiciously predictive feature, ask whether you'd genuinely have it available in production, before the prediction is needed.
• Temporal leakage. Was time-ordered data split randomly instead of chronologically? Covered in both the data leakage and splitting-data lessons. The fix: split by date for anything with a real time dimension.
• Class imbalance ignored. Is accuracy the only metric being checked on an imbalanced target? Covered in the metrics and imbalanced classification lessons. The fix: check precision, recall, and F1, and consider class_weight="balanced".
• Unscaled features feeding a distance- or magnitude-sensitive model. Covered in the KNN, SVM, and regularization lessons. The fix: scale before fitting, inside a pipeline.
• Hyperparameters tuned against the training set, or worse, the test set. Covered in the overfitting and hyperparameter tuning lessons. The fix: tune against cross-validated scores, touch the test set exactly once.

Silent NaNs: The Bug That Doesn't Throw an Error

Some bugs crash loudly. Missing values are often worse, because they frequently don't crash at all. Depending on the library and the model, a NaN slipping through preprocessing can silently turn into a feature that's effectively ignored, can crash a model that doesn't accept missing values (loudly, at least), or, worst of all, can get treated as a literal number by something downstream that wasn't expecting it, quietly corrupting results without ever raising an exception.

import numpy as np
import pandas as pd

df = pd.DataFrame({
    "age": [25, np.nan, 35, 40, np.nan],
    "income": [50000, 60000, np.nan, 80000, 55000],
})

# A naive mean that silently ignores NaNs can hide how much data is missing
print("Mean age (NaN-skipping default):", df["age"].mean())

# Always check missing counts explicitly, every time, before trusting any summary statistic
print("\nMissing value counts:")
print(df.isna().sum())
print("\nPercent missing:")
print((df.isna().sum() / len(df) * 100).round(1))

The fix is procedural, not technical: always run .isna().sum() explicitly as a first step, the way the EDA lesson's Stage 1 already recommends, rather than trusting that a summary statistic like .mean() would have told you if something were wrong. It won't. It will just quietly compute over whatever non-missing values happen to remain.

Train/Test Mismatch: When Production Stops Looking Like Training Data

A model evaluated honestly on a held-out test set can still fail in production if the production data's distribution drifts away from what the model was trained and tested on. A fraud model trained on last year's transaction patterns may degrade as fraud tactics change. A pricing model trained before a market shift may keep predicting confidently while being systematically wrong.

This isn't something a single train/test split can catch, since both the training and test data come from the same snapshot in time. It's the reason production ML systems monitor live performance continuously rather than trusting a one-time evaluation forever. At the fundamentals level covered in this course, the actionable takeaway is awareness: a good test-set score is a statement about the past, not a permanent guarantee.

A Practical Debugging Order

When a model performs worse than expected, or worse than it did last time, work through these checks in order, cheapest and most common first:

1. Check for missing values and unexpected data types, the same Stage 1 EDA check, rerun.
2. Check whether any preprocessing step was fit on the wrong data (test set, or the full dataset before splitting).
3. Check the train/test class balance, especially after any change to how the data was filtered or collected.
4. Check for any suspiciously perfect-looking feature, the target leakage red flag.
5. Check whether the random seed changed, accidentally turning a previously stable comparison into a noisy one.
6. Only after all of the above: consider that the model or hyperparameters genuinely need to change.

Most real debugging sessions end somewhere in steps 1 through 5. Reaching for a different algorithm or a bigger hyperparameter search, step 6, before ruling out the cheaper, more common causes is one of the most frequent ways beginners waste time on a problem that was never about the model in the first place.