Data Leakage and Reproducibility

What you will learn in this lesson:

• Train/test contamination: the leakage you already know how to prevent, named explicitly
• Target leakage: when a feature secretly encodes the answer
• Temporal leakage: when future information sneaks into a model that's only supposed to know the past
• Why a leaked model looks great in development and fails in production, every time
• The reproducibility habits that make your results trustworthy and repeatable

Why Leakage Deserves Its Own Lesson

You've already practiced the fix for one kind of leakage: fitting a scaler only on the training split, never on the full dataset. That single habit, formalized by pipelines in the previous lesson, prevents the most common leakage bug. But "leakage" is a broader category than just preprocessing order, and it's worth naming every common form explicitly, because each one is easy to miss without knowing what to look for.

The common thread across every type of leakage: information that wouldn't actually be available at prediction time, in the real world, sneaks into training anyway. The model looks great on every metric you check, then performs far worse, or fails outright, once it meets genuinely new data.

Train/Test Contamination

This is the leakage type you've already met: fitting any transformer, a scaler, an encoder, an imputer, on data that includes the test set, even partially. The fix is the one you already know: fit every transformer only on the training data, and only ever call .transform(), never .fit() or .fit_transform(), on the test set. Using a Pipeline consistently is the most reliable way to enforce this, since it makes the correct order the default behavior rather than something you have to remember every time.

Target Leakage

Target leakage happens when a feature is, directly or indirectly, derived from the target itself, or from information that's only available after the target is already known. The classic example: predicting whether a customer will churn, using a feature called "days since last support ticket closure before cancellation." That column only has a meaningful value for customers who already cancelled, which means it's encoding the answer, not predicting it.

Target leakage is dangerous specifically because it produces a model with suspiciously excellent performance. If a single feature gives you 99% accuracy and nothing else comes close, that's exactly the moment to go back to the EDA lesson's target-relationship stage and ask "would I actually have this feature's value before I needed the prediction, in the real world?" If the answer is no, that feature has to go.

Temporal Leakage

Temporal leakage happens when a model trained on time-ordered data gets to see information from the future relative to what it's predicting. A plain random train/test split, the kind you've used in every lab so far, can accidentally do this with time-series-flavored data: a random split might put a January row in training and a December row from the same customer in test, but also a December row in training and a January row in test, letting the model implicitly learn from the future to predict the past.

The fix is to split chronologically when data has a time dimension: everything before a cutoff date goes into training, everything after goes into test, never randomly interleaved. This is a genuine exception to the random train/test splitting you've practiced elsewhere in this course, worth flagging explicitly so you recognize when the default random split is the wrong tool.

A Concrete Leakage Walkthrough

import numpy as np
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

# WRONG: fitting the scaler on the full dataset before splitting leaks
# test-set mean/variance into the "training" transformation
scaler_leaky = StandardScaler().fit(X)  # fit on ALL data, including test rows
X_train_leaky = scaler_leaky.transform(X_train)
X_test_leaky = scaler_leaky.transform(X_test)

model_leaky = LogisticRegression(max_iter=5000).fit(X_train_leaky, y_train)
print("Leaky setup test accuracy:  ", model_leaky.score(X_test_leaky, y_test))

# CORRECT: fit the scaler only on the training split
scaler_clean = StandardScaler().fit(X_train)
X_train_clean = scaler_clean.transform(X_train)
X_test_clean = scaler_clean.transform(X_test)

model_clean = LogisticRegression(max_iter=5000).fit(X_train_clean, y_train)
print("Clean setup test accuracy:  ", model_clean.score(X_test_clean, y_test))

On a dataset this size, the difference between the two accuracy numbers is often small, which is itself an important lesson: leakage doesn't always announce itself with a dramatic accuracy gap. Sometimes it inflates your estimate by a fraction of a percent, just enough to make a deployment decision you'll regret. The discipline of avoiding leakage matters even when, on a given dataset, the visible damage looks small.

Reproducibility: Making Your Results Trustworthy

Every code example in this course sets random_state or np.random.seed() explicitly. That's not a style preference, it's what makes a result reproducible: rerunning the exact same code should produce the exact same number, so that you, or a teammate, or your future self, can verify a result rather than just trust it.

A practical reproducibility checklist worth adopting as a habit: set a random seed once, at the top of a notebook or script, and reuse it everywhere data gets shuffled or a model gets initialized. Save the exact train/test split you evaluated against, rather than re-splitting randomly every time you rerun a notebook.

Pin library versions, since scikit-learn's defaults occasionally change between versions in ways that shift results slightly. Never edit a dataset file by hand outside of code that's checked into version control, since a manual edit is invisible to anyone trying to reproduce your work later.