Pipelines and ColumnTransformer

What you will learn in this lesson:

• Why calling scaler/encoder methods by hand, step by step, is fragile
• How Pipeline chains preprocessing and a model into one object
• How ColumnTransformer applies different preprocessing to different columns at once
• Why fitting a pipeline once on training data and reusing it prevents an entire class of bugs
• Runnable code building a full preprocessing-plus-model pipeline from scratch

The Problem With Hand-Rolled Preprocessing

Every lab so far that needed scaling has done something like this: create a StandardScaler, call fit_transform on the training data, call transform (not fit_transform) on the test data, then pass the result to a model. That's correct, but it's also fragile.

Every step has to be done in the right order, the scaler object has to be kept around and reused exactly once more on the test set, and if you ever add a second preprocessing step, like one-hot encoding, you now have two objects to keep track of in the right order, on both the train and test data, every time.

This fragility is exactly where data leakage bugs come from: it only takes one accidental fit_transform call on test data, instead of transform, to leak test-set statistics into your "test" evaluation.

Pipeline: Chaining Steps Into One Object

scikit-learn's Pipeline bundles a sequence of steps, each with a name, into a single object that behaves like any other estimator. Call .fit() on the pipeline, and it fits each step in order, feeding each step's output to the next. Call .predict(), and it applies every transformation step using what was already learned during fitting, then hands the result to the final model.

import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("model", LogisticRegression(max_iter=5000)),
])

# One fit call handles scaling AND model training, in the correct order
pipe.fit(X_train, y_train)

# One predict call applies the already-fitted scaler, then the already-fitted model
print("Test accuracy:", pipe.score(X_test, y_test))

Notice there's no separate scaler.transform(X_test) call anywhere. The pipeline remembers that the scaler was fitted on the training data, and automatically applies that same fitted transformation, never a fresh one, to anything passed to predict or score. The opportunity for the fit-on-test-data bug has been removed entirely, not just avoided through careful coding.

ColumnTransformer: Different Preprocessing for Different Columns

Real datasets usually mix numeric columns (which need scaling) and categorical columns (which need encoding). ColumnTransformer lets you specify a different preprocessing pipeline for each group of columns, then combines all the results back into one feature matrix.

import numpy as np
import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

# A small synthetic dataset mixing numeric and categorical columns
df = pd.DataFrame({
    "age": [25, 45, 35, 50, 23, 41, 33, 29, 60, 38],
    "income": [40000, 85000, 60000, 95000, 38000, 72000, 55000, 47000, 110000, 64000],
    "city": ["A", "B", "A", "C", "B", "A", "C", "B", "C", "A"],
})
y = np.array([0, 1, 0, 1, 0, 1, 0, 0, 1, 1])

numeric_features = ["age", "income"]
categorical_features = ["city"]

preprocessor = ColumnTransformer([
    ("num", StandardScaler(), numeric_features),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical_features),
])

full_pipeline = Pipeline([
    ("preprocessing", preprocessor),
    ("model", LogisticRegression()),
])

X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.3, random_state=0)
full_pipeline.fit(X_train, y_train)

print("Test accuracy:", full_pipeline.score(X_test, y_test))
print("Number of features after preprocessing:", full_pipeline.named_steps["preprocessing"].transform(X_train).shape[1])

Notice that age and income go through StandardScaler, while city goes through OneHotEncoder, all inside one ColumnTransformer, and the whole thing is then just one more step inside the outer Pipeline. The df passed to fit and score can be the raw, unprocessed DataFrame every time. The pipeline handles every transformation internally.

Why This Matters Beyond Convenience

Pipelines aren't just less typing. A fitted pipeline is a single object that fully captures every preprocessing decision made during training, in the exact order they were made. That object can be passed directly to cross_val_score or GridSearchCV, and scikit-learn will correctly refit the entire pipeline, preprocessing included, on each fold, rather than accidentally reusing statistics computed on data outside that fold.

This is the single most reliable way to prevent the most common form of data leakage, fitting a scaler or encoder on data the model will later be evaluated against, the next lesson covers leakage in more general forms, but pipelines are the practical tool that prevents most of it automatically.