Feature Engineering Basics

What you will learn in this lesson:

• Why feature engineering often moves performance more than swapping algorithms does, with a concrete example
• How one-hot encoding works, traced through the exact resulting columns
• The difference between standardization and min-max scaling, and which models actually care
• How to build interaction features that expose signal no single raw column carries on its own
• Practical, concrete strategies for handling missing values

Why Features Often Matter More Than the Algorithm

It's tempting to assume the path to a better model is always a fancier algorithm: swap linear regression for a random forest, swap the random forest for gradient boosting. In practice, the single biggest lever for improving performance is frequently not the algorithm at all, but the quality of the features fed into it. Feature engineering is the process of transforming raw data into representations that make the underlying pattern easier for any model to learn.

Here is a concrete version of why this matters. Suppose you're predicting loan default, and your raw data has two columns: monthly_debt_payments and monthly_income. Neither column alone tells the real story. Someone with $2,000/month in debt payments is in fine shape if they earn $10,000/month, but in serious trouble if they only earn $2,500/month.

A sophisticated model given only the two raw columns has to somehow discover, purely from data, that it's the ratio between them that matters. Discovering a ratio relationship from raw inputs is a much harder learning problem than being handed the ratio directly.

If you instead engineer a single new column, debt_to_income = monthly_debt_payments / monthly_income, you've handed even a simple model the answer on a plate. In practice, this one engineered feature often improves a logistic regression more than switching that same logistic regression to a random forest would.

No algorithm can learn a relationship that isn't expressed somewhere in its input. That's the core reason feature engineering tends to dominate algorithm choice.

One-Hot Encoding, Worked Through Column by Column

Most ML algorithms expect numeric input, but real data is full of categorical variables: a customer's country, a product's color, a payment method. You cannot just assign arbitrary integers, Red=1, Blue=2, Green=3, because that imposes a false ordering and false distance. It implies Green is "twice as far" from Red as Blue is, which is meaningless for an unordered category, and a model that uses distance or magnitude, like KNN or linear regression, would take that fake numeric relationship literally.

One-hot encoding avoids this by creating one binary (0/1) column per possible category. Take a small table of 4 rows with a color column holding Red, Blue, Green, Red:

One-hot encoding replaces that single column with three:

Exactly one column is 1 in any given row, the rest are 0. No ordering, no false distance, just presence or absence of each category. The tradeoff is column count: a categorical feature with hundreds of distinct values (like "city") explodes into hundreds of mostly-empty columns, which is something to watch for with high-cardinality categories. Strategies like grouping rare categories into "other," or using a learned embedding instead, exist specifically to deal with that case.

import pandas as pd

df = pd.DataFrame({
    "row": [1, 2, 3, 4],
    "color": ["Red", "Blue", "Green", "Red"],
})

print("Before one-hot encoding:")
print(df)

encoded = pd.get_dummies(df, columns=["color"], prefix="color")
print()
print("After one-hot encoding:")
print(encoded)

Scaling and Normalization: Which Models Actually Care, and Why

Features often live on wildly different scales. "Age in years" might range 0 to 100, while "annual income" might range into the hundreds of thousands. Two common rescaling techniques fix this:

• Standardization (StandardScaler): subtract the mean and divide by the standard deviation, so every feature ends up with mean 0 and standard deviation 1. The result isn't bounded to a fixed range.
• Min-max normalization (MinMaxScaler): rescale each feature linearly into a fixed range, typically [0, 1], based on its observed minimum and maximum.

Why does this matter mechanically, not just "as good practice"? KNN classifies a point by finding its nearest neighbors using a distance formula (commonly Euclidean distance), and that formula sums squared differences across all features equally.

If "income" ranges over 100,000 and "age" ranges over 100, the income feature's raw differences will be thousands of times larger in magnitude, so it will completely dominate the distance calculation regardless of whether age is actually more predictive. KNN will, in effect, ignore age. Scaling both features to comparable ranges first ensures the distance calculation reflects actual predictive relevance, not arbitrary units.

Regularized regression (ridge, lasso) is sensitive for a related but distinct reason: the penalty term shrinks coefficients toward zero, but it shrinks them in raw coefficient units. A feature measured in tiny units (like income in millions of dollars) will naturally have a tiny coefficient even if it's highly predictive, while a feature in large units (age in years) will have a larger coefficient for the same predictive power. The regularization penalty, applied uniformly, ends up penalizing those two features unfairly differently.

Standardizing first puts every feature's coefficient on the same footing, so the penalty treats genuine predictive strength consistently rather than being an artifact of measurement units.

Tree-based models (decision trees, random forests, gradient boosting) are a notable exception. They split on thresholds ("is age > 40?"), not distances or magnitudes, so the scale of a feature doesn't change which split point a tree picks. Scaling is harmless for trees but not necessary.

As for which scaler to pick: StandardScaler is usually the safer default and tends to behave better when a feature has outliers (a single extreme value affects the mean and standard deviation, but not catastrophically). MinMaxScaler is more sensitive to outliers, since a single extreme value directly redefines the entire scale, but it's useful when you need values bounded in a fixed range, such as for certain neural network input layers.

Creating Interaction Features

Sometimes the predictive signal isn't in any single feature, but in how two or more features combine. An interaction feature is a new column explicitly built by multiplying, dividing, or otherwise combining existing ones. The debt-to-income ratio from the opening example is exactly this kind of feature.

Polynomial features (squaring a feature, or multiplying two features together systematically) give linear models a controlled way to capture some non-linear relationships and interactions without switching to a fundamentally different algorithm, at the cost of adding more columns. That's part of why regularization is so often paired with engineered feature sets: more columns means more opportunity to overfit, and the penalty keeps that in check.

Handling Missing Values

Real-world data is rarely complete. Two concrete, commonly used strategies:

• Imputation with median (not mean) for skewed or outlier-prone columns. Filling gaps with the column's median is more robust than the mean when a feature has outliers, for the same reason the median resists outliers better than the mean in general: one extreme value can drag a mean far from the "typical" value, but barely moves the median.
• Add a "was missing" indicator column. Alongside the imputed value, add a second binary column flagging which rows originally had a gap. The fact that a value was missing can itself be predictive. A customer who left an income field blank may behave differently from one who filled it in, and a plain imputed value alone would erase that signal entirely.

From Raw to Useful: A Worked Date Example

A raw signup_date string like "2024-03-15" is nearly useless to most algorithms on its own. A few transformations turn it into several genuinely predictive features:

import pandas as pd

df = pd.DataFrame({
    "signup_date": ["2024-03-15", "2023-11-02", "2024-06-20"],
    "today": pd.Timestamp("2024-07-01"),
})
df["signup_date"] = pd.to_datetime(df["signup_date"])

# Raw timestamps are nearly useless to most models; derive informative features instead
df["account_age_days"] = (df["today"] - df["signup_date"]).dt.days
df["signup_month"] = df["signup_date"].dt.month
df["signup_is_weekend"] = df["signup_date"].dt.dayofweek >= 5

print(df[["signup_date", "account_age_days", "signup_month", "signup_is_weekend"]])

"Account age in days" is directly useful for predicting churn or engagement, since newer accounts often behave differently from established ones. "Signup month" can capture seasonal patterns. "Signup is weekend" might correlate with casual versus deliberate signups.

None of this was accessible to a model handed only the raw date string. It had to be deliberately engineered out. That's the whole discipline in one example: doing a slice of the pattern-recognition work yourself, in a way no algorithm could discover purely from unprocessed inputs.

With well-engineered features, properly scaled inputs, and the generalization checks from the previous lesson all in hand, you now have every tool this course covers. The capstone project next ties them all together into one end-to-end pipeline.