Machine Learning Fundamentals: Notation & Terms Glossary

Core Concepts

x

An input: a single example's features.

y

The true label or target value for an example.

f / ŷ ("y-hat")

The model's prediction.

H

The hypothesis space: the family of functions an algorithm is allowed to search through. Training means finding the f ∈ H that makes f(x) as close to y as possible.

Loss function

A number measuring how wrong a model's predictions are. Training minimizes this number.

Bias

Error from a model being too simple to capture the true pattern. Systematic and repeatable, regardless of which training sample it saw.

Variance

Error from a model being too sensitive to the specific training sample it happened to see. Retrain on different data, get a noticeably different model.

Linear Models

m, b

Slope and intercept of a simple linear regression line: y = mx + b.

b0, b1, ..., bn (or β0, β1, ..., βn)

Intercept and per-feature coefficients in multiple linear regression: y = b0 + b1·x1 + ... + bn·xn.

Residual

The gap between an actual value and a model's predicted value: actual − predicted.

α (alpha)

The regularization strength in ridge, lasso, and Elastic Net. Larger α means a stronger penalty on coefficient size.

L2 penalty (ridge)

Penalizes the sum of squared coefficients. Shrinks every coefficient toward zero smoothly, but rarely to exactly zero.

L1 penalty (lasso)

Penalizes the sum of absolute coefficient values. Can shrink coefficients to exactly zero, which performs feature selection.

l1_ratio

In Elastic Net, the mix between L1 and L2 penalty, from 0 (pure ridge) to 1 (pure lasso).

Multicollinearity

When two or more features are highly correlated, making individual coefficients unstable to interpret even though overall predictions stay accurate.

Evaluation Metrics

MAE

Mean Absolute Error: the average of |actual − predicted|. Weighs every error linearly.

MSE / RMSE

Mean Squared Error and its square root. Squares errors before averaging, so large errors count disproportionately more.

R²

Coefficient of determination: 1 − (Sum of Squared Residuals / Total Sum of Squares). The fraction of variance in the target explained by the model, relative to always guessing the mean.

Adjusted R²

R² penalized for the number of features used, so it doesn't automatically rise when you add useless features.

TP, FP, TN, FN

True/false positive, true/false negative: the four cells of a confusion matrix.

Precision

TP / (TP + FP). Of everything flagged positive, how much actually was.

Recall

TP / (TP + FN). Of everything actually positive, how much was caught.

F1 score

The harmonic mean of precision and recall: 2 · (Precision · Recall) / (Precision + Recall). High only when both are reasonably high.

Trees & Ensembles

Gini impurity

1 − Σ pᵢ². Roughly the probability that two randomly chosen points from a group have different labels. 0 means perfectly pure.

Entropy

−Σ pᵢ log₂(pᵢ). An information-theoretic alternative to Gini impurity, used to compute information gain.

Bagging

Bootstrap aggregating: training many models on random subsamples of the data (drawn with replacement) and averaging their predictions. The basis of random forests.

ρ (rho)

The correlation between trees in an ensemble. The variance of an average of n trees is ρσ² + (1−ρ)σ²/n, so ρσ² is a variance floor more trees alone can't beat.

Boosting

Training models sequentially, where each new model corrects the residual errors of the ensemble built so far. The basis of gradient boosting and XGBoost.

n_estimators, learning_rate, max_depth

The three core gradient-boosting hyperparameters: number of trees, how much each tree's correction counts, and how deep each tree is allowed to grow.

Distance-Based & Margin-Based Methods

K (KNN)

The number of nearest neighbors consulted when classifying a new point. Small K means high variance; large K means high bias.

Curse of dimensionality

As feature count grows, distances between points become less meaningful, weakening any purely distance-based method.

Margin (SVM)

The distance between a decision boundary and the closest training points on either side. SVM maximizes this.

Support vectors

The training points closest to the decision boundary. They're the only points that determine where the boundary sits.

C (SVM)

Controls how heavily margin violations are penalized. Large C means a narrower, stricter margin; small C means a wider, more tolerant one.

Kernel trick

Computing what a dot product would be in a higher-dimensional space, without explicitly building that space, letting SVM handle non-linear boundaries.

Bayes' theorem

P(class | features) = P(features | class) · P(class) / P(features). Lets you flip a conditional probability around.

Conditional independence (Naive Bayes)

The assumption that every feature is independent of every other feature, given the class. Rarely exactly true, but often harmless for ranking classes.

Unsupervised Learning

K (K-means)

The number of clusters K-means is told to find. Chosen up front, often via the elbow method.

Centroid

The mean position of all points currently assigned to a cluster in K-means.

Inertia

The total squared distance from every point to its assigned centroid. Always decreases as K increases; the elbow method looks for where it stops decreasing fast.

Principal component

A new axis found by PCA, aligned with a direction of variance in the data. Ordered from most to least variance captured.

Explained variance ratio

The fraction of a dataset's total variance a single principal component captures on its own.

Model Quality

Overfitting

Training error keeps falling while validation error rises. The model has started fitting noise instead of signal.

Underfitting

Both training and validation error are high and close together. The model is too constrained to capture the true pattern.

Train / validation / test split

Three distinct data subsets: one to fit parameters, one to make decisions about the model, and one held back untouched for a single final honest check.

K-fold cross-validation

Rotating which portion of the data is used for validation across K rounds, so every sample contributes to both training and validation without ever doing both at once.