Support Vector Machines

What you will learn in this lesson:

• The maximum-margin idea: why SVM picks the boundary with the widest gap, not just any separating line
• What support vectors are, and why most training points don't actually matter to the final boundary
• The soft-margin parameter C and the tradeoff it controls
• The kernel trick, and how it lets a linear method handle non-linear boundaries
• How to fit an SVM classifier in Python and inspect its support vectors

Choosing the Widest Possible Boundary

If two classes are linearly separable, there are infinitely many straight lines (or, in higher dimensions, flat hyperplanes) that separate them perfectly on the training data. Logistic regression picks one such line based on maximizing likelihood. A Support Vector Machine (SVM) picks the one specific line that maximizes the margin: the distance between the boundary and the closest training point on either side of it.

The intuition for why this matters: a boundary that barely squeezes between the two classes, with almost no breathing room, is fragile. A new point that's only slightly different from a training point could easily land on the wrong side. A boundary with a wide margin has more room for new points to vary before crossing to the wrong side, which is why maximizing the margin tends to generalize better.

A Worked Margin Calculation

The distance from a point x to a hyperplane w · x + b = 0 is |w · x + b| / ||w||. Take a tiny 2-feature example with boundary w = (1, 1), b = -4 (the line x1 + x2 = 4), and two candidate support vectors, A = (1, 1) from the negative class and B = (3, 3) from the positive class:

Distance of A = |1(1) + 1(1) − 4| / √(1² + 1²) = |−2| / 1.414 ≈ 1.414

Distance of B = |1(3) + 1(3) − 4| / √(1² + 1²) = |2| / 1.414 ≈ 1.414

Both points sit exactly 1.414 units from the boundary, on opposite sides, which is exactly what makes them support vectors for this boundary: the margin is only as wide as its closest points allow, and a maximum-margin boundary places itself so the nearest point from each class is equally close. The total margin width here is the sum of both distances, about 2.83.

Support Vectors: Why Most Points Don't Matter

The points that sit exactly on the edge of the margin, the closest points from each class, are called support vectors. They are the only training points that actually determine where the boundary and margin sit. Every other point, anything further from the boundary than the margin, could be moved or removed entirely without changing the fitted boundary at all.

This is a meaningfully different behavior from algorithms like logistic regression or linear regression, where every training point contributes to the fitted coefficients. An SVM's decision boundary depends only on a (usually small) subset of "hard" points sitting closest to the other class. This is also why SVMs can be memory efficient at prediction time: once trained, you only need to keep the support vectors, not the full training set.

The Soft Margin and the C Parameter

Real data is rarely perfectly separable. A handful of mislabeled or overlapping points usually make a perfect separating boundary impossible, or force a boundary with an unreasonably narrow margin just to accommodate one stubborn point. The soft margin SVM allows some points to violate the margin, or even end up on the wrong side, in exchange for a wider, more robust margin overall.

The hyperparameter C controls this tradeoff directly. A large C penalizes margin violations heavily, pushing the model toward a narrower margin that tries hard to classify every training point correctly, which risks overfitting to noise. A small C tolerates more violations in exchange for a wider, smoother margin, which risks underfitting if set too low. This is the same bias-variance tradeoff you've seen throughout this course, expressed through a different knob.

The Kernel Trick: Handling Non-Linear Boundaries

Many real datasets aren't linearly separable at all, no straight line or flat hyperplane can separate the classes in the original feature space. One classic fix is to transform the data into a higher-dimensional space where it becomes linearly separable, fit a linear boundary there, and map it back.

Computing that transformation explicitly for every point can be expensive, sometimes the higher-dimensional space has far more dimensions than the original data. The kernel trick avoids this entirely. SVM's optimization only ever needs the dot product between pairs of points, never the points' raw coordinates in the transformed space. A kernel function computes what that dot product would be in the higher-dimensional space directly from the original points, without ever constructing the transformed coordinates at all.

The most common non-linear kernel is the RBF (radial basis function) kernel, which effectively measures similarity based on distance and can carve out smooth, curved decision boundaries. A linear kernel (no transformation at all) is the right starting point when you have reason to believe the classes are roughly linearly separable, or when you have many more features than samples, where linear boundaries already have plenty of flexibility.

Fitting an SVM in Python

import numpy as np
from sklearn.svm import SVC
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = SVC(kernel="rbf", C=1.0, gamma="scale")
model.fit(X_train_scaled, y_train)

print("Test accuracy:", model.score(X_test_scaled, y_test))
print("Number of support vectors per class:", model.n_support_)
print("Total support vectors:", sum(model.n_support_), "out of", len(X_train_scaled), "training points")

Feature scaling matters a great deal here, for the same distance-sensitivity reason it mattered for KNN: SVM's margin is a geometric distance, so a feature on a much larger numeric scale would dominate that distance calculation. Try the code above with kernel="linear" instead of "rbf" and compare both the accuracy and the number of support vectors used.