Logistic Regression

What you will learn in this lesson:

• Why plain linear regression can't be used directly for classification
• The sigmoid function, and how it turns any real number into a valid probability
• What log-odds are, and how logistic regression is actually linear in that space
• The decision threshold, and what changing it actually does to predictions
• A worked numeric example and runnable code fitting a logistic regression classifier

Why Not Just Use Linear Regression?

Suppose you want to predict whether a tumor is malignant (1) or benign (0) from its size. You could try fitting a plain linear regression line to the 0/1 labels directly. This runs into an immediate problem: a line extends infinitely in both directions, so for a large enough tumor size, the line will predict a value like 1.4, or for a small enough size, −0.3. Neither number means anything as a probability, which has to stay between 0 and 1.

Logistic regression fixes this by not predicting the 0/1 label directly. Instead, it fits a linear combination of the features, exactly like ordinary regression, and then squeezes that unbounded number through a function that guarantees the output always lands between 0 and 1.

The Sigmoid Function

That squeezing function is the sigmoid (or logistic) function:

σ(z) = 1 / (1 + e^−z)

Here z is the linear combination b0 + b1·x1 + ... + bn·xn, the exact same expression multiple linear regression computes. The sigmoid takes whatever real number that expression produces and maps it into the range (0, 1). When z is very large and positive, σ(z) approaches 1. When z is very large and negative, σ(z) approaches 0. When z is exactly 0, σ(z) is exactly 0.5.

This is exactly why logistic regression is well-suited to classification despite the name: the model still learns a linear function of the inputs, the same hypothesis space as ordinary regression, but the sigmoid reinterprets that linear output as a probability rather than a raw prediction.

Log-Odds: Where the Model Is Actually Linear

It helps to see exactly what logistic regression is linear in. The sigmoid function can be inverted: if p is the predicted probability, then:

log( p / (1 − p) ) = b0 + b1·x1 + ... + bn·xn

The quantity p / (1 − p) is called the odds (the probability of the positive class divided by the probability of the negative class), and its logarithm is the log-odds, also called the logit. Logistic regression fits a straight line, the familiar linear combination of features, in log-odds space.

A one-unit increase in a feature adds a fixed amount to the log-odds, which is why each coefficient b1, ..., bn can be interpreted as that feature's effect on the log-odds of the positive class, holding the others constant, the same "holding constant" interpretation you met in the multiple regression lesson.

The Decision Threshold

Logistic regression outputs a probability, not a hard label, so converting it into a yes/no prediction requires choosing a threshold. The default is 0.5: predict the positive class if the predicted probability is at least 0.5, otherwise predict negative.

Nothing forces you to use 0.5. Lowering the threshold makes the model predict positive more often, which increases recall (you catch more true positives) at the cost of precision (you also flag more false positives). Raising it does the opposite. This is exactly the precision-recall tradeoff from the evaluation lessons, and the threshold is the dial that controls where on that tradeoff curve a logistic regression model actually operates.

A Worked Numeric Example

Suppose a fitted model for predicting loan default gives z = -2.5 + 0.04 · (credit_utilization_pct). For a borrower with 80% credit utilization:

z = -2.5 + 0.04 × 80 = -2.5 + 3.2 = 0.7

σ(0.7) = 1 / (1 + e^-0.7) ≈ 1 / (1 + 0.4966) ≈ 0.668

The model predicts roughly a 66.8% probability of default for this borrower. With the default 0.5 threshold, that crosses the line, so the model predicts "will default." A borrower with 40% utilization instead gives z = -2.5 + 0.04 × 40 = -0.9, and σ(-0.9) ≈ 0.289, comfortably below the threshold.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

b0, b1 = -2.5, 0.04

for utilization in [80, 40, 62.5]:
    z = b0 + b1 * utilization
    p = sigmoid(z)
    prediction = "default" if p >= 0.5 else "no default"
    print(f"Utilization {utilization}% -> z={z:.2f}, P(default)={p:.3f} -> predicted: {prediction}")

Fitting Logistic Regression in Python

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_breast_cancer

X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

model = LogisticRegression(max_iter=5000)
model.fit(X_train_scaled, y_train)

print("Test accuracy:", model.score(X_test_scaled, y_test))

# Predicted probabilities, not just hard labels, for the first 5 test samples
probs = model.predict_proba(X_test_scaled[:5])
print("\nPredicted probabilities [class 0, class 1] for first 5 samples:")
print(np.round(probs, 3))

As with any distance- or magnitude-sensitive optimization, scaling features first helps the solver converge reliably, the same reasoning from the feature engineering lesson. predict_proba exposes the actual probabilities the sigmoid produces, while predict just applies the default 0.5 threshold on top of them.