Loss Functions, Gradient Descent, and Optimization Basics

What you will learn in this lesson:

• What a loss function is, in general, not just for one specific algorithm
• The gradient descent intuition: rolling downhill on a loss surface
• Why the learning rate matters, and what happens when it's too large or too small
• How this same idea reappears later in logistic regression, gradient boosting, and beyond
• A runnable visualization of gradient descent converging on a simple loss surface

What a Loss Function Actually Is

Every model you'll train in this course is, underneath the specific algorithm, answering the same question: "out of every possible setting of my parameters, which one makes my predictions least wrong?" A loss function is the precise, numeric definition of "wrong" the algorithm is told to minimize.

You've already met two: mean squared error for regression (the lower the squared distance between predictions and actual values, the better), and something like cross-entropy for classification (the lower the disagreement between predicted probabilities and actual labels, the better). Different problems call for different loss functions, but every one of them takes the same shape: a single number that gets smaller as the model gets better, and the entire training process is a search for the parameters that make that number as small as possible.

Gradient Descent: Rolling Downhill

For some loss functions, like ordinary linear regression's squared error, there's a closed-form formula that jumps straight to the answer, no searching required, which is what the linear regression lesson used. Most loss functions in machine learning don't have that luxury. Logistic regression, neural networks, and many other models need to search for the minimum iteratively, and gradient descent is the most common way to do that search.

The intuition: picture the loss function as a landscape, with height representing how wrong the model currently is. Lower is better. Gradient descent starts at some random point on this landscape and repeatedly asks "which direction is downhill from here," then takes a small step in that direction. The gradient is exactly that: a mathematical description of which direction is steepest uphill at the current point. Stepping in the opposite direction is, by definition, the fastest way to go downhill from where you're standing.

Repeat this process, recompute the gradient at the new point, take another step, and the model's parameters gradually move toward whatever setting minimizes the loss. You already implemented exactly this process by hand in the "Build Linear Regression from Scratch" lab, for the specific case of squared error. This lesson is naming the general version of that same idea, since it reappears throughout the rest of this course.

The Learning Rate: How Big a Step to Take

The size of each step is controlled by the learning rate, the same parameter you set as lr in the from-scratch lab. Too small a learning rate, and training crawls forward, needing many more iterations than necessary to reach the minimum. Too large a learning rate, and each step can overshoot the minimum entirely, sometimes bouncing back and forth without ever settling down, or even diverging to increasingly large, useless values.

There is no universal correct learning rate. It depends on the shape of the specific loss surface, which itself depends on the data and the model. In practice, you try a small set of candidate learning rates (often spaced on a logarithmic scale, like 0.001, 0.01, 0.1, 1.0) and watch which one's loss curve decreases steadily without oscillating wildly, the same diagnostic skill you'll formalize further in the hyperparameter tuning lesson later in this course.

Seeing Convergence Happen

import numpy as np
import matplotlib.pyplot as plt

# A simple bowl-shaped loss function: loss(w) = (w - 3)^2 + 2
# Its minimum is at w = 3. The gradient (derivative) is 2*(w - 3).
def loss(w):
    return (w - 3) ** 2 + 2

def gradient(w):
    return 2 * (w - 3)

def run_gradient_descent(learning_rate, n_steps=30, start=-5.0):
    w = start
    history = [w]
    for _ in range(n_steps):
        w = w - learning_rate * gradient(w)
        history.append(w)
    return history

fig, axes = plt.subplots(1, 3, figsize=(13, 4))
for ax, lr in zip(axes, [0.05, 0.5, 1.05]):
    history = run_gradient_descent(lr)
    ax.plot(history, marker="o", markersize=3)
    ax.axhline(3, color="gray", linestyle="--", linewidth=1, label="true minimum (w=3)")
    ax.set_title(f"learning_rate = {lr}")
    ax.set_xlabel("step")
    ax.set_ylabel("w")
    ax.legend(fontsize=8)

plt.tight_layout()
show_plot()

Run this and compare the three panels. lr=0.05 converges smoothly but slowly. lr=0.5 converges quickly and smoothly, close to ideal for this particular bowl shape. lr=1.05, just past the stable range for this specific loss surface, oscillates and grows rather than settling down, a small worked illustration of exactly the overshooting problem described above.

Where You'll See This Again

Gradient descent (or close variants of it) is what fits logistic regression's coefficients when there's no closed-form shortcut, what trains the individual weak learners inside gradient boosting (the name is not a coincidence), and what would train a neural network if this course covered them. The vocabulary from this lesson, loss function, gradient, learning rate, convergence, is general-purpose across nearly every model you will ever train.