Backpropagation, Explained

What you will learn in this lesson:

• Why backpropagation is the chain rule, nothing more exotic than that
• Why gradients must be computed backward, starting at the loss, rather than forward from the input
• How the gradient signal at one layer depends on the gradient signal at the layer after it
• What it means, practically, for a gradient to be exactly zero

The Question Backpropagation Answers

Training a network means adjusting every weight to reduce the loss. Gradient descent tells you how: w ← w − η·∂L/∂w for every weight w in the network, where L is the loss and η is the learning rate. The entire difficulty is computing ∂L/∂w: how much does the loss change if you nudge this one particular weight, buried inside layer 3 of a 10-layer network, by a tiny amount? Backpropagation is the algorithm that answers this question, for every weight in the network, in a single efficient backward pass.

It's the Chain Rule, Layer by Layer

Recall from the forward propagation lesson that a network is a composition of functions: ŷ = f3(f2(f1(x))). The loss L is itself a function of ŷ, so the whole thing is L(f3(f2(f1(x)))). The chain rule from calculus tells you exactly how to differentiate a composition like this:

Read right to left: how layer 1's output changes with respect to W1, times how layer 2's output changes with respect to layer 1's output, times how layer 3's output changes with respect to layer 2's output, times how the loss changes with respect to layer 3's output. Multiply all of those local derivatives together and you get the gradient of the loss with respect to a weight several layers removed from the output. Nothing here is more advanced than the chain rule you'd see in a first calculus course; what makes it feel intricate is simply that there are many layers to chain through.

Why Backward, Not Forward

Notice the chain rule expression above shares a lot of structure across different weights. The gradient with respect to W2 reuses ∂L/∂a3 and ∂a3/∂a2, exactly the same terms needed for W1's gradient. This is the key efficiency insight: instead of recomputing the whole chain from scratch for every single weight, compute the shared terms once, starting at the loss, and reuse them as you move backward.

This is also why the computation has to start at the output and move backward, not the other way around. The loss is only known after a full forward pass produces a prediction. ∂L/∂a3 can't be computed until you know L, and ∂L/∂a2 depends on ∂L/∂a3 via the chain rule, which depends in turn on having already computed it. Each layer's gradient is only computable once the gradient of the layer after it is already known, which forces a strict back-to-front order: output layer first, then the layer before it, and so on back to the input.

Walking Through a Tiny Example by Hand

Consider a network with a single hidden neuron: a = φ(z), z = wx + b, and squared-error loss L = (a − y)^2. To find ∂L/∂w, chain through every intermediate quantity:

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_grad(z):
    s = sigmoid(z)
    return s * (1 - s)

x, w, b, y = 2.0, 0.5, 0.1, 1.0

# Forward pass
z = w * x + b
a = sigmoid(z)
L = (a - y) ** 2
print(f"Forward:  z={z:.4f}  a={a:.4f}  L={L:.4f}")

# Backward pass, one chain-rule term at a time
dL_da = 2 * (a - y)          # dL/da
da_dz = sigmoid_grad(z)      # da/dz
dz_dw = x                    # dz/dw

dL_dw = dL_da * da_dz * dz_dw  # chain rule
print(f"\nBackward: dL/da={dL_da:.4f}  da/dz={da_dz:.4f}  dz/dw={dz_dw:.4f}")
print(f"dL/dw = dL/da * da/dz * dz/dw = {dL_dw:.4f}")

Each line in the backward section is one term in the chain. Multiplying them together gives the gradient of the loss with respect to w, exactly what gradient descent needs to update it. The next lab walks through this same process for a multi-layer network, where the only real difference is that there are more terms to chain through, and matrices replace scalars.

What a Zero Gradient Actually Means

A gradient of exactly zero for some weight means: changing that weight by a small amount right now has no effect on the loss. It does not mean training is finished, and it does not mean something is broken. It's a purely local statement about the loss surface at the network's current weights. A ReLU neuron that's currently outputting zero for every example in the batch will show a zero gradient for its weights on this particular step, simply because its local derivative is zero there, exactly the dead-neuron scenario from the activation functions lesson.