Forward Propagation

What you will learn in this lesson:

• How a single layer's computation (matrix multiply, add bias, apply activation) chains into a full network
• How to read and interpret weight matrix shapes
• Why a deep network is best understood as a composition of functions, f3(f2(f1(x)))
• Why that composition view is the key to understanding backpropagation in the next lesson

From One Neuron to a Layer

The previous phase covered a single neuron: z = w·x + b, a = φ(z). A layer is simply many neurons computed in parallel from the same input, each with its own weight vector and bias. Stacking those weight vectors as rows of a matrix W, and the biases into a vector b, lets you compute an entire layer's output in one matrix operation:

If W has shape (n_out, n_in), each of its n_out rows is one neuron's weight vector, dotted against the n_in-dimensional input. The output z is a vector of length n_out: one pre-activation value per neuron in the layer.

import numpy as np

np.random.seed(0)
n_in, n_out = 3, 4
W = np.random.randn(n_out, n_in)
b = np.random.randn(n_out)
x = np.array([1.0, -2.0, 0.5])

z = W @ x + b
a = np.maximum(0, z)  # ReLU

print("W shape:", W.shape, " (n_out, n_in)")
print("z =", np.round(z, 3))
print("a = ReLU(z) =", np.round(a, 3))

This single layer took a 3-dimensional input and produced a 4-dimensional activation vector. That activation vector is exactly the right shape to be fed as the input to a second layer.

Chaining Layers: A Composition of Functions

A network with three layers computes:

Each layer's output becomes the next layer's input. Written as nested functions, this is exactly ŷ = f3(f2(f1(x))), where each fi bundles up that layer's weights, bias, and activation. This "composition of functions" framing isn't just notation. It's the precise mathematical structure that makes the chain rule, and therefore backpropagation, applicable: computing how the final output depends on an early layer's weights is a calculus question about differentiating a composition, which the chain rule answers directly.

import numpy as np

np.random.seed(1)

def relu(z): return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-z))

# Layer shapes: 3 -> 4 -> 4 -> 1  (a tiny binary-classification network)
W1, b1 = np.random.randn(4, 3) * 0.5, np.zeros(4)
W2, b2 = np.random.randn(4, 4) * 0.5, np.zeros(4)
W3, b3 = np.random.randn(1, 4) * 0.5, np.zeros(1)

def forward(x):
    a1 = relu(W1 @ x + b1)
    a2 = relu(W2 @ a1 + b2)
    y_hat = sigmoid(W3 @ a2 + b3)
    return y_hat, (a1, a2)  # return intermediate activations too, used later in backprop

x = np.array([1.0, -0.5, 2.0])
y_hat, (a1, a2) = forward(x)

print("Layer 1 activations (a1):", np.round(a1, 3))
print("Layer 2 activations (a2):", np.round(a2, 3))
print("Final prediction (y_hat):", np.round(y_hat, 4))

Notice the function returns the intermediate activations a1 and a2, not just the final prediction. That isn't incidental: backpropagation needs every layer's activations from the forward pass to compute that layer's gradients. Discarding them would mean recomputing the entire forward pass again during the backward pass.

Reading Weight Matrix Shapes

Given a weight matrix's shape, you can immediately read off what that layer does. A matrix of shape (64, 128) takes a 128-dimensional input and produces a 64-dimensional output: 64 neurons, each with 128 weights (one per input feature) plus its own bias. This convention, (n_out, n_in), is what lets you sanity-check an architecture at a glance: the number of columns in one layer's weight matrix must equal the number of rows (the output size) of the layer before it, or the matrix multiplication is simply undefined.

Forward Propagation Is Just Evaluation, Not Learning

It's worth being precise about what forward propagation does and doesn't do: it evaluates the network's current weights on an input to produce a prediction. Nothing about this process changes any weight. Forward propagation is "run the function as currently defined." Learning, adjusting the weights so the prediction gets closer to the true target, is the job of backpropagation plus an optimizer, which is exactly where the next two lessons go.