Activation Functions

What you will learn in this lesson:

• Why no amount of depth helps without non-linear activations between layers
• How sigmoid and tanh saturate, and why that causes vanishing gradients in deep networks
• Why ReLU became the default hidden-layer activation, and what its own failure mode looks like
• The fixes for ReLU's "dead neuron" problem: Leaky ReLU, ELU, and friends
• A practical rule of thumb for choosing an activation per layer

Why Depth Needs Non-Linearity

Consider stacking two layers with no activation function between them: y = W2(W1x + b1) + b2. Expand that out and you get y = (W2W1)x + (W2b1 + b2), which is just another linear function of x, with a new weight matrix and a new bias. No matter how many purely linear layers you stack, the composition collapses into a single linear transformation. Depth without non-linearity buys you nothing beyond what one layer can already do.

import numpy as np

np.random.seed(0)
W1, b1 = np.random.randn(4, 3), np.random.randn(4)
W2, b2 = np.random.randn(2, 4), np.random.randn(2)

x = np.random.randn(3)

# Two "layers" with no activation in between
y_two_layers = W2 @ (W1 @ x + b1) + b2

# The exact same result as a single combined linear layer
W_combined = W2 @ W1
b_combined = W2 @ b1 + b2
y_one_layer = W_combined @ x + b_combined

print("Two linear layers: ", np.round(y_two_layers, 4))
print("One combined layer:", np.round(y_one_layer, 4))

The two outputs are identical, confirming the algebra: stacking linear layers is pointless. Inserting a non-linear activation between layers is what stops this collapse and lets each added layer represent something the previous layers could not.

Sigmoid and Tanh: The Original Smooth Switches

Sigmoid, σ(z) = 1 / (1 + e^(−z)), squashes any real number into (0, 1). Tanh, tanh(z) = (e^z − e^(−z)) / (e^z + e^(−z)), squashes into (−1, 1) and is zero-centered, which historically made it converge somewhat better than sigmoid for hidden layers.

Both share a fatal flaw for deep networks: they saturate. For large positive or negative z, both functions flatten out, and their derivative approaches zero. Sigmoid's derivative, σ(z)·(1 − σ(z)), peaks at just 0.25 (at z = 0) and shrinks toward zero everywhere else.

import numpy as np

def sigmoid(z):
    return 1 / (1 + np.exp(-z))

def sigmoid_grad(z):
    s = sigmoid(z)
    return s * (1 - s)

z = np.array([-6, -3, -1, 0, 1, 3, 6])
print("z:              ", z)
print("sigmoid(z):     ", np.round(sigmoid(z), 3))
print("sigmoid'(z):    ", np.round(sigmoid_grad(z), 4))

Now consider backpropagation through, say, eight such layers. The chain rule multiplies the local gradients together. If most of those local gradients are well under 0.25, multiplying eight of them together produces a number that's vanishingly small, on the order of 0.25^8 ≈ 0.000015 in the best case, and far smaller if any layer is even slightly saturated. This is the vanishing gradient problem: early layers in a deep sigmoid/tanh network receive almost no gradient signal and barely update at all.

ReLU: The Modern Default

ReLU (Rectified Linear Unit), ReLU(z) = max(0, z), looks almost too simple to matter. Its derivative is exactly 1 for any positive z, and exactly 0 for any negative z. For the positive region, that constant gradient of 1 means backpropagation can multiply through many ReLU layers without the signal shrinking the way it does with sigmoid or tanh.

ReLU is also far cheaper to compute, just a comparison and a max, with no exponentials involved, which matters when a layer is evaluated millions of times during training.

import numpy as np

def relu(z):
    return np.maximum(0, z)

def relu_grad(z):
    return (z > 0).astype(float)

z = np.array([-6, -3, -1, 0, 1, 3, 6])
print("z:          ", z)
print("relu(z):    ", relu(z))
print("relu'(z):   ", relu_grad(z))

Notice the gradient is a flat 1 for every positive value, regardless of magnitude. There's no saturation on the positive side at all.

ReLU's Own Failure Mode: Dead Neurons

ReLU trades the vanishing-gradient problem for a different one. If a neuron's weighted sum ends up negative for every example in the training set, perhaps because of an unlucky initialization or a large negative gradient update, its output is 0 for every input, and its gradient is also 0 for every input. That neuron has no path to ever recover: it contributes nothing to the forward pass and receives no gradient to update its weights. This is called a dead neuron.

In a large network this is rarely catastrophic, since plenty of other neurons stay active, but it does waste capacity, and in extreme cases (badly chosen learning rates, poor initialization) a large fraction of a layer can die.

Variants That Patch the Dead-Neuron Problem

A family of ReLU variants exists specifically to keep a small, non-zero gradient flowing even for negative inputs, so a neuron's weights always have at least a small chance to recover:

• Leaky ReLU: f(z) = z if z > 0 else αz, with a small fixed α (e.g. 0.01) instead of a flat zero for negative inputs.
• Parametric ReLU (PReLU): the same shape as Leaky ReLU, but α is a learned parameter rather than a fixed constant.
• ELU (Exponential Linear Unit): smoothly curves toward a negative constant for negative inputs instead of a sharp kink at zero, which can help optimization in some networks at the cost of being more expensive to compute.

import numpy as np
import matplotlib.pyplot as plt

z = np.linspace(-5, 5, 200)

def relu(z): return np.maximum(0, z)
def leaky_relu(z, alpha=0.1): return np.where(z > 0, z, alpha * z)
def elu(z, alpha=1.0): return np.where(z > 0, z, alpha * (np.exp(z) - 1))

plt.plot(z, relu(z), label="ReLU")
plt.plot(z, leaky_relu(z), label="Leaky ReLU (alpha=0.1)")
plt.plot(z, elu(z), label="ELU (alpha=1.0)")
plt.axhline(0, color="gray", linewidth=0.5)
plt.axvline(0, color="gray", linewidth=0.5)
plt.legend()
plt.title("ReLU and its leaky/smooth variants")
show_plot()

The exaggerated alpha=0.1 above makes the leak visible; in practice it's usually much smaller (0.01 or so), just enough to keep a non-zero gradient alive without changing the function's behavior much for positive inputs.

A Practical Rule of Thumb

1. Hidden layers, general-purpose feedforward/convolutional networks: start with ReLU. It's cheap, it works well in practice, and most architectures are tuned around it.
2. Seeing a lot of dead neurons in practice? Try Leaky ReLU or ELU before reaching for anything more exotic.
3. Binary classification output: sigmoid, so the output is interpretable as a probability.
4. Multi-class classification output: softmax, covered in the loss functions lesson later this phase.
5. Regression output: usually no activation at all (a raw linear output), since you want the network able to predict any real number, not one squashed into a bounded range.