Check your understanding of this module before moving on.
?
0/3
Checkpoint quiz 3 questions
A quick check on this module's key ideas before you move on
Question 01
Why can't a deep network made entirely of linear layers (no non-linear activations) represent more complex functions than a single linear layer?
A is correct. Matrix multiplication composed with matrix multiplication is itself just a matrix multiplication. Without a non-linearity between layers, depth buys nothing beyond what one linear layer can already do.
Question 02
What is the main practical problem with using sigmoid activations in many hidden layers of a deep network?
B is correct. Each layer's backward pass multiplies by the local gradient. Since sigmoid's derivative tops out at 0.25 and approaches 0 in saturated regions, multiplying many such small numbers together across layers makes early-layer gradients vanish.
Question 03
Why is ReLU (max(0, x)) generally preferred over sigmoid/tanh as a default hidden-layer activation in deep networks?
A is correct. ReLU avoids the vanishing-gradient problem for positive inputs since its derivative is a constant 1, and it's just a max operation, far cheaper than an exponential. It does have its own issue (dead neurons for permanently negative inputs), but it's still the common default.