Weight Initialization — Checkpoint Quiz

B is correct. If every weight starts at zero, every neuron in a layer computes the same function of the input and gets the same gradient update, a symmetry that never breaks. Random initialization breaks this symmetry so neurons can specialize.

A is correct. Without careful scaling, the variance of activations can shrink toward zero or grow without bound as signals pass through many layers. These schemes choose initial weight scales based on the number of inputs/outputs of a layer to keep signal variance roughly constant.

B is correct. ReLU sets all negative inputs to zero, which roughly halves the variance of the signal passing through. He initialization scales up the initial weight variance to compensate for that, while Xavier's derivation assumes an activation that doesn't zero out half the signal.