Deep Learning: Notation & Terms Glossary

Every notation card and key term introduced across Deep Learning, collected on one page so you don't have to hunt back through lessons to remember what a symbol meant. Terms from every course are also searchable together in the site-wide glossary.

Neurons & Activations

w, b: A neuron's weights and bias. Its pre-activation output is z = w·x + b.
z: A neuron's pre-activation value, before an activation function is applied.
a: A neuron's activation: the output after applying a non-linear activation function to z.
ReLU: max(0, z). The default hidden-layer activation in most modern networks; cheap to compute and avoids vanishing gradients for positive inputs.
Sigmoid / tanh: Smooth, saturating activations bounded in (0, 1) or (−1, 1). Prone to vanishing gradients in deep networks since their derivative shrinks toward zero at the extremes.
Dead neuron: A ReLU neuron stuck outputting zero for every input, because its weights drove it permanently into the negative, zero-gradient region.

Forward & Backward Propagation

Forward propagation: Computing a network's output by passing an input through each layer's weights, bias, and activation in sequence.
Backpropagation: Computing the gradient of the loss with respect to every weight by applying the chain rule backward from the output layer to the input layer.
Backpropagation through time (BPTT): Backpropagation applied to an unrolled recurrent network, propagating gradients backward across time steps as well as layers.
Vanishing gradient: Gradients shrinking toward zero as they're propagated backward across many layers or time steps, stalling learning in early layers.
Exploding gradient: Gradients growing without bound as they're propagated backward, causing unstable, divergent updates.
Cross-entropy loss: The standard classification loss, paired with a softmax output. Its gradient stays informative even for confidently wrong predictions.
Softmax: Converts a vector of raw scores (logits) into a probability distribution: exponentiate each value, then normalize so they sum to 1.

Optimization

SGD: Stochastic gradient descent: updating weights using the gradient from one example or a small mini-batch, rather than the full dataset.
Momentum: A running average of past gradients added to the current update, smoothing noisy steps and accelerating progress in consistent directions.
Learning rate (η): The step size used in a gradient descent update: w ← w − η·∇w.
Learning rate decay: Shrinking the learning rate over the course of training, taking large early steps and smaller, more precise steps later.
Warmup: Starting training with a small learning rate and gradually increasing it, to avoid destabilizing updates while weights are still freshly initialized.
Xavier / Glorot initialization: Scales initial random weights based on layer size to keep activation variance roughly stable across layers, suited to symmetric activations like tanh.
He initialization: Like Xavier, but scaled up to compensate for ReLU zeroing out roughly half its inputs.

Regularization

Dropout: Randomly zeroing a fraction of activations on each forward pass during training, so the network can't over-rely on any single neuron.
Batch normalization: Normalizing activations within each mini-batch to stable mean and variance, reducing internal shift between layers and stabilizing training.
Early stopping: Halting training once validation performance stops improving, keeping the weights from before the model began overfitting.
Data augmentation: Applying realistic transformations (crops, flips, rotations, etc.) to training examples to expose the model to more variation without collecting new data.

Convolutional Networks

Kernel / filter: A small grid of learned weights that slides across an input, computing a weighted sum of each local patch it covers.
Feature map: The grid of outputs produced by sliding one filter across an input; high values mark where that filter's pattern was detected.
Padding: Adding a border (usually zeros) around an input before convolving, used to control the output's spatial size.
Stride: The step size a filter moves between positions. Larger strides produce smaller output feature maps.
Pooling (max / average): Downsampling a feature map by keeping only the max (or average) value within each small local region.
Residual / skip connection: A shortcut that adds a layer's input directly to its output, letting gradients flow around the layer and making very deep networks easier to train.

Sequence Models

Hidden state: The vector an RNN carries forward between time steps, summarizing everything the network has processed in the sequence so far.
Unrolling: Representing an RNN's repeated application across time steps as one long computational graph with shared weights, so backpropagation can be applied across the whole sequence.
Cell state (LSTM): A separate memory channel in an LSTM, regulated by gates, that lets information persist over much longer sequences than a vanilla RNN's hidden state alone.
Forget / input / output gate: The three learned gates in an LSTM that control what is discarded from, added to, and read out of the cell state at each time step.
GRU: Gated Recurrent Unit: merges the cell and hidden state into one and uses two gates instead of three, a simpler alternative to the LSTM.
Encoder-decoder: An architecture where an encoder compresses an input sequence into a representation that a decoder uses to generate an output sequence, possibly of a different length.

Attention & Transformers

Query, key, value (Q, K, V): The three projections of an input used in attention: a query is compared against keys to compute weights, which combine the values.
Attention weights: A softmax distribution over input positions, derived from query/key similarity, used to weight a combination of values.
Self-attention: Attention where queries, keys, and values all come from the same sequence, letting every position relate to every other position.
Multi-head attention: Running several attention computations in parallel, each with its own learned projections, so different heads can specialize in different relationships.
Positional encoding: A signal added to input embeddings that injects information about token order, since self-attention has no inherent sense of position.
Fine-tuning: Continuing to train a pretrained model's weights (often with a small learning rate, sometimes with early layers frozen) on a new, usually smaller, task-specific dataset.

Practical Training

Epoch: One full pass through the entire training dataset.
Batch size: The number of training examples used to compute one gradient update.
Mixed precision: Using lower-precision number formats (e.g. float16) for most computation, combined with float32 for numerically sensitive steps, to speed up training.
Overfit-a-tiny-batch test: A debugging sanity check: a correctly wired model should be able to drive its loss near zero on a handful of examples it should easily memorize.