Loss Functions for Deep Networks

What you will learn in this lesson:

• How softmax converts raw output scores into a probability distribution
• Why cross-entropy loss is the standard pairing for a softmax output, and how their gradients combine
• Why mean squared error remains the default for regression outputs
• Why the choice of loss function is really a choice about gradient behavior, not just "what number to report"

Softmax: Turning Scores Into Probabilities

A multi-class classification network's final layer typically outputs one raw score (called a logit) per class, with no activation applied. These logits can be any real number, positive or negative, and don't sum to anything meaningful on their own. Softmax converts them into a valid probability distribution:

Exponentiating makes every value positive; dividing by the sum makes everything sum to exactly 1. Larger logits get larger probabilities, and the relative gaps between logits are preserved (exponentially amplified, in fact) in the resulting probabilities.

import numpy as np

def softmax(z):
    z_shifted = z - np.max(z)  # numerical stability trick, explained below
    exp_z = np.exp(z_shifted)
    return exp_z / np.sum(exp_z)

logits = np.array([2.0, 1.0, 0.1])
probs = softmax(logits)
print("Logits:        ", logits)
print("Probabilities: ", np.round(probs, 4))
print("Sum of probs:  ", round(float(np.sum(probs)), 6))

Subtracting the maximum logit before exponentiating doesn't change the result mathematically (it cancels out in the division), but it prevents np.exp from overflowing on large logits, a standard numerical-stability trick worth keeping in any softmax implementation.

Cross-Entropy: Measuring How Wrong a Probability Distribution Is

Once a network outputs a probability distribution over classes, cross-entropy measures how far that distribution is from the true label. For a single example with true class index c:

If the model assigns the correct class a probability close to 1, −log(p_c) is close to 0: low loss. If the model assigns the correct class a probability close to 0, −log(p_c) blows up toward infinity: a severe penalty for confidently wrong predictions.

import numpy as np

def cross_entropy(probs, true_class_idx):
    return -np.log(probs[true_class_idx] + 1e-12)  # epsilon avoids log(0)

# Same true class (index 0), three different predicted distributions
confident_correct = np.array([0.97, 0.02, 0.01])
unsure = np.array([0.4, 0.35, 0.25])
confident_wrong = np.array([0.02, 0.9, 0.08])

print("Confident & correct loss:", round(cross_entropy(confident_correct, 0), 4))
print("Unsure loss:              ", round(cross_entropy(unsure, 0), 4))
print("Confident & wrong loss:   ", round(cross_entropy(confident_wrong, 0), 4))

The confident-and-wrong case produces by far the largest loss, exactly the behavior you want: the network should be punished hardest for being both wrong and overconfident about it.

Why Softmax + Cross-Entropy Is a Particularly Good Pairing

The reason this combination is the default for classification isn't just convention. When you work out ∂L/∂z (the gradient of cross-entropy loss with respect to the pre-softmax logits), the exponentials and logarithms cancel in a remarkably clean way:

That's it: predicted minus actual, elementwise. This gradient never saturates the way it would if you ran mean squared error through a sigmoid or softmax output. A confidently wrong prediction (predicted probability near 0 for the correct class) produces a gradient near 1 in magnitude, exactly when you want the strongest push to correct the network. This is the concrete answer to "why cross-entropy and not MSE for classification": MSE through a saturating output can produce a tiny gradient exactly in the cases where you most need a large corrective signal.

import numpy as np

def softmax(z):
    z = z - np.max(z)
    e = np.exp(z)
    return e / np.sum(e)

# A confidently wrong prediction: true class is 0, but the model favors class 1
logits = np.array([-3.0, 4.0, -1.0])
probs = softmax(logits)
y_true = np.array([1, 0, 0])  # one-hot for class 0

grad = probs - y_true
print("Predicted probabilities:", np.round(probs, 4))
print("Gradient (p - y):       ", np.round(grad, 4))

The gradient on the true class's logit is strongly negative (push it up), which is exactly the large, unsaturated correction signal the network needs when it's this wrong.

Mean Squared Error: The Default for Regression

For predicting a continuous value rather than a class, mean squared error remains the standard choice:

Its gradient is proportional to the size of the error: a prediction that's off by 10 produces a gradient ten times larger than a prediction off by 1. This naturally focuses learning on the examples where the model is currently most wrong, and it has no saturation issue analogous to sigmoid/softmax because, for a regression output, there's typically no bounding activation function applied at all, the network's final layer is just a raw linear combination.

Choosing a Loss Is Choosing Gradient Behavior

The throughline of this lesson: a loss function isn't merely a scorecard you check after training finishes. It's the function backpropagation differentiates first, and its gradient is the very first link in the chain rule that flows through every layer of the network. Two losses that rank model quality identically can still produce wildly different training dynamics if their gradients behave differently near the extremes, which is exactly the practical reason cross-entropy, not accuracy and not MSE, is the default for classification.