From the Perceptron to the Modern Neuron

What you will learn in this lesson:

• How a perceptron computes a decision from weighted inputs and a step function
• Why the step function makes perceptrons untrainable by gradient descent
• How replacing the step function with a smooth activation gives you the modern artificial neuron
• Why a single neuron, no matter which activation it uses, is still limited to a linear decision boundary
• What the bias term buys you that the weights alone cannot

The Perceptron: A Decision Machine with a Hard Switch

In 1958, Frank Rosenblatt described the perceptron: a model that takes a vector of inputs, multiplies each by a weight, sums the results together with a bias, and then fires a hard "yes" or "no" depending on whether that sum crosses zero.

Formally, given inputs x = [x1, x2, ..., xn], weights w = [w1, w2, ..., wn], and a bias b, the perceptron computes:

This is a genuinely useful idea: it can learn a linear decision boundary that separates two classes, and Rosenblatt proved a learning rule that updates w and b whenever the perceptron gets a training example wrong. For data that is linearly separable, the rule provably converges to a boundary that gets every training example right.

Why the Step Function Is a Dead End

The problem appears the moment you try to stack perceptrons into layers, or train them with anything more sophisticated than "nudge weights when wrong." The step function's derivative is zero everywhere except at z = 0, where it is undefined. That means: if you ask "how should I change this weight to slightly improve the output," the honest answer, almost everywhere, is "changing it by a tiny amount makes no difference at all, the output doesn't move."

Gradient-based learning, the engine behind essentially all of modern deep learning, depends on exactly this kind of question having a useful, non-zero answer. A function whose derivative is zero (or undefined) almost everywhere gives gradient descent nothing to work with. You cannot compute a useful gradient signal through a hard step, and you certainly cannot propagate one backward through several stacked steps.

The Modern Neuron: Same Shape, Smooth Switch

The fix that unlocked deep learning was conceptually simple: keep the weighted sum, but replace the step function with a smooth, differentiable activation function, φ:

Early choices for φ were the sigmoid (squashing z into (0, 1)) and tanh (squashing it into (−1, 1)). Both are smooth approximations of the step function: they still produce something switch-like, but with a derivative that's well-defined and non-zero across most of their range, instead of zero everywhere. Modern networks mostly use ReLU and its variants instead, which you'll meet in the next lesson, for reasons that matter once networks get deep.

Here is the entire idea in runnable code: the same weighted sum, evaluated once with a hard step and once with a sigmoid, showing how the sigmoid gives you a continuous, differentiable output instead of an abrupt jump.

import numpy as np

def step(z):
    return np.where(z > 0, 1.0, 0.0)

def sigmoid(z):
    return 1.0 / (1.0 + np.exp(-z))

z = np.linspace(-5, 5, 11)
print("z:          ", np.round(z, 2))
print("step(z):    ", step(z))
print("sigmoid(z): ", np.round(sigmoid(z), 3))

Notice how step(z) jumps instantly from 0 to 1 at z = 0, while sigmoid(z) eases smoothly between the two. That smoothness is precisely what gives you a non-zero gradient everywhere, which is what backpropagation, covered two lessons from now, needs to do its job.

A Single Neuron Is Still a Linear Boundary

It's worth being precise about what a single neuron, by itself, can and cannot do. The weighted sum w·x + b is a linear combination of the inputs: it can only express a flat line, plane, or hyperplane. The activation function φ is applied after that sum, and while it does introduce a non-linearity, applying a non-linear function to a linear boundary still leaves you with a single boundary whose shape is determined entirely by the linear part. A sigmoid neuron's decision boundary (the set of points where its output crosses 0.5) is still exactly the same flat hyperplane a perceptron would draw.

What changes is not the shape of any single neuron's boundary, but what you can build by composing many neurons across multiple layers. A single neuron, no matter its activation, is a linear classifier. A network of them, with non-linear activations between layers, can represent curved, arbitrarily complex boundaries. That compositional power is the entire subject of the next two phases.

What the Bias Actually Buys You

It's easy to treat the bias term b as an afterthought, but it plays a specific geometric role. Without it, the pre-activation z = w·x is forced to equal exactly zero whenever every input is zero, which means every neuron's decision boundary is forced to pass through the origin. The bias is a free offset that lets the boundary sit anywhere in space, not just through the origin.

Concretely: imagine trying to separate two classes of points that are both clustered far from the origin, with no boundary through the origin able to split them. A neuron without a bias is structurally incapable of solving this, no matter how its weights are tuned. Add a single bias term, and the same neuron can now place its boundary anywhere a straight line could plausibly go.

Why This Matters for Everything That Follows

Every architecture in this course, fully connected layers, convolutions, recurrent cells, attention, is built out of the same primitive: a weighted sum followed by a differentiable non-linearity. The specific weights differ, the way inputs are wired together differs, but the requirement that every operation in the network be differentiable, so gradients can flow backward through it, never goes away. Understanding why the perceptron's step function had to go is understanding the single design constraint that shapes the rest of deep learning.