Objective: Implement a dropout layer as a small class with explicit train/inference modes, and empirically verify that the rescaling trick keeps the expected output magnitude consistent regardless of the dropout rate. All in runnable, in-browser Python.
Task 1: A Dropout Function With a Training Flag
import numpy as np
def dropout(a, p, training, rng):
if not training:
return a
mask = (rng.random(a.shape) > p).astype(float)
return a * mask / (1 - p)
rng = np.random.default_rng(42)
a = np.array([2.0, 4.0, 6.0, 8.0, 10.0])
print("Training mode (p=0.4):", np.round(dropout(a, p=0.4, training=True, rng=rng), 3))
print("Inference mode: ", dropout(a, p=0.4, training=False, rng=rng))
Task 2: Verify the Rescaling Keeps the Expected Sum Constant
Run dropout many times over the same input and average the results. If the rescaling is correct, that average should converge to the original, undropped activations.
import numpy as np
def dropout(a, p, training, rng):
if not training:
return a
mask = (rng.random(a.shape) > p).astype(float)
return a * mask / (1 - p)
rng = np.random.default_rng(0)
a = np.array([2.0, 4.0, 6.0, 8.0, 10.0])
n_trials = 20000
results = np.array([dropout(a, p=0.5, training=True, rng=rng) for _ in range(n_trials)])
average_output = results.mean(axis=0)
print("Original activations: ", a)
print(f"Average over {n_trials} dropout passes:", np.round(average_output, 3))
The averaged output should land very close to the original a, confirming the rescaling by
1 / (1 - p) exactly compensates for zeroing a fraction p of activations on average.
Try changing p to 0.2 or 0.8 and rerunning: the averaged output should still converge to the
original activations either way, since the rescaling adapts automatically to whatever p you use.
Task 3: A Small Dropout Layer Class
Wrapping this in a class with explicit train() and eval() modes mirrors how real
deep learning frameworks structure their layers, and makes the train/inference switch explicit and impossible
to forget.
import numpy as np
class Dropout:
def __init__(self, p, seed=0):
self.p = p
self.training = True
self.rng = np.random.default_rng(seed)
def train(self):
self.training = True
def eval(self):
self.training = False
def __call__(self, a):
if not self.training:
return a
mask = (self.rng.random(a.shape) > self.p) / (1 - self.p)
return a * mask
layer = Dropout(p=0.3)
a = np.array([1.0, 2.0, 3.0, 4.0, 5.0])
layer.train()
print("Train mode output:", np.round(layer(a), 3))
layer.eval()
print("Eval mode output: ", layer(a))
This is exactly the pattern used by PyTorch's model.train() / model.eval() and
similar calls in other frameworks: a single flag that flips every dropout (and batch normalization) layer in
a network between its training behavior and its inference behavior, without touching the rest of the model's
code.
Task 4: Dropout's Effect on a Small Network's Predictions
Apply dropout between two layers of a tiny forward pass and observe how predictions vary across repeated training-mode forward passes on the exact same input, versus the single, stable prediction you get in eval mode.
import numpy as np
class Dropout:
def __init__(self, p, seed=0):
self.p = p
self.training = True
self.rng = np.random.default_rng(seed)
def train(self): self.training = True
def eval(self): self.training = False
def __call__(self, a):
if not self.training:
return a
mask = (self.rng.random(a.shape) > self.p) / (1 - self.p)
return a * mask
def relu(z): return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-z))
np.random.seed(1)
W1, b1 = np.random.randn(8, 3) * 0.5, np.zeros(8)
W2, b2 = np.random.randn(1, 8) * 0.5, np.zeros(1)
dropout_layer = Dropout(p=0.5, seed=7)
x = np.array([1.0, -0.5, 2.0])
def forward(x, training):
dropout_layer.training = training
a1 = relu(W1 @ x + b1)
a1 = dropout_layer(a1)
y_hat = sigmoid(W2 @ a1 + b2)
return float(y_hat[0])
print("Training mode predictions (vary run to run):")
for _ in range(5):
print(" ", round(forward(x, training=True), 4))
print("\nEval mode predictions (identical every time):")
for _ in range(3):
print(" ", round(forward(x, training=False), 4))
The training-mode predictions jitter from call to call, since a different random subset of hidden neurons is dropped each time, while the eval-mode predictions are perfectly stable, exactly the deterministic behavior you want when actually serving predictions to a user.