Objective: Implement a vanilla RNN cell, unroll it across a sequence, and use it to solve a tiny sequence task: detecting whether a running sum of inputs has crossed a threshold. All in runnable, in-browser Python.
Task 1: Implement a Single RNN Cell
import numpy as np
def rnn_cell(x_t, h_prev, W_xh, W_hh, b):
z = W_xh @ x_t + W_hh @ h_prev + b
h_t = np.tanh(z)
return h_t
np.random.seed(0)
input_dim, hidden_dim = 2, 3
W_xh = np.random.randn(hidden_dim, input_dim) * 0.5
W_hh = np.random.randn(hidden_dim, hidden_dim) * 0.5
b = np.zeros(hidden_dim)
x_t = np.array([1.0, -0.5])
h_prev = np.zeros(hidden_dim)
h_t = rnn_cell(x_t, h_prev, W_xh, W_hh, b)
print("New hidden state:", np.round(h_t, 4))
Task 2: Unroll the Cell Across a Sequence
import numpy as np
def rnn_cell(x_t, h_prev, W_xh, W_hh, b):
return np.tanh(W_xh @ x_t + W_hh @ h_prev + b)
def rnn_forward(sequence, W_xh, W_hh, b, hidden_dim):
h = np.zeros(hidden_dim)
hidden_states = [h.copy()]
for x_t in sequence:
h = rnn_cell(x_t, h, W_xh, W_hh, b)
hidden_states.append(h.copy())
return hidden_states
np.random.seed(0)
input_dim, hidden_dim = 2, 3
W_xh = np.random.randn(hidden_dim, input_dim) * 0.5
W_hh = np.random.randn(hidden_dim, hidden_dim) * 0.5
b = np.zeros(hidden_dim)
sequence = [np.array([1.0, 0.0]), np.array([0.0, 1.0]), np.array([1.0, 1.0]), np.array([-1.0, 0.5])]
hidden_states = rnn_forward(sequence, W_xh, W_hh, b, hidden_dim)
for t, h in enumerate(hidden_states):
print(f"h_{t}: {np.round(h, 4)}")
h_0 is the zero vector every RNN starts from before seeing any input. Each subsequent
h_t depends on both the current input and the previous hidden state, which is itself a function
of everything before it.
Task 3: A Tiny Sequence Task — Has the Running Sum Crossed a Threshold?
Hand-craft a small set of weights that approximate detecting whether the cumulative sum of a 1-dimensional input sequence has crossed a threshold of 3, a simple stand-in for the kind of "remember something about the past, decide based on it" tasks RNNs are built for.
import numpy as np
def rnn_cell(x_t, h_prev, W_xh, W_hh, b):
return np.tanh(W_xh @ x_t + W_hh @ h_prev + b)
# Hand-designed (not learned) weights: h tracks an approximate running sum
W_xh = np.array([[1.0]]) # add the new input directly
W_hh = np.array([[1.0]]) # carry the previous hidden state forward unchanged
b = np.array([0.0])
def running_sum_rnn(sequence):
h = np.zeros(1)
history = []
for x_t in sequence:
# use a pre-tanh accumulator so it behaves like a literal running sum,
# then squash through tanh only to read out a "crossed threshold" signal
raw = W_xh @ np.array([x_t]) + W_hh @ h
h = raw # accumulate without squashing, to track the sum exactly
history.append(float(h[0]))
return history
sequence = [1.0, 1.0, 0.5, 1.0, -0.5, 1.0]
running_totals = running_sum_rnn(sequence)
threshold = 3.0
print("Input sequence: ", sequence)
print("Running sum: ", [round(v, 2) for v in running_totals])
print("Crossed threshold:", [v >= threshold for v in running_totals])
This deliberately simplified cell (no tanh squashing the accumulator) makes the "memory" explicit
and literal: the hidden state really is the running sum. A real, trained RNN would learn weights like these
from data rather than have them hand-set, but the structural idea, the hidden state carrying forward exactly
the information needed for the task, is identical.
Task 4: Watch the Gradient Vanish Across a Long Sequence
Reuse the gradient-shrinking illustration from the lesson, but tie it directly to sequence length, to see concretely how much signal is left after backpropagating through a long unrolled sequence.
import numpy as np
import matplotlib.pyplot as plt
np.random.seed(0)
hidden_dim = 8
W_hh = np.random.randn(hidden_dim, hidden_dim) * 0.2
n_steps = 30
grad = np.ones(hidden_dim)
magnitudes = [np.linalg.norm(grad)]
for _ in range(n_steps):
grad = W_hh.T @ grad
magnitudes.append(np.linalg.norm(grad))
plt.plot(magnitudes)
plt.yscale("log")
plt.xlabel("Steps propagated backward through time")
plt.ylabel("Gradient magnitude (log scale)")
plt.title("Vanishing gradient across an unrolled RNN")
show_plot()
The gradient magnitude collapses by many orders of magnitude well before reaching 30 steps back. Any dependency between the current loss and an input from 30+ steps earlier in the sequence would receive a vanishingly small training signal with this particular (randomly chosen, deliberately shrinking) recurrent weight matrix, exactly the motivation for the gated architectures in the next lesson.