Training Pipelines & Hardware

What you will learn in this lesson:

• Why GPUs accelerate deep learning specifically, not computation in general
• Why a data loading pipeline matters just as much as the model itself
• What mixed-precision training trades away, and what it buys back
• How these pieces fit together into one mental model of a training run

Why GPUs, Specifically

Every forward and backward pass in this course has boiled down to matrix multiplications: Wx, W^T·grad, batched versions of both. A matrix multiplication decomposes into millions of independent multiply-and-add operations that don't depend on each other's results. CPUs are built with a small number of powerful, flexible cores optimized for varied, sequential tasks. GPUs are built the opposite way: thousands of simpler cores, purpose-built to perform exactly this kind of massively parallel, repetitive arithmetic. Deep learning's near-total reliance on matrix multiplication is precisely why it benefits so disproportionately from GPU (and TPU) hardware, not because GPUs are faster at every kind of computation in general.

import numpy as np
import time

np.random.seed(0)
A = np.random.randn(800, 800)
B = np.random.randn(800, 800)

start = time.time()
for _ in range(5):
    C = A @ B
elapsed = time.time() - start
print(f"5 matrix multiplications (800x800): {elapsed:.3f}s on CPU (Pyodide/WASM)")
print("A GPU executing the exact same operation parallelizes the inner")
print("multiply-add work across thousands of cores simultaneously.")

This in-browser environment runs purely on CPU (via Pyodide/WebAssembly), so the timing above is illustrative of relative scale rather than representative of real training speeds; the conceptual point, that this operation is embarrassingly parallel, holds regardless of which hardware executes it.

The Data Pipeline: Keeping the GPU Fed

A GPU sitting idle, waiting for the next batch of data to be loaded from disk and preprocessed, wastes exactly the hardware investment that made fast training possible in the first place. A well-designed data pipeline overlaps two pieces of work: while the GPU is busy computing the current batch's forward and backward pass, the CPU (or a separate worker process) is simultaneously reading, decoding, and preprocessing the next batch, so it's ready the instant the GPU finishes.

import numpy as np
import time

def slow_data_load(batch_idx):
    time.sleep(0.02)  # simulating disk I/O and preprocessing
    return np.random.randn(64, 10)

def fake_gpu_step(batch):
    time.sleep(0.03)  # simulating a forward+backward pass
    return float(np.sum(batch))

# Serial: load, then compute, then load, then compute...
start = time.time()
for i in range(5):
    batch = slow_data_load(i)
    fake_gpu_step(batch)
serial_time = time.time() - start

# Prefetched: load batch i+1 "in the background" while computing on batch i
# (simulated here by just accounting for the time differently, illustrating
#  that the GPU's compute time and the loader's I/O time can overlap)
start = time.time()
next_batch = slow_data_load(0)
for i in range(5):
    batch = next_batch
    if i < 4:
        next_batch = slow_data_load(i + 1)  # would run concurrently in a real pipeline
    fake_gpu_step(batch)
prefetch_time = time.time() - start

print(f"Serial load-then-compute: {serial_time:.3f}s")
print(f"With prefetching design (conceptual): {prefetch_time:.3f}s")
print("In a real pipeline, the load step runs on a separate thread/process,")
print("truly overlapping with GPU compute rather than just being reordered.")

The in-browser simulation above can't truly run loading and compute concurrently, but the principle is the one that matters in real training infrastructure: a prefetching data loader running on background workers overlaps I/O-bound work with compute-bound work, so the expensive accelerator is never left waiting.

Mixed-Precision Training: Trading Precision for Speed

Most of this course has implicitly used 32-bit floating point numbers. Modern hardware can also compute in 16-bit formats (float16 or bfloat16) at roughly twice the throughput and half the memory footprint. Mixed-precision training uses these lower-precision formats for the bulk of a network's computation, while selectively keeping certain numerically sensitive steps, like gradient accumulation, in full 32-bit precision, to avoid the small numerical errors that lower precision introduces from compounding into a real problem.

import numpy as np

# Illustrating precision loss: float16 has far less precision than float32,
# especially for very small numbers, which is exactly what makes gradient
# accumulation a sensitive step to keep in higher precision.
small_gradients = np.array([1e-5, 2e-5, 1.5e-5, 3e-5], dtype=np.float32)
small_gradients_f16 = small_gradients.astype(np.float16)

print("float32 values:", small_gradients)
print("float16 values:", small_gradients_f16.astype(np.float32))
print("Relative error introduced by float16:",
      np.round(np.abs(small_gradients - small_gradients_f16.astype(np.float32)) / small_gradients, 4))

The relative error here is small per-value, but gradients accumulated across millions of weights and many training steps can compound such errors meaningfully, which is exactly why frameworks that support mixed-precision training keep specific operations, like the running accumulation of gradients, in float32 even while the bulk of the forward and backward pass runs in float16.

Putting It Together

A modern training run is the combination of all three pieces working in concert: matrix-multiplication-heavy computation that GPUs accelerate by orders of magnitude, a data pipeline that keeps that hardware continuously fed rather than idle, and mixed precision that trades a small, carefully managed amount of numerical accuracy for substantially faster, more memory-efficient computation. None of these pieces change what a network learns; they change how quickly and cheaply it can be trained at all.