The Heart of Learning

You know about neurons, activation functions, layers, loss functions, and gradient descent. You know the network needs gradients to improve. But how does it compute the gradient for a weight buried 20 layers deep? The answer connects everything you've learned into one elegant mechanism — and it's built entirely on the chain rule from Path I.C.

Backpropagation is not a new invention. It is the systematic application of the chain rule to a neural network's computational graph — a bookkeeping method that makes gradient descent practical for deep networks.

The Complete Training Step

A training step consists of three inseparable phases: forward pass (compute prediction), backward pass (compute gradients), and update (adjust weights). Backpropagation is the backward pass — nothing more, nothing less.

Worked Example: A 2-Weight Mini-Network

Input x = 2, weights w₁ = 0.5 and w₂ = 3.0, target y_true = 5. The network computes: h = x · w₁, then y = h · w₂. Loss: L = (y − y_true)².

The network has learned: both weights were adjusted so the prediction will be closer to the target next time. The gradient for w₁ is six times larger than for w₂ — because the chain rule multiplies in the additional factors w₂ and x.

The forward pass computes the prediction, the backward pass computes the gradients, the update step computes the new weights. This trio repeats thousands of times per training epoch — and that is exactly how a neural network learns.

Interactive: Forward & Backward Pass

You've worked through the training step on the mini-network. Now you can walk through it step by step: watch how values flow forward through the network, how gradients are computed backwards, and how the weights change at the end.

The Chain Rule in Action — Layer by Layer Backwards

From 2 weights to N layers: the principle stays identical, only the chain gets longer. For a weight in the first layer of a deep network, the chain rule must be applied through ALL layers.

Backpropagation's efficiency comes from reuse: once the error signal for layer 2 is computed, it's stored. Layer 1 simply multiplies by its local derivative — no recomputation from the loss. Every partial derivative is computed exactly once and cached.

Interactive: Clickable Chain Rule

The chain rule decomposes the gradient into local derivatives. Click on the individual terms to see what each factor in the chain means — and how they combine to produce the gradient for w₁.

Autograd — Why You'll Never Do It by Hand

In practice, you never write backpropagation manually. Modern frameworks like PyTorch and TensorFlow include autograd — an automatic differentiation system that handles the entire backward pass for you.

for epoch in range(100):
    prediction = net(X_train)
    loss = criterion(prediction, y_train)
    optimizer.zero_grad()   # Clear old gradients
    loss.backward()         # Backpropagation in ONE line
    optimizer.step()        # Gradient descent update

How does it work? While you code the forward pass, the framework silently builds a dynamic computational graph in the background. Every mathematical operation is recorded. When you call .backward(), the framework automatically traverses this graph in reverse, applying the chain rule at every node. This works for any differentiable computation — not just neural networks.

Yet autograd doesn't mean you can ignore the math. Three concrete scenarios where manual understanding is indispensable: (1) Debugging: When your loss plateaus, you need to diagnose vanishing gradients. (2) Architecture design: Choosing activation functions and skip connections depends on understanding gradient flow. (3) Custom layers: Writing custom autograd functions requires correct derivatives.