The Heart of Learning
How a neural network sends its errors home — and makes everything along the way a little better.
You know about neurons, activation functions, layers, loss functions, and gradient descent. You know the network needs gradients to improve. But how does it compute the gradient for a weight buried 20 layers deep? The answer connects everything you've learned into one elegant mechanism — and it's built entirely on the chain rule from the “Linear Algebra & Calculus” learning path.
Backpropagation is not new mathematics. It is the systematic application of the chain rule to a neural network's computational graph — a bookkeeping method that makes gradient descent practical for deep networks.
Backpropagation Algorithm
The birth of modern machine learning through an elegant training algorithm. In October 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published the paper 'Learning representations by back-propagating errors' in Nature. This algorithm considerably changed the training of neural networks by providing an efficient method for adjusting weights in multi-layer networks. The procedure repeatedly adjusts the connection weights to minimize the difference between actual and desired output. The decisive innovation lay in the ability to train hidden layers that automatically recognize important features of the task. The mathematical foundations had already been derived earlier — by Paul Werbos (1974) and Seppo Linnainmaa (1970), among others — but it was this paper that made backpropagation widely known and convincingly demonstrated its effectiveness. Backpropagation became the workhorse of machine learning and today enables all modern deep learning applications.
The Complete Training Step
- 1. Forward PassInput flows forward through all layers, producing a prediction and a loss value
- 2. Backward PassThe error signal flows backwards, the chain rule yields the gradient for every weight
- 3. UpdateGradient descent adjusts each weight: w = w − lr · gradient
Definition
Backpropagation
Analogy
Example
Definition
A training step consists of three inseparable phases: forward pass (compute prediction), backward pass (compute gradients), and update (adjust weights). Backpropagation is the backward pass — nothing more, nothing less.
Worked Example: A 2-Weight Mini-Network
Input x = 2, weights w₁ = 0.5 and w₂ = 3.0, target y_true = 5. The network computes: h = x · w₁, then y = h · w₂. Loss: L = (y − y_true)².
The network has learned: both weights were adjusted so the prediction will be closer to the target next time. The gradient for w₁ is six times larger than for w₂ — because the chain rule multiplies in the additional factors w₂ and x.
Caution
Misconception: Backpropagation Is a Separate Learning Algorithm
No! Backpropagation only computes gradients — by systematically applying the chain rule to the computational graph. The actual learning algorithm is gradient descent (or a variant like Adam), which uses these gradients to update the weights. Backpropagation without gradient descent learns nothing; gradient descent without backpropagation cannot efficiently compute the gradients.
The forward pass computes the prediction, the backward pass computes the gradients, the update step computes the new weights. This trio often repeats thousands of times during training — and that is exactly how a neural network learns.
Interactive: Forward & Backward Pass
You've worked through the training step on the mini-network. Now you can walk through it step by step: watch how values flow forward through the network, how gradients are computed backwards, and how the weights change at the end.
The mini-network has input x, two weights w₁ and w₂, and a target value. No computations have been performed yet.
The Chain Rule in Action — Layer by Layer Backwards
From 2 weights to N layers: the principle stays identical, only the chain gets longer. For a weight in the first layer of a deep network, the chain rule must be applied through ALL layers.
Backpropagation's efficiency comes from reuse: once the error signal for layer 2 is computed, it's stored. Layer 1 simply multiplies by its local derivative — no recomputation from the loss. Every partial derivative is computed exactly once and cached.
Maximum derivative ≈0.25. With 10 layers (and weights with a magnitude below 1) at most 0.25¹⁰ ≈ 0.00000095. The gradient vanishes — early layers barely learn anymore.
Derivative is 0 or 1. For positive inputs, the gradient flows through unattenuated — no exponential decay.
Caution
Dead Neuron: When the Gradient Vanishes Forever
Concrete example: x = 1, w₁ = −2. Forward pass: z₁ = 1 · (−2) = −2. ReLU(−2) = 0. The local derivative of ReLU at −2 is exactly 0. No matter how large the error at the end of the network — as soon as the error signal flows through this ReLU node, it is multiplied by 0. The gradient for w₁ becomes 0, the weight is never updated. If z₁ stays negative for all training inputs, the neuron is permanently dead.
Caution
Misconception: Vanishing Gradients Mean the Network Has a Bug
No! Vanishing gradients are a mathematical inevitability when multiplying many small numbers. With 10 sigmoid layers, the gradient is reduced to less than one millionth — that's not a code error, it's mathematics. The solution is architectural (ReLU instead of sigmoid, residual connections, batch normalization), not debugging.
ReLU Family: ReLU has a derivative of 1 for positive values and passes gradients through unfiltered. Leaky ReLU and ELU also prevent dead neurons by allowing small gradients for negative inputs. Residual Connections: ResNet skip connections add a layer's input directly to its output, creating shortcut paths for gradient flow — even through hundreds of layers. Batch Normalization: Normalizes each layer's activations, stabilizing gradient distributions and accelerating training. Gradient Clipping: Caps gradients exceeding a threshold, preventing numerical instability from exploding gradients.
Interactive: Clickable Chain Rule
The chain rule decomposes the gradient into local derivatives. Click on the individual terms to see what each factor in the chain means — and how they combine to produce the gradient for w₁ in the mini network.
Chain rule for w₁
The total gradient: How much does the loss change when w₁ changes? This is the product of all local derivatives along the path.
Concrete Example
Given: h = x·w₁, ŷ = h·w₂, L = (ŷ−target)²
Decomposition: L(ŷ) = (ŷ−target)² (Loss function) , ŷ(h) = h·w₂ (Output function) , h(w₁) = x·w₁ (Hidden layer)
Try it yourself: the Backprop Bench
Enough theory — get hands-on. Drag the weights, send the loss backward through the network one step at a time, and watch gradient descent shrink the error. This network has a ReLU activation: push it below zero and you will see a dead neuron in action.
Autograd — Why You'll Never Do It by Hand
In practice, you never write backpropagation manually. Modern frameworks like PyTorch, JAX and TensorFlow include autograd (that is PyTorch's name for it; TensorFlow calls its tool GradientTape) — an automatic differentiation system that handles the entire backward pass for you. Since March 2026, TensorFlow has only been maintained (security and bug fixes); for new work in generative AI, Google itself recommends Keras 3, JAX or PyTorch.
Note
PyTorch: A Complete Training Step
for epoch in range(100):
prediction = net(X_train)
loss = criterion(prediction, y_train)
optimizer.zero_grad() # Clear old gradients
loss.backward() # Backpropagation in ONE line
optimizer.step() # Gradient descent updateHow does it work? While you code the forward pass, the framework silently builds a dynamic computational graph in the background. Every mathematical operation is recorded. When you call .backward(), the framework automatically traverses this graph in reverse, applying the chain rule at every node. This works for any differentiable computation — not just neural networks.
Yet autograd doesn't mean you can ignore the math. Three concrete scenarios where manual understanding is indispensable: (1) Debugging: When your loss plateaus, you need to diagnose vanishing gradients. (2) Architecture design: Choosing activation functions and skip connections depends on understanding gradient flow. (3) Custom layers: Writing custom autograd functions requires correct derivatives.
Caution
Misconception: Autograd Makes Math Understanding Obsolete
No! Autograd is the calculator, but you need to understand arithmetic to judge whether the calculator's result makes sense. Those who don't understand backpropagation are flying blind — they won't notice why their model isn't learning.
Dynamic vs. static graphs: PyTorch builds the computational graph fresh on every forward pass (dynamic). This allows Python control flow (if/else, loops) directly in the model. Old TensorFlow (v1) built the graph once before execution (static). Tape-based recording: Autograd "records" operations onto a tape. .backward() replays the tape in reverse. The .grad attribute: After .backward(), tensor.grad contains the computed gradient for leaf tensors such as the weights. requires_grad=True activates recording for a tensor. optimizer.zero_grad(): Gradients accumulate by default — without explicit clearing, gradients add up across iterations.
Note
Key Takeaways
- A training step is always Forward Pass → Backward Pass → Update. Backpropagation is the backward pass.
- Backpropagation = chain rule on the computational graph. Each node computes one local derivative; the product of all local derivatives along a path gives the gradient for a weight.
- Efficiency through reuse: every partial derivative is computed exactly once and cached.
- Deep networks risk vanishing or exploding gradients. Modern solutions: ReLU, residual connections, gradient clipping.
- Autograd automates gradient computation, but understanding the math remains essential for debugging and architecture design.
Check
Learning Goals
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: Backpropagation
What are the three phases of a complete training step in the correct order?
Select one answer
1. What are the three phases of a complete training step in the correct order?
2. In the mini-network example (x=2, w₁=0.5, w₂=3.0, y_true=5), the gradient for w₁ is −24. Why is this gradient much larger than the gradient for w₂ (−4)?
3. A neural network with 10 layers uses the sigmoid activation function (max derivative ≈0.25), and all weights have a magnitude below 1. At most, how large is the gradient signal that reaches layer 1 from layer 10, relative to the signal at layer 10?
4. A colleague claims: "Since PyTorch computes all gradients automatically with loss.backward(), understanding the chain rule is no longer necessary for deep learning practitioners." Which scenario directly disproves this claim?
5. A ReLU neuron receives input z = −3 during the forward pass. During the backward pass, the error signal arriving from the next layer is δ = 5.0. What gradient does this neuron pass backwards to its input weights?
6. A colleague proposes expanding a sigmoid-based network that already suffers from slow training to over a hundred layers. Evaluate whether this would improve or worsen the situation, and choose the best architectural alternative for such a deep network.