Back to Learning Area

The Heart of Learning

How a neural network sends its errors home — and makes everything along the way a little better.

  • Fundamentals
  • 12 min
  • Expert

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: How neural networks learn

What is backpropagation?

A neural network makes predictions. A prediction is an answer from the computer. Sometimes the answer is wrong. The network sends the error back. This is called backpropagation.

Learning in three steps

The network learns in three steps. Step 1: It makes a prediction. This is the forward pass. Step 2: It sends the error back. This is backpropagation. Step 3: It adjusts the numbers. This is an update.

Errors can vanish

Deep networks have many levels. A level is called a layer. The network passes the error from layer to layer. The error becomes smaller. At some point, it is too small. The network stops learning. This is called vanishing gradient.

You know about neurons, activation functions, layers, loss functions, and gradient descent. You know the network needs gradients to improve. But how does it compute the gradient for a weight buried 20 layers deep? The answer connects everything you've learned into one elegant mechanism — and it's built entirely on the chain rule from the “Linear Algebra & Calculus” learning path.

Backpropagation is not new mathematics. It is the systematic application of the chain rule to a neural network's computational graph — a bookkeeping method that makes gradient descent practical for deep networks.

The Complete Training Step

  1. 1. Forward PassInput flows forward through all layers, producing a prediction and a loss value
  2. 2. Backward PassThe error signal flows backwards, the chain rule yields the gradient for every weight
  3. 3. UpdateGradient descent adjusts each weight: w = w − lr · gradient

Definition

Backpropagation

Analogy

Imagine a relay race in reverse. The forward pass is like runners carrying the baton from start to finish. The backward pass is like a coach walking backwards along the course, telling each runner exactly how to adjust their stride — each runner only needs the feedback from the next runner and their own local performance. The update is each runner actually making the adjustment.

Example

In a relay race, each runner physically adjusts their stride. In mathematics, the backward pass doesn't directly change the weights — it only computes gradients. The actual adjustment happens in the separate update step via gradient descent.

A training step consists of three inseparable phases: forward pass (compute prediction), backward pass (compute gradients), and update (adjust weights). Backpropagation is the backward pass — nothing more, nothing less.

Worked Example: A 2-Weight Mini-Network

Input x = 2, weights w₁ = 0.5 and w₂ = 3.0, target y_true = 5. The network computes: h = x · w₁, then y = h · w₂. Loss: L = (y − y_true)².

h = x · w₁ = 2 · 0.5 = 1   |   y = h · w₂ = 1 · 3.0 = 3   |   L = (y − yₜ)² = (3 − 5)² = 4
Forward Pass
∂L/∂y = 2(y − yₜ) = −4   |   ∂y/∂w₂ = h = 1  →  grad(w₂) = −4
Backward Pass: Gradient for w₂
∂y/∂h = w₂ = 3   |   ∂h/∂w₁ = x = 2  →  grad(w₁) = −4 · 3 · 2 = −24
Backward Pass: Gradient for w₁ (Chain Rule!)
w₂ = 3.0 − 0.01 · (−4) = 3.04   |   w₁ = 0.5 − 0.01 · (−24) = 0.74
Update (Learning Rate = 0.01)

The network has learned: both weights were adjusted so the prediction will be closer to the target next time. The gradient for w₁ is six times larger than for w₂ — because the chain rule multiplies in the additional factors w₂ and x.

Caution

Misconception: Backpropagation Is a Separate Learning Algorithm

No! Backpropagation only computes gradients — by systematically applying the chain rule to the computational graph. The actual learning algorithm is gradient descent (or a variant like Adam), which uses these gradients to update the weights. Backpropagation without gradient descent learns nothing; gradient descent without backpropagation cannot efficiently compute the gradients.

The forward pass computes the prediction, the backward pass computes the gradients, the update step computes the new weights. This trio often repeats thousands of times during training — and that is exactly how a neural network learns.

Interactive: Forward & Backward Pass

You've worked through the training step on the mini-network. Now you can walk through it step by step: watch how values flow forward through the network, how gradients are computed backwards, and how the weights change at the end.

w₁=0.5w₂=3target=5x2h?ŷ?L?
Step 1 / 8Setup

The mini-network has input x, two weights w₁ and w₂, and a target value. No computations have been performed yet.

x=2, w₁=0.5, w₂=3, target=5
Controls
Steps

The Chain Rule in Action — Layer by Layer Backwards

From 2 weights to N layers: the principle stays identical, only the chain gets longer. For a weight in the first layer of a deep network, the chain rule must be applied through ALL layers.

∂L/∂w₁ = ∂L/∂a₃ · ∂a₃/∂z₃ · ∂z₃/∂a₂ · ∂a₂/∂z₂ · ∂z₂/∂a₁ · ∂a₁/∂z₁ · ∂z₁/∂w₁
Chain rule for w₁ in a 3-layer network

Backpropagation's efficiency comes from reuse: once the error signal for layer 2 is computed, it's stored. Layer 1 simply multiplies by its local derivative — no recomputation from the loss. Every partial derivative is computed exactly once and cached.

Sigmoid

Maximum derivative ≈0.25. With 10 layers (and weights with a magnitude below 1) at most 0.25¹⁰ ≈ 0.00000095. The gradient vanishes — early layers barely learn anymore.

ReLU

Derivative is 0 or 1. For positive inputs, the gradient flows through unattenuated — no exponential decay.

Caution

Dead Neuron: When the Gradient Vanishes Forever

Concrete example: x = 1, w₁ = −2. Forward pass: z₁ = 1 · (−2) = −2. ReLU(−2) = 0. The local derivative of ReLU at −2 is exactly 0. No matter how large the error at the end of the network — as soon as the error signal flows through this ReLU node, it is multiplied by 0. The gradient for w₁ becomes 0, the weight is never updated. If z₁ stays negative for all training inputs, the neuron is permanently dead.

Caution

Misconception: Vanishing Gradients Mean the Network Has a Bug

No! Vanishing gradients are a mathematical inevitability when multiplying many small numbers. With 10 sigmoid layers, the gradient is reduced to less than one millionth — that's not a code error, it's mathematics. The solution is architectural (ReLU instead of sigmoid, residual connections, batch normalization), not debugging.

ReLU Family: ReLU has a derivative of 1 for positive values and passes gradients through unfiltered. Leaky ReLU and ELU also prevent dead neurons by allowing small gradients for negative inputs. Residual Connections: ResNet skip connections add a layer's input directly to its output, creating shortcut paths for gradient flow — even through hundreds of layers. Batch Normalization: Normalizes each layer's activations, stabilizing gradient distributions and accelerating training. Gradient Clipping: Caps gradients exceeding a threshold, preventing numerical instability from exploding gradients.

Interactive: Clickable Chain Rule

The chain rule decomposes the gradient into local derivatives. Click on the individual terms to see what each factor in the chain means — and how they combine to produce the gradient for w₁ in the mini network.

Chain rule for w₁

∂L∂w₁∂L∂ŷ∂ŷ∂h∂h∂w₁
∂L/∂w₁

The total gradient: How much does the loss change when w₁ changes? This is the product of all local derivatives along the path.

Concrete Example

Given: h = x·w₁, ŷ = h·w₂, L = (ŷ−target)²

Decomposition: L(ŷ) = (ŷ−target)² (Loss function) , ŷ(h) = h·w₂ (Output function) , h(w₁) = x·w₁ (Hidden layer)

1Derivative of loss: ∂L/∂ŷ = 2(ŷ−target) = −4
2Derivative through w₂: ∂ŷ/∂h = w₂ = 3
3Derivative w.r.t. w₁: ∂h/∂w₁ = x = 2
4Product of all factors: ∂L/∂w₁ = (−4)·3·2 = −24

Try it yourself: the Backprop Bench

Enough theory — get hands-on. Drag the weights, send the loss backward through the network one step at a time, and watch gradient descent shrink the error. This network has a ReLU activation: push it below zero and you will see a dead neuron in action.

Autograd — Why You'll Never Do It by Hand

In practice, you never write backpropagation manually. Modern frameworks like PyTorch, JAX and TensorFlow include autograd (that is PyTorch's name for it; TensorFlow calls its tool GradientTape) — an automatic differentiation system that handles the entire backward pass for you. Since March 2026, TensorFlow has only been maintained (security and bug fixes); for new work in generative AI, Google itself recommends Keras 3, JAX or PyTorch.

Note

PyTorch: A Complete Training Step

for epoch in range(100):
    prediction = net(X_train)
    loss = criterion(prediction, y_train)
    optimizer.zero_grad()   # Clear old gradients
    loss.backward()         # Backpropagation in ONE line
    optimizer.step()        # Gradient descent update

How does it work? While you code the forward pass, the framework silently builds a dynamic computational graph in the background. Every mathematical operation is recorded. When you call .backward(), the framework automatically traverses this graph in reverse, applying the chain rule at every node. This works for any differentiable computation — not just neural networks.

Yet autograd doesn't mean you can ignore the math. Three concrete scenarios where manual understanding is indispensable: (1) Debugging: When your loss plateaus, you need to diagnose vanishing gradients. (2) Architecture design: Choosing activation functions and skip connections depends on understanding gradient flow. (3) Custom layers: Writing custom autograd functions requires correct derivatives.

Caution

Misconception: Autograd Makes Math Understanding Obsolete

No! Autograd is the calculator, but you need to understand arithmetic to judge whether the calculator's result makes sense. Those who don't understand backpropagation are flying blind — they won't notice why their model isn't learning.

Dynamic vs. static graphs: PyTorch builds the computational graph fresh on every forward pass (dynamic). This allows Python control flow (if/else, loops) directly in the model. Old TensorFlow (v1) built the graph once before execution (static). Tape-based recording: Autograd "records" operations onto a tape. .backward() replays the tape in reverse. The .grad attribute: After .backward(), tensor.grad contains the computed gradient for leaf tensors such as the weights. requires_grad=True activates recording for a tensor. optimizer.zero_grad(): Gradients accumulate by default — without explicit clearing, gradients add up across iterations.

Note

Key Takeaways

  • A training step is always Forward Pass → Backward Pass → Update. Backpropagation is the backward pass.
  • Backpropagation = chain rule on the computational graph. Each node computes one local derivative; the product of all local derivatives along a path gives the gradient for a weight.
  • Efficiency through reuse: every partial derivative is computed exactly once and cached.
  • Deep networks risk vanishing or exploding gradients. Modern solutions: ReLU, residual connections, gradient clipping.
  • Autograd automates gradient computation, but understanding the math remains essential for debugging and architecture design.

Check

Learning Goals

Can you explain this in your own words? Tick what you've got.

0 of 3 done

Check

Quiz: Backpropagation

Not completed

What are the three phases of a complete training step in the correct order?

Select one answer

Answer Key: 1) B · 2) B · 3) B · 4) C · 5) C · 6) C