The Chain Rule: Differentiating Nested Functions
A high school math rule that made the AI revolution possible in the first place.
You can differentiate x² and compute gradients. But a neural network does not compute x² — it computes f₃(f₂(f₁(x))), functions inside functions inside functions, dozens or hundreds of layers deep. How do you find the derivative of that entire tower?
The chain rule answers this question. It decomposes the derivative of a nested function into a product of individual derivatives — and this is the mathematical engine that powers practically every neural network trained today.
Function Composition: Functions as Building Blocks
Definition
Function Composition
Analogy
Example
Definition
Let us see how composition works. h(x) = (3x + 1)² consists of two functions: g(x) = 3x + 1 (inner) and f(u) = u² (outer). For x = 2: g(2) = 7, f(7) = 49.
# Function composition: h(x) = (3x + 1)²
def g(x):
return 3 * x + 1 # Inner function
def f(u):
return u ** 2 # Outer function
def h(x):
return f(g(x)) # Composition
x = 2
print(f"g({x}) = {g(x)}") # g(2) = 7
print(f"f(g({x})) = {h(x)}") # f(7) = 49Caution
Misconception: The Order Does Not Matter
It does! f(g(x)) is generally different from g(f(x)). The inner function is ALWAYS evaluated first. Example: (3·2 + 1)² = 49, but 3·(2²) + 1 = 13. The order of composition changes the result.
A 100-layer network is a 100-fold function composition. This is not a metaphor — it is the literal mathematical structure. Each layer transforms its input and passes the output onward.
The Chain Rule: Derivative of Nested Functions
Definition
The Chain Rule
Analogy
Example
Definition
Why multiplication, not addition? The sum rule adds derivatives: (f + g)' = f' + g'. The chain rule multiplies them. Intuition: If you stretch a rubber band 3× and then stretch the result 2×, the total stretch is 3 × 2 = 6×, not 3 + 2 = 5×.
Stage 1: 3 times as fast. Stage 2: 2 times as fast. Total: 3 × 2 = 6 times as fast. Ratios multiply.
g'(x) = 3. f'(u) = 2u. At x = 2, u = 7: f'(7) = 14. Total: h'(2) = 14 × 3 = 42. Derivatives multiply.
Worked Example: h(x) = (3x + 1)²
Decompose: g(x) = 3x + 1 (inner), f(u) = u² (outer). Derivatives: f'(u) = 2u, g'(x) = 3. Chain rule: h'(x) = 2(3x + 1) · 3 = 6(3x + 1). At x = 2: h'(2) = 6 · 7 = 42. Numerical check: (h(2.0001) - h(2)) / 0.0001 ≈ 42.0009 ✔
def h(x):
return (3 * x + 1) ** 2
def h_prime(x):
return 6 * (3 * x + 1) # Chain rule
x = 2
print(f"Chain rule: h'({x}) = {h_prime(x)}") # 42
eps = 0.0001
numerical = (h(x + eps) - h(x)) / eps
print(f"Numerical: h'({x}) = {numerical:.4f}") # 42.0009Caution
Misconception: The Chain Rule Adds Derivatives
No! The chain rule MULTIPLIES. The sum rule adds: (f + g)' = f' + g'. The chain rule multiplies: (f ∘ g)' = (f' ∘ g) · g'. If you stretch a rubber band 3× and then 2×, the total stretch is 6× (not 5×). That is exactly how rates of change work.
Computing ∂Loss/∂w for a weight in layer 1 of an L-layer network requires applying the chain rule through ALL L layers — and that IS backpropagation.
Interactive: Explore the Chain Rule Formula
Click on each term of the chain rule formula to understand its meaning. Switch between standard and Leibniz notation and follow step by step how the rule is applied to a concrete example.
Chain Rule — Standard Form
The derivative of the composite function h(x) = f(g(x)). This is the value we're looking for: How quickly does h change when x changes? The chain rule shows how to calculate this result from the derivatives of the individual parts.
Concrete Example
Given: h(x) = (3x + 1)²
Decomposition: f(u) = u² (outer function) , u = g(x) = 3x + 1 (inner function)
The Computational Graph: From Formula to Framework
A computational graph decomposes a complex function into elementary operations. Nodes are operations (+, ×, ReLU, ...), edges transport values. Forward: values flow from input to output. Backward: derivatives flow back — the chain rule in graph form.
- 1. Input × Weightz₁ = w · x = 0.5 · 2 = 1.0
- 2. Add Biasz₂ = z₁ + b = 1.0 + 0.1 = 1.1
- 3. ReLU Activationz₃ = ReLU(z₂) = ReLU(1.1) = 1.1
- 4. Compute Errorz₄ = y_true - z₃ = 3.0 - 1.1 = 1.9
- 5. LossLoss = z₄² = 1.9² = 3.61
Backward Pass: Chain Rule Node by Node
As a reminder: ∂ (“partial d”, often pronounced just “partial”). It works like d in derivatives, but is used when a function depends on multiple variables. ∂Loss/∂w asks: How does the loss change when only w changes, while everything else stays fixed?
Now backward: ∂Loss/∂z₄ = 2z₄ = 3.8. ∂z₄/∂z₃ = -1. ∂z₃/∂z₂ = 1 (since z₂ > 0). ∂z₂/∂z₁ = 1, ∂z₁/∂w = x = 2. Chain rule: ∂Loss/∂w = 3.8 · (-1) · 1 · 1 · 2 = -7.6. This is exactly what PyTorch computes automatically.
import torch
x = torch.tensor(2.0)
w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(0.1, requires_grad=True)
y_true = torch.tensor(3.0)
z = torch.relu(w * x + b)
loss = (y_true - z) ** 2
loss.backward() # Chain rule automatically!
print(f"∂Loss/∂w = {w.grad:.4f}") # -7.6000
print(f"∂Loss/∂b = {b.grad:.4f}") # -3.8000Caution
Misconception: Backpropagation is a Separate Algorithm
No! Backpropagation IS the chain rule applied systematically to a computational graph. The efficiency comes from dynamic programming (reusing intermediate results), not from new mathematics. loss.backward() in PyTorch applies the chain rule backward through the graph.
PyTorch builds a dynamic computational graph during the forward pass. Every tensor computed from tensors with requires_grad=True remembers (in grad_fn) which operation created it. backward() traverses the graph in reverse and computes gradients via the chain rule. You never need to compute derivatives by hand — but understanding is essential when gradients vanish or explode.
import torch
# 3 nested operations
w = torch.tensor(2.0, requires_grad=True)
z1 = w ** 2 # z1 = 4
z2 = z1 * 3 # z2 = 12
z3 = z2 + 1 # z3 = 13
z3.backward()
print(f"dz3/dw = {w.grad}") # 12.0
# Chain rule: dz3/dz2=1, dz2/dz1=3, dz1/dw=2w=4
# Total: 1 * 3 * 4 = 12The chain rule multiplies local derivatives. In deep networks, this product determines success or failure. Sigmoid has a maximum derivative of 0.25. With 10 layers: 0.25¹⁰ ≈ 0.000001 — one millionth. With weights of magnitude below 1, the gradient in layer 1 is thus at least about a million times smaller than at the output. The early layers learn practically nothing — this is the vanishing gradient problem (Hochreiter, 1991).
Sigmoid: gradient after 10 layers at most ≈ 0.000001
0.25 ¹⁰
ReLU: derivative 1 for positive inputs (no vanishing)
1 ¹⁰
ResNet layers: skip connections preserve gradients
152
Solutions: ReLU activation (derivative = 1 for positive inputs), residual connections (ResNet, He et al. 2015), and normalization. These techniques enable training networks with hundreds of layers — because they keep the chain rule product under control.
Note
Key Takeaways
- A neural network with L layers is an L-fold function composition — each layer's output becomes the next layer's input.
- The chain rule gives the derivative of nested functions by multiplying local derivatives: h'(x) = f'(g(x)) · g'(x). This is exactly how backpropagation computes gradients.
- Frameworks like PyTorch automate the chain rule via computational graphs: the forward pass stores values, the backward pass multiplies derivatives node by node.
Check
Learning Goals
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: The Chain Rule
What does the chain rule compute?
Select one answer
1. What does the chain rule compute?
2. h(x) = (5x - 2)³. What is h'(x)?
3. A 20-layer network uses sigmoid (max derivative 0.25). If all weights have a magnitude below 1, how large is the gradient product from layer 20 to layer 1 at most?
4. A developer finds that weights in early layers of a deep network barely change during training, while later layers train normally. Which explanation best fits?