The Chain Rule: Differentiating Nested Functions

You can differentiate x² and compute gradients. But a neural network does not compute x² — it computes f₃(f₂(f₁(x))), functions inside functions inside functions, dozens or hundreds of layers deep. How do you find the derivative of that entire tower?

The chain rule answers this question. It decomposes the derivative of a nested function into a product of individual derivatives — and this is the mathematical engine that powers every neural network ever trained.

Function Composition: Functions as Building Blocks

Let us see how composition works. h(x) = (3x + 1)² consists of two functions: g(x) = 3x + 1 (inner) and f(u) = u² (outer). For x = 2: g(2) = 7, f(7) = 49.

# Function composition: h(x) = (3x + 1)²
def g(x):
    return 3 * x + 1  # Inner function

def f(u):
    return u ** 2      # Outer function

def h(x):
    return f(g(x))     # Composition

x = 2
print(f"g({x}) = {g(x)}")     # g(2) = 7
print(f"f(g({x})) = {h(x)}")  # f(7) = 49

A 100-layer network is a 100-fold function composition. This is not a metaphor — it is the literal mathematical structure. Each layer transforms its input and passes the output onward.

The Chain Rule: Derivative of Nested Functions

Why multiplication, not addition? The sum rule adds derivatives: (f + g)' = f' + g'. The chain rule multiplies them. Intuition: If you stretch a rubber band 3× and then stretch the result 2×, the total stretch is 3 × 2 = 6×, not 3 + 2 = 5×.

Worked Example: h(x) = (3x + 1)²

Decompose: g(x) = 3x + 1 (inner), f(u) = u² (outer). Derivatives: f'(u) = 2u, g'(x) = 3. Chain rule: h'(x) = 2(3x + 1) · 3 = 6(3x + 1). At x = 2: h'(2) = 6 · 7 = 42. Numerical check: (h(2.0001) - h(2)) / 0.0001 ≈ 42.0006 ✔

def h(x):
    return (3 * x + 1) ** 2

def h_prime(x):
    return 6 * (3 * x + 1)  # Chain rule

x = 2
print(f"Chain rule: h'({x}) = {h_prime(x)}")  # 42

eps = 0.0001
numerical = (h(x + eps) - h(x)) / eps
print(f"Numerical:  h'({x}) = {numerical:.4f}")  # 42.0006

Computing ∂Loss/∂w for a weight in layer 1 of an L-layer network requires applying the chain rule through ALL L layers — and that IS backpropagation.

Interactive: Explore the Chain Rule Formula

Click on each term of the chain rule formula to understand its meaning. Switch between standard and Leibniz notation and follow step by step how the rule is applied to a concrete example.

The Computational Graph: From Formula to Framework

A computational graph decomposes a complex function into elementary operations. Nodes are operations (+, ×, ReLU, ...), edges transport values. Forward: values flow from input to output. Backward: derivatives flow back — the chain rule in graph form.

Backward Pass: Chain Rule Node by Node

Here a new symbol appears: ∂ (pronounced "partial"). It works like d in derivatives, but is used when a function depends on multiple variables. ∂Loss/∂w asks: How does the loss change when only w changes, while everything else stays fixed?

Now backward: ∂Loss/∂z₄ = 2z₄ = 3.8. ∂z₄/∂z₃ = -1. ∂z₃/∂z₂ = 1 (since z₂ > 0). ∂z₂/∂w = x = 2. Chain rule: ∂Loss/∂w = 3.8 · (-1) · 1 · 2 = -7.6. This is exactly what PyTorch computes automatically.

import torch

x = torch.tensor(2.0)
w = torch.tensor(0.5, requires_grad=True)
b = torch.tensor(0.1, requires_grad=True)
y_true = torch.tensor(3.0)

z = torch.relu(w * x + b)
loss = (y_true - z) ** 2

loss.backward()  # Chain rule automatically!
print(f"∂Loss/∂w = {w.grad:.4f}")  # -7.6000
print(f"∂Loss/∂b = {b.grad:.4f}")  # -3.8000

import torch

# 3 nested operations
w = torch.tensor(2.0, requires_grad=True)
z1 = w ** 2      # z1 = 4
z2 = z1 * 3       # z2 = 12
z3 = z2 + 1       # z3 = 13

z3.backward()
print(f"dz3/dw = {w.grad}")  # 12.0
# Chain rule: dz3/dz2=1, dz2/dz1=3, dz1/dw=2w=4
# Total: 1 * 3 * 4 = 12