Measuring Change: Derivatives

When a neural network trains, it adjusts millions of weights — but how does it know WHICH direction to adjust them? The answer: It computes the derivative of the loss function. The derivative measures how much the error changes when a weight changes by a tiny amount.

Positive derivative: error increases — decrease the weight. Negative derivative: error decreases — increase the weight. This simple sign-check, repeated billions of times, is how AI learns.

Slope: From Average to Instantaneous

Let us see how the limit works. For f(x) = x² at x = 3: the exact derivative is f'(3) = 6. Watch how the approximation converges:

def f(x):
    return x ** 2

x = 3
for h in [1, 0.1, 0.01, 0.001, 0.0001]:
    slope = (f(x + h) - f(x)) / h
    print(f"h = {h:8.4f}  ->  slope = {slope:.4f}")

# h =   1.0000  ->  slope = 7.0000
# h =   0.1000  ->  slope = 6.1000
# h =   0.0100  ->  slope = 6.0100
# h =   0.0010  ->  slope = 6.0010
# h =   0.0001  ->  slope = 6.0001  (converges to 6)

The loss function L depends on weights w. The derivative dL/dw tells you: "If I increase this weight by a tiny amount, does the error go up or down — and by how much?" This is exactly the information gradient descent uses.

Interactive: From secant to tangent

Try the limit yourself. The slider moves the point P along the parabola f(x) = x²; the tangent shows the slope exactly at P. The dashed secant through P and Q measures the average slope over the distance h. Make h smaller: at x = 3 the secant slope drops from 7 via 6.1 to 6.001 and approaches f'(3) = 6. And find the spot where the tangent lies flat.

Differentiation Rules: The Toolkit

Instead of computing the limit by hand every time, there are four rules that let you differentiate any polynomial — and with it many simple loss functions. Like recipes: you grab ready-made flour instead of grinding grain.

Example: Loss Function L(w) = (w-3)² + 1

Expanded: L(w) = w² - 6w + 10. Apply rules: (w²)' = 2w (power), (-6w)' = -6 (scaling), (10)' = 0 (constant). Sum rule: L'(w) = 2w - 6.

def L(w):
    return (w - 3) ** 2 + 1

def L_prime(w):
    return 2 * w - 6

for w in [0, 1, 2, 3, 4, 5]:
    print(f"w={w}  L(w)={L(w):5.1f}  L'(w)={L_prime(w):+.1f}")

# w=0  L(w)= 10.0  L'(w)=-6.0   (falling -> increase w)
# w=1  L(w)=  5.0  L'(w)=-4.0   (falling -> increase w)
# w=2  L(w)=  2.0  L'(w)=-2.0   (falling -> increase w)
# w=3  L(w)=  1.0  L'(w)=+0.0   (flat -> minimum!)
# w=4  L(w)=  2.0  L'(w)=+2.0   (rising -> decrease w)
# w=5  L(w)=  5.0  L'(w)=+4.0   (rising -> decrease w)

import torch

# Weight with gradient tracking
w = torch.tensor([5.0], requires_grad=True)

# Compute loss
loss = (w - 3) ** 2 + 1  # L(5) = 5

# Compute gradient automatically
loss.backward()
print(w.grad)  # tensor([4.]) -> L'(5) = 4

Finding Minima: Why Derivatives Power AI

You now know how to compute derivatives and what their sign means. The decisive question: where is the error smallest?

Preview: Gradient Descent

The simplest form of training: start at a random weight, compute the derivative, take a small step in the opposite direction. Repeat.

# Simplest Gradient Descent
def L(w):
    return (w - 3) ** 2 + 1

def L_prime(w):
    return 2 * w - 6

w = 0.0
lr = 0.1  # learning rate

for step in range(20):
    grad = L_prime(w)
    w = w - lr * grad
    if step % 5 == 0:
        print(f"Step {step:2d}: w={w:.4f}  L={L(w):.4f}  L'(w_old)={grad:+.4f}")

# Step  0: w=0.6000  L=6.7600  L'(w_old)=-6.0000
# Step  5: w=2.2136  L=1.6185  L'(w_old)=-1.9661
# Step 10: w=2.7423  L=1.0664  L'(w_old)=-0.6442
# Step 15: w=2.9156  L=1.0071  L'(w_old)=-0.2111
# -> w converges to 3.0 (the true minimum)

Training a neural network means: minimizing the loss function by adjusting weights. The derivative of loss with respect to each weight shows which direction leads to the minimum. Gradient descent formalizes this: w_new = w_old - learning_rate × gradient. The next articles (Gradient, Chain Rule) complete the picture for real networks.