Measuring Change: Derivatives

When a neural network trains, it adjusts millions of weights — but how does it know WHICH direction to adjust them? The answer: It computes the derivative of the loss function. The derivative measures how much the error changes when a weight changes by a tiny amount.

Positive derivative: error increases — decrease the weight. Negative derivative: error decreases — increase the weight. This simple sign-check, repeated billions of times, is how AI learns.

Slope: From Average to Instantaneous

Let us see how the limit works. For f(x) = x\u00B2 at x = 3: the exact derivative is f'(3) = 6. Watch how the approximation converges:

def f(x):
    return x ** 2

x = 3
for h in [1, 0.1, 0.01, 0.001, 0.0001]:
    slope = (f(x + h) - f(x)) / h
    print(f"h = {h:8.4f}  ->  slope = {slope:.4f}")

# h =   1.0000  ->  slope = 7.0000
# h =   0.1000  ->  slope = 6.1000
# h =   0.0100  ->  slope = 6.0100
# h =   0.0010  ->  slope = 6.0010
# h =   0.0001  ->  slope = 6.0001  (converges to 6)

The loss function L depends on weights w. The derivative dL/dw tells you: "If I increase this weight by a tiny amount, does the error go up or down — and by how much?" This is exactly the information gradient descent uses.

Differentiation Rules: The Toolkit

Instead of computing the limit by hand every time, four rules cover almost all AI-relevant derivatives. Like recipes: you grab ready-made flour instead of grinding grain.

Example: Loss Function L(w) = (w-3)\u00B2 + 1

Expanded: L(w) = w\u00B2 - 6w + 10. Apply rules: (w\u00B2)' = 2w (power), (-6w)' = -6 (scaling), (10)' = 0 (constant). Sum rule: L'(w) = 2w - 6.

def L(w):
    return (w - 3) ** 2 + 1

def L_prime(w):
    return 2 * w - 6

for w in [0, 1, 2, 3, 4, 5]:
    print(f"w={w}  L(w)={L(w):5.1f}  L'(w)={L_prime(w):+.1f}")

# w=0  L(w)= 10.0  L'(w)=-6.0   (falling -> increase w)
# w=1  L(w)=  5.0  L'(w)=-4.0   (falling -> increase w)
# w=2  L(w)=  2.0  L'(w)=-2.0   (falling -> increase w)
# w=3  L(w)=  1.0  L'(w)=+0.0   (flat -> minimum!)
# w=4  L(w)=  2.0  L'(w)=+2.0   (rising -> decrease w)
# w=5  L(w)=  5.0  L'(w)=+4.0   (rising -> decrease w)

import torch

# Weight with gradient tracking
w = torch.tensor([5.0], requires_grad=True)

# Compute loss
loss = (w - 3) ** 2 + 1  # L(5) = 5

# Compute gradient automatically
loss.backward()
print(w.grad)  # tensor([4.]) -> L'(5) = 4

Interactive: Compare Function Growth

The power rule reveals: the higher the exponent, the faster the function grows. Move the slider and observe how f(x) = x² and f(x) = x³ pull away from f(x) = x as x increases. The derivative measures exactly this difference in growth — f'(x) = 2x for x² grows twice as fast as f'(x) = 1 for x.

Finding Minima: Why Derivatives Power AI

You now know how to compute derivatives and what their sign means. The decisive question: where is the error smallest?

Preview: Gradient Descent

The simplest form of training: start at a random weight, compute the derivative, take a small step in the opposite direction. Repeat.

# Simplest Gradient Descent
def L(w):
    return (w - 3) ** 2 + 1

def L_prime(w):
    return 2 * w - 6

w = 0.0
lr = 0.1  # learning rate

for step in range(20):
    grad = L_prime(w)
    w = w - lr * grad
    if step % 5 == 0:
        print(f"Step {step:2d}: w={w:.4f}  L={L(w):.4f}  L'={grad:+.4f}")

# Step  0: w=0.6000  L=6.7600  L'=-6.0000
# Step  5: w=2.8024  L=1.0390  L'=-0.3277
# Step 10: w=2.9893  L=1.0001  L'=-0.0178
# Step 15: w=2.9994  L=1.0000  L'=-0.0010
# -> w converges to 3.0 (the true minimum)

Training a neural network means: minimizing the loss function by adjusting weights. The derivative of loss with respect to each weight points toward the minimum. Gradient descent formalizes this: w_new = w_old - learning_rate \u00D7 gradient. The next articles (Gradient, Chain Rule) complete the picture for real networks.