Measuring Change: Derivatives
Derivatives in differential calculus, explained for people who survived mathematics.
When a neural network trains, it adjusts millions of weights — but how does it know WHICH direction to adjust them? The answer: It computes the derivative of the loss function. The derivative measures how much the error changes when a weight changes by a tiny amount.
Positive derivative: error increases — decrease the weight. Negative derivative: error decreases — increase the weight. This simple sign-check, repeated billions of times, is how AI learns.
Slope: From Average to Instantaneous
Definition
Derivative
Analogy
Example
Definition
Let us see how the limit works. For f(x) = x² at x = 3: the exact derivative is f'(3) = 6. Watch how the approximation converges:
- 1. h = 1[f(4) - f(3)] / 1 = [16 - 9] / 1 = 7.0
- 2. h = 0.1[f(3.1) - f(3)] / 0.1 = [9.61 - 9] / 0.1 = 6.1
- 3. h = 0.01[f(3.01) - f(3)] / 0.01 = [9.0601 - 9] / 0.01 = 6.01
- 4. h → 0Limit = 6.0 — the exact derivative f'(3) = 2·3 = 6
def f(x):
return x ** 2
x = 3
for h in [1, 0.1, 0.01, 0.001, 0.0001]:
slope = (f(x + h) - f(x)) / h
print(f"h = {h:8.4f} -> slope = {slope:.4f}")
# h = 1.0000 -> slope = 7.0000
# h = 0.1000 -> slope = 6.1000
# h = 0.0100 -> slope = 6.0100
# h = 0.0010 -> slope = 6.0010
# h = 0.0001 -> slope = 6.0001 (converges to 6)Caution
Misconception: The Derivative IS the Slope
Almost: The derivative is a FUNCTION f'(x) that gives the slope at every point. At a specific point x = 3, f'(3) = 6 is the slope there. The derivative maps every input to its corresponding slope.
The loss function L depends on weights w. The derivative dL/dw tells you: "If I increase this weight by a tiny amount, does the error go up or down — and by how much?" This is exactly the information gradient descent uses.
Interactive: From secant to tangent
Try the limit yourself. The slider moves the point P along the parabola f(x) = x²; the tangent shows the slope exactly at P. The dashed secant through P and Q measures the average slope over the distance h. Make h smaller: at x = 3 the secant slope drops from 7 via 6.1 to 6.001 and approaches f'(3) = 6. And find the spot where the tangent lies flat.
Differentiation Rules: The Toolkit
Instead of computing the limit by hand every time, there are four rules that let you differentiate any polynomial — and with it many simple loss functions. Like recipes: you grab ready-made flour instead of grinding grain.
Example: Loss Function L(w) = (w-3)² + 1
Expanded: L(w) = w² - 6w + 10. Apply rules: (w²)' = 2w (power), (-6w)' = -6 (scaling), (10)' = 0 (constant). Sum rule: L'(w) = 2w - 6.
def L(w):
return (w - 3) ** 2 + 1
def L_prime(w):
return 2 * w - 6
for w in [0, 1, 2, 3, 4, 5]:
print(f"w={w} L(w)={L(w):5.1f} L'(w)={L_prime(w):+.1f}")
# w=0 L(w)= 10.0 L'(w)=-6.0 (falling -> increase w)
# w=1 L(w)= 5.0 L'(w)=-4.0 (falling -> increase w)
# w=2 L(w)= 2.0 L'(w)=-2.0 (falling -> increase w)
# w=3 L(w)= 1.0 L'(w)=+0.0 (flat -> minimum!)
# w=4 L(w)= 2.0 L'(w)=+2.0 (rising -> decrease w)
# w=5 L(w)= 5.0 L'(w)=+4.0 (rising -> decrease w)Loss increases with larger w
Loss decreases with larger w
L'(5) = +4: Loss rises steeply
+4
L'(3) = 0: Minimum reached
0
L'(1) = -4: Loss falls steeply
-4
The derivative has many notations: f'(x) (Lagrange), dy/dx or df/dx (Leibniz), ∂f/∂x (partial derivative). All mean the same thing for functions of one variable. You will need the ∂ notation starting with the next article (Partial Derivatives), when functions depend on multiple variables.
PyTorch and TensorFlow compute all derivatives automatically via Autograd. You define a function, call .backward() (PyTorch; TensorFlow uses tf.GradientTape and has been in maintenance mode since March 2026), and the framework delivers the gradients. Yet understanding the concept is essential: when training does not converge, you need to know WHY the gradient is too large, too small, or zero.
import torch
# Weight with gradient tracking
w = torch.tensor([5.0], requires_grad=True)
# Compute loss
loss = (w - 3) ** 2 + 1 # L(5) = 5
# Compute gradient automatically
loss.backward()
print(w.grad) # tensor([4.]) -> L'(5) = 4Finding Minima: Why Derivatives Power AI
You now know how to compute derivatives and what their sign means. The decisive question: where is the error smallest?
Definition
Function Minimum
Analogy
Example
Definition
Caution
Misconception: f'(x) = 0 Always Means Minimum
Not necessarily. f'(x) = 0 only means the tangent is horizontal — the terrain is flat here. It could be a valley (minimum), a hilltop (maximum), or a saddle point. In deep neural networks, saddle points are more common than true local minima — they can slow down or stall training.
Preview: Gradient Descent
The simplest form of training: start at a random weight, compute the derivative, take a small step in the opposite direction. Repeat.
# Simplest Gradient Descent
def L(w):
return (w - 3) ** 2 + 1
def L_prime(w):
return 2 * w - 6
w = 0.0
lr = 0.1 # learning rate
for step in range(20):
grad = L_prime(w)
w = w - lr * grad
if step % 5 == 0:
print(f"Step {step:2d}: w={w:.4f} L={L(w):.4f} L'(w_old)={grad:+.4f}")
# Step 0: w=0.6000 L=6.7600 L'(w_old)=-6.0000
# Step 5: w=2.2136 L=1.6185 L'(w_old)=-1.9661
# Step 10: w=2.7423 L=1.0664 L'(w_old)=-0.6442
# Step 15: w=2.9156 L=1.0071 L'(w_old)=-0.2111
# -> w converges to 3.0 (the true minimum)Training a neural network means: minimizing the loss function by adjusting weights. The derivative of loss with respect to each weight shows which direction leads to the minimum. Gradient descent formalizes this: w_new = w_old - learning_rate × gradient. The next articles (Gradient, Chain Rule) complete the picture for real networks.
Note
Key Takeaways
- The derivative f'(x) gives the instantaneous rate of change — how steeply the function rises or falls at one specific point. The limit definition (h → 0) turns average slope into exact slope.
- Four simple rules (power, constant, sum, scaling) are enough to differentiate any polynomial. In AI, the derivative of the loss function tells you whether to increase or decrease a weight.
- Setting f'(x) = 0 finds candidates for extrema — but not every zero of f' is a minimum. It could also be a maximum or a saddle point. Real neural network loss landscapes have many local minima and saddle points — not just one clean valley.
Check
Learning Goals
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: Derivatives
What does the sign of the derivative L'(w) tell you about the loss function?
Select one answer
1. What does the sign of the derivative L'(w) tell you about the loss function?
2. You are training a network. At the current weight w, L'(w) = -3. What should gradient descent do?
3. Given f(x) = x² - 6x + 10. Apply the differentiation rules. What is f'(x)?
4. Gradient descent lands at a point where L'(w) = 0, but the loss is not the smallest possible value. What happened?