Back to Learning Area

Measuring Change: Derivatives

Derivatives in differential calculus, explained for people who survived mathematics.

  • Fundamentals
  • 13 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Derivatives and AI learning

What is a derivative?

A derivative measures the slope of a curve. Slope means: how much does a value change? A speedometer shows the current speed. This is a derivative. It measures the change of position.

How AI uses derivatives

AI learns by adjusting weights. Weights are numbers in an AI network. The derivative shows the direction. If the derivative is positive, the weight becomes smaller. If it is negative, the weight becomes larger. This is how the network finds the smallest error.

Gradient descent

Gradient descent is a learning method. Gradient descent means: always go towards the smallest error. The model calculates the derivative. Then it adjusts the weights. This is how the error becomes smaller and smaller.

When a neural network trains, it adjusts millions of weights — but how does it know WHICH direction to adjust them? The answer: It computes the derivative of the loss function. The derivative measures how much the error changes when a weight changes by a tiny amount.

Positive derivative: error increases — decrease the weight. Negative derivative: error decreases — increase the weight. This simple sign-check, repeated billions of times, is how AI learns.

Slope: From Average to Instantaneous

Definition

Derivative

Analogy

Your car drives through mountains. The average speed over 100 km (100 km / 2 h = 50 km/h) is the average slope of the distance-time curve — the secant. The speedometer shows the speed RIGHT NOW — the tangent slope, the derivative.

Example

A real speedometer measures over a small but finite time window (milliseconds). The mathematical derivative idealizes this: h goes to exactly zero — physically impossible, but mathematically precise.

Let us see how the limit works. For f(x) = x² at x = 3: the exact derivative is f'(3) = 6. Watch how the approximation converges:

  1. 1. h = 1[f(4) - f(3)] / 1 = [16 - 9] / 1 = 7.0
  2. 2. h = 0.1[f(3.1) - f(3)] / 0.1 = [9.61 - 9] / 0.1 = 6.1
  3. 3. h = 0.01[f(3.01) - f(3)] / 0.01 = [9.0601 - 9] / 0.01 = 6.01
  4. 4. h → 0Limit = 6.0 — the exact derivative f'(3) = 2·3 = 6
def f(x):
    return x ** 2

x = 3
for h in [1, 0.1, 0.01, 0.001, 0.0001]:
    slope = (f(x + h) - f(x)) / h
    print(f"h = {h:8.4f}  ->  slope = {slope:.4f}")

# h =   1.0000  ->  slope = 7.0000
# h =   0.1000  ->  slope = 6.1000
# h =   0.0100  ->  slope = 6.0100
# h =   0.0010  ->  slope = 6.0010
# h =   0.0001  ->  slope = 6.0001  (converges to 6)

Caution

Misconception: The Derivative IS the Slope

Almost: The derivative is a FUNCTION f'(x) that gives the slope at every point. At a specific point x = 3, f'(3) = 6 is the slope there. The derivative maps every input to its corresponding slope.

The loss function L depends on weights w. The derivative dL/dw tells you: "If I increase this weight by a tiny amount, does the error go up or down — and by how much?" This is exactly the information gradient descent uses.

Interactive: From secant to tangent

Try the limit yourself. The slider moves the point P along the parabola f(x) = x²; the tangent shows the slope exactly at P. The dashed secant through P and Q measures the average slope over the distance h. Make h smaller: at x = 3 the secant slope drops from 7 via 6.1 to 6.001 and approaches f'(3) = 6. And find the spot where the tangent lies flat.

Differentiation Rules: The Toolkit

Instead of computing the limit by hand every time, there are four rules that let you differentiate any polynomial — and with it many simple loss functions. Like recipes: you grab ready-made flour instead of grinding grain.

(xⁿ)' = n · xⁿ⁻¹
Power Rule
c' = 0
Constant Rule
(f + g)' = f' + g'
Sum Rule
(c · f)' = c · f'
Scaling Rule

Example: Loss Function L(w) = (w-3)² + 1

Expanded: L(w) = w² - 6w + 10. Apply rules: (w²)' = 2w (power), (-6w)' = -6 (scaling), (10)' = 0 (constant). Sum rule: L'(w) = 2w - 6.

def L(w):
    return (w - 3) ** 2 + 1

def L_prime(w):
    return 2 * w - 6

for w in [0, 1, 2, 3, 4, 5]:
    print(f"w={w}  L(w)={L(w):5.1f}  L'(w)={L_prime(w):+.1f}")

# w=0  L(w)= 10.0  L'(w)=-6.0   (falling -> increase w)
# w=1  L(w)=  5.0  L'(w)=-4.0   (falling -> increase w)
# w=2  L(w)=  2.0  L'(w)=-2.0   (falling -> increase w)
# w=3  L(w)=  1.0  L'(w)=+0.0   (flat -> minimum!)
# w=4  L(w)=  2.0  L'(w)=+2.0   (rising -> decrease w)
# w=5  L(w)=  5.0  L'(w)=+4.0   (rising -> decrease w)
L'(w) > 0

Loss increases with larger w

L'(w) < 0

Loss decreases with larger w

L'(5) = +4: Loss rises steeply

+4

L'(3) = 0: Minimum reached

0

L'(1) = -4: Loss falls steeply

-4

The derivative has many notations: f'(x) (Lagrange), dy/dx or df/dx (Leibniz), ∂f/∂x (partial derivative). All mean the same thing for functions of one variable. You will need the ∂ notation starting with the next article (Partial Derivatives), when functions depend on multiple variables.

PyTorch and TensorFlow compute all derivatives automatically via Autograd. You define a function, call .backward() (PyTorch; TensorFlow uses tf.GradientTape and has been in maintenance mode since March 2026), and the framework delivers the gradients. Yet understanding the concept is essential: when training does not converge, you need to know WHY the gradient is too large, too small, or zero.

import torch

# Weight with gradient tracking
w = torch.tensor([5.0], requires_grad=True)

# Compute loss
loss = (w - 3) ** 2 + 1  # L(5) = 5

# Compute gradient automatically
loss.backward()
print(w.grad)  # tensor([4.]) -> L'(5) = 4

Finding Minima: Why Derivatives Power AI

You now know how to compute derivatives and what their sign means. The decisive question: where is the error smallest?

Definition

Function Minimum

Analogy

You stand on a hill in dense fog and want to reach the valley (minimum). You cannot see the valley, but you can feel which direction the ground slopes downward under your feet (= derivative). You walk in the direction of steepest descent. Step by step, you approach the valley.

Example

A real hill has maybe 3 valleys. A neural network's loss landscape has millions of dimensions with countless valleys of varying depth — unimaginably complex.

Caution

Misconception: f'(x) = 0 Always Means Minimum

Not necessarily. f'(x) = 0 only means the tangent is horizontal — the terrain is flat here. It could be a valley (minimum), a hilltop (maximum), or a saddle point. In deep neural networks, saddle points are more common than true local minima — they can slow down or stall training.

Preview: Gradient Descent

The simplest form of training: start at a random weight, compute the derivative, take a small step in the opposite direction. Repeat.

w_new = w_old - learning_rate × L'(w_old)
Gradient Descent Rule
# Simplest Gradient Descent
def L(w):
    return (w - 3) ** 2 + 1

def L_prime(w):
    return 2 * w - 6

w = 0.0
lr = 0.1  # learning rate

for step in range(20):
    grad = L_prime(w)
    w = w - lr * grad
    if step % 5 == 0:
        print(f"Step {step:2d}: w={w:.4f}  L={L(w):.4f}  L'(w_old)={grad:+.4f}")

# Step  0: w=0.6000  L=6.7600  L'(w_old)=-6.0000
# Step  5: w=2.2136  L=1.6185  L'(w_old)=-1.9661
# Step 10: w=2.7423  L=1.0664  L'(w_old)=-0.6442
# Step 15: w=2.9156  L=1.0071  L'(w_old)=-0.2111
# -> w converges to 3.0 (the true minimum)

Training a neural network means: minimizing the loss function by adjusting weights. The derivative of loss with respect to each weight shows which direction leads to the minimum. Gradient descent formalizes this: w_new = w_old - learning_rate × gradient. The next articles (Gradient, Chain Rule) complete the picture for real networks.

Note

Key Takeaways

  • The derivative f'(x) gives the instantaneous rate of change — how steeply the function rises or falls at one specific point. The limit definition (h → 0) turns average slope into exact slope.
  • Four simple rules (power, constant, sum, scaling) are enough to differentiate any polynomial. In AI, the derivative of the loss function tells you whether to increase or decrease a weight.
  • Setting f'(x) = 0 finds candidates for extrema — but not every zero of f' is a minimum. It could also be a maximum or a saddle point. Real neural network loss landscapes have many local minima and saddle points — not just one clean valley.

Check

Learning Goals

Can you explain this in your own words? Tick what you've got.

0 of 3 done

Check

Quiz: Derivatives

Not completed

What does the sign of the derivative L'(w) tell you about the loss function?

Select one answer

Answer Key: 1) B · 2) C · 3) B · 4) B