Partial Derivatives & The Gradient

The previous article gave you a superpower: the derivative tells you whether nudging ONE weight makes the error go up or down. But a real neural network has millions of weights — not just one. How do you figure out which of those million knobs to turn, and in which direction?

You need two new tools: Partial derivatives isolate the effect of each weight independently. The gradient bundles all of this information into a single vector — a compass that points toward the steepest slope. Following this compass in reverse is the engine behind every AI training step on the planet.

The Multidimensional Landscape — From Curve to Terrain

In the previous article, the loss function L(w) was a curve — one dimension, two directions: forward or backward. Now let us raise the scope: what happens when the loss depends on two, three, or millions of variables?

Consider the simple bowl function f(x, y) = x² + y²:

At point (3, 4): f = 9 + 16 = 25. Slope in x-direction: ∂f/∂x = 2·3 = 6. Slope in y-direction: ∂f/∂y = 2·4 = 8. Two numbers instead of one — that is the jump from 1D to 2D. At real scale: GPT-4 reportedly has ~1.8 trillion parameters (Mixture-of-Experts, 16 expert networks of ~111 billion parameters each, 2 active per forward pass). Its loss landscape lives in a space with 1.8 trillion dimensions.

Every combination of weights (w₁, ..., wₙ) is a point in the loss landscape. Training means finding a deep valley in this landscape. The partial derivatives reveal the steepness in each direction — the gradient combines them into a single compass.

Partial Derivatives — One Variable at a Time

You know how to differentiate a function of one variable. The new skill: differentiating a function of many variables by freezing all the others.

Example: Linear Model with Two Weights

Prediction = w₁·x + w₂, with training point (x=2, y_true=7), current weights w₁=1, w₂=3. Prediction: 1·2 + 3 = 5. Loss: (5-7)² = 4. Partial derivative ∂L/∂w₁ = 2·(5-7)·2 = -8 (strongly negative: increasing w₁ decreases the loss significantly). ∂L/∂w₂ = 2·(5-7) = -4 (less negative: increasing w₂ helps less). Why the factor x=2 in ∂L/∂w₁? The loss depends on the prediction, and the prediction depends on w₁ through the product w₁·x. Every change in w₁ is amplified by x=2 — that is why x appears as a factor. (This nesting has a name — the chain rule — which the next article covers.)

Why does w₁ have more leverage? Because it is multiplied by the input x=2. A weight multiplied by a larger input value has more influence on the prediction — and thus on the loss.

During training, the partial derivative of the loss with respect to each weight tells the optimizer exactly how sensitive the loss is to that specific weight. Weights with large partial derivatives get larger updates; weights with near-zero partial derivatives barely change. This is how the network learns which connections matter most.

The Gradient — Your Compass in the Fog

You now have individual partial derivatives. The next step: assemble them all into a single vector — the gradient.

Gradient Descent in Action

For f(x, y) = x² + y², starting at (3, 4) with learning rate η = 0.1:

def f(x, y):
    return x**2 + y**2

def grad_f(x, y):
    return (2*x, 2*y)

x, y = 3.0, 4.0
lr = 0.1

for step in range(20):
    gx, gy = grad_f(x, y)
    x = x - lr * gx
    y = y - lr * gy
    if step % 5 == 0:
        print(f"Step {step:2d}: ({x:.2f}, {y:.2f})  f={f(x,y):.2f}")

# Step  0: (2.40, 3.20)  f=16.00
# Step  5: (0.79, 1.05)  f=1.73
# Step 10: (0.26, 0.35)  f=0.19
# Step 15: (0.08, 0.11)  f=0.02
# -> converges to (0, 0), the minimum

Cauchy described the steepest descent method in 1847 for astronomical calculations (Comptes rendus, vol. 25, pp. 536-538). In 1986, Rumelhart, Hinton and Williams showed how to compute the gradient of the loss through a multi-layer neural network efficiently — Backpropagation (Nature, vol. 323, pp. 533-536). Today, every training step of every neural network computes the gradient and follows it downhill.

Interactive: Explore Partial Derivatives

Click on individual terms to see what each part means. Switch between tabs to compare the derivative with respect to x and y — and follow the worked example step by step to see how the gradient emerges.