Gradient Descent

How a model learns — step by step down the mountain

Imagine you are standing in thick fog on a mountain and want to reach the valley. You can't see a thing — but you can feel with your feet where the slope drops most steeply.

You take a small step in that direction. Then you feel again, and repeat. Eventually the ground is flat — you reached the valley. That is exactly how a model learns.

Analogy:

Imagine you are standing in thick fog on a mountain and want to reach the valley. You can't see a thing — but you can feel with your feet where the slope drops most steeply.

You take a small step in that direction. Then you feel again, and repeat. Eventually the ground is flat — you reached the valley. That is exactly how a model learns.

Definition:

Gradient descent is an iterative optimization method. The gradient ∇L points in the direction of steepest ascent of the loss function; we step in the opposite direction with step size η (learning rate).

θ_{t+1} = θ_t − η · ∇L(θ_t)

Every iteration of a model — from a tiny regressor to a billion-parameter network — follows the same recipe of three building blocks:

Loss — the model's mapThe loss function L(θ) measures how bad the model currently is with its parameters θ. In the demo it is the colorful map: deep blue = low loss (good), red = high loss (bad). Learning means: find a low point on this map.
Gradient — the arrow uphillAt every point, the gradient ∇L points in the direction of steepest ascent. But we want to go downhill — so we take the negative and step in that direction. In 1D mode the gradient is shown as a yellow tangent on the parabola.
Learning rate η — how big each step isThe learning rate scales each step. Too small → the model crawls forever. Too large → the step overshoots the valley and the path oscillates or even explodes. The Aha presets in the demo show both extremes.

Repeat these three building blocks a few thousand to a few million times and a trained model emerges from random initial values. That is essentially all that "learning" in a neural network really is — the magic is just that the loss landscape is high-dimensional and the gradient can still be computed automatically via backpropagation.

What you see here is model training as a picture: the colored map is the loss landscape (the hills and valleys of a model's error), the white path is the optimizer rolling into a valley. Pick a starting point, switch the loss function, switch the optimizer — even a different starting point or a different optimizer can lead to a very different outcome. That is the core of every modern AI training run.

What you see on the error landscape

This demo shows how a model learns: a point rolls down an error landscape into the valley. Here is what the picture shows.

Loss landscape (3D)

ready

Step0

Loss0.000

Best loss—

Distance to min1.69

|Gradient|0.000

Status ready

Loss history

Click the map to start recording.

Loss function

Two valleys — the starting point decides

Non-convex: two equally deep minima left and right of center, with a 'hill' between them. The optimizer blindly rolls into the nearest valley — it doesn't know which one is the global minimum. Real neural networks have billions of such valleys side by side; that is why different training runs of the same network end at slightly different solutions.

Optimizer

SGD

Momentum

Adam

SGD — Stochastic Gradient Descent

The basic form: θ = θ − η·g. Follows the current gradient blindly, no memory, no inertia. The textbook standard and reference point — surprisingly good on smooth landscapes, zigzags heavily in flat valleys (banana). Rarely used as a default today because complex landscapes stall it.

Learning rate η: 0.064

Logarithmic scale 0.001 to 3.0. Try the extremes!

LR schedule

Constant

Step decay

Cosine

Real trainings shrink η over time. Step decay halves every 40 steps, cosine decays smoothly toward zero.

Speed: 8×

Aha presets

One click sets the learning rate, resets and starts. Watch the difference.

Escape the valley

Stuck in a local minimum? Kick just nudges, Restart rolls the dice, Best-of-8 explores systematically.

Three optimizers, same starting point, same learning rate, same loss landscape. The only thing that differs is the update rule each optimizer uses to decide where the next step goes. Click 'Start race' and watch the three paths run in parallel.

What is this experiment about?

We control everything except the optimizer — like a clean scientific experiment with only one variable. If the three paths still look different, that difference is caused entirely by the update rule. This makes visible what is otherwise an abstract detail in a training configuration.

What to watch for

SGD: follows the current gradient blindly. In flat or curved valleys it oscillates — stepping left-right across the valley instead of along it.
Momentum: stores past steps as 'inertia'. It takes a few steps for the flywheel to spin up — after that, it accelerates consistently in one direction.
Adam: scales each axis individually — directions where the gradient is noisy get damped, calm directions get amplified. On the banana it acts like a homing missile.

SGD

L: 0.522·Best loss: —

Momentum

L: 0.522·Best loss: —

Adam

L: 0.522·Best loss: —

Why the results differ

All three see the same gradient at the same point — but they process it differently. SGD uses it as-is. Momentum averages it with past gradients (β=0.9), so consistent directions get reinforced while back-and-forth swings cancel out. Adam does the same as Momentum, plus it normalizes each component by the recent variance of the gradient — noisy axes get damped, calm axes get boosted. These math details decide whether the path zigzags, flows smoothly, or overshoots the valley.

What this means for real training

On screen it looks like three lines on a map — in real training it is hours of GPU time, several percent accuracy, and sometimes the difference between model works and model fails to converge. That is why Adam (and its variants AdamW, Adafactor, Lion …) is the default for deep neural networks: it is robust to wildly different gradient scales across millions of parameters — exactly the situation in real-world networks. The demo shows in miniature what at full scale decides whether a training run succeeds.

Theory

Pseudocode

Step by step

Flow diagram

The wanderer in the fog

Imagine you are standing in thick fog on a mountain and want to reach the valley. You can't see a thing — but you can feel with your feet where the slope drops most steeply, and you step in that direction. Repeat until it's flat. That is exactly what gradient descent does. It is the learning rule behind almost every trained AI model of the last 30 years.

The math in one line

Formally: $\theta_{t+1} = \theta_t - \eta \cdot \nabla L(\theta_t)$. In words: take the current parameters $\theta$, compute the gradient $\nabla L$ of the loss function at this point, and step a distance $\eta$ (learning rate) against the gradient — because the gradient points uphill, and we want to go downhill.

Three building blocks

Loss function L(θ): the "map" — how bad is the model with parameters θ?
Gradient ∇L: the arrow that points uphill at every point. We take its negative.
Learning rate η: the step size. Too small = painfully slow. Too large = oscillates or explodes.

SGD, Momentum, Adam — three flavors

Plain gradient descent (SGD) follows the gradient blindly. Momentum accumulates past steps like a rolling ball with inertia — it helps in flat, long valleys. Adam combines momentum with a per-parameter scaling based on gradient variance — today's default for deep networks.

Where it doesn't always work

Local minima: for non-convex functions (multiple valleys) the algorithm gets stuck in the nearest valley — the starting point decides.
Saddle points: flat regions where the gradient vanishes without being a real minimum.
Explosion: with a way-too-large learning rate the step overshoots and the path flies apart.

Play in the demo! Try the three loss functions (bowl = convex, two valleys = local minima, banana = flat valley) and see where SGD fails and Adam shines. The Aha presets ("Too slow", "Just right", "Explode!") show the learning-rate effect in a single click.

English

                      
                        
                          1
                          # Gradient Descent — iterative optimization
                        
                      
                        
                          2
                          function gradient_descent(L, theta_init, lr, max_steps, tol):
                        
                      
                        
                          3
                              theta = theta_init
                        
                      
                        
                          4
                              m, v = 0, 0   # optimizer state (for Momentum/Adam)
                        
                      
                        
                          5
                           
                        
                      
                        
                          6
                              for step in 1..max_steps:
                        
                      
                        
                          7
                           
                        
                      
                        
                          8
                                  # 1. Gradient — direction of steepest ascent
                        
                      
                        
                          9
                                  g = gradient(L, theta)
                        
                      
                        
                          10
                           
                        
                      
                        
                          11
                                  # 2. Flat enough? Then we're in the valley
                        
                      
                        
                          12
                                  if |g| < tol: break
                        
                      
                        
                          13
                           
                        
                      
                        
                          14
                                  # 3. The optimizer turns the gradient into a step
                        
                      
                        
                          15
                                  if optimizer == "Momentum":
                        
                      
                        
                          16
                                      m = 0.9·m + g                  # build inertia
                        
                      
                        
                          17
                                      step = lr · m
                        
                      
                        
                          18
                                  elif optimizer == "Adam":
                        
                      
                        
                          19
                                      m = 0.9·m + 0.1·g              # 1st moment
                        
                      
                        
                          20
                                      v = 0.999·v + 0.001·g²         # 2nd moment (variance)
                        
                      
                        
                          21
                                      step = lr · m / (√v + ε)
                        
                      
                        
                          22
                                  else:                             # plain SGD
                        
                      
                        
                          23
                                      step = lr · g
                        
                      
                        
                          24
                           
                        
                      
                        
                          25
                                  # 4. Update parameters — one step downhill
                        
                      
                        
                          26
                                  theta = theta - step
                        
                      
                        
                          27
                           
                        
                      
                        
                          28
                              return theta

1 / 6

🚩 Initialization

Set the parameters θ to a starting value (in real networks: random). Set the optimizer helper variables (momentum buffer m, variance buffer v for Adam) to zero. This is your starting position on the loss landscape.

function gradient_descent(L, theta_init, lr, max_steps, tol):
    theta = theta_init
    m, v = 0, 0   # optimizer state (for Momentum/Adam)

What does the gradient of a loss function describe?

The current value of the loss function.

The direction of steepest ascent of the loss.

The optimal learning rate for the next step.

The number of iterations already performed.

1. What does the gradient of a loss function describe?

☐ A) The current value of the loss function.
☐ B) The direction of steepest ascent of the loss.
☐ C) The optimal learning rate for the next step.
☐ D) The number of iterations already performed.

2. What typically happens with a much-too-large learning rate?

☐ A) The model converges especially quickly to the global minimum.
☐ B) The loss decreases slowly but steadily.
☐ C) The optimizer overshoots the minimum, oscillates, or even diverges.
☐ D) The gradient is automatically scaled down.

3. Why does momentum help especially in long, flat valleys (like the Rosenbrock banana)?

☐ A) It increases the learning rate dynamically.
☐ B) It accumulates past gradients and keeps the direction, so it stops zig-zagging.
☐ C) It replaces the gradient by the mean of all data points.
☐ D) It automatically switches between SGD and Adam.

4. Does gradient descent guarantee finding the global minimum?

☐ A) Yes, always — as long as the learning rate is small enough.
☐ B) Yes, but only with the Adam optimizer.
☐ C) No. For non-convex functions (e.g. with multiple valleys) it gets stuck in the nearest local minimum.
☐ D) No, it always only finds saddle points.

Answer Key: 1) B · 2) C · 3) B · 4) C

Gradient Descent

What is gradient descent?

Analogy:

Definition:

Three building blocks, one learning step

Interactive demo

What you see on the error landscape

Loss landscape (3D)

Race mode — three optimizers head-to-head