Gradient Descent

How a model learns — step by step down the mountain

TipClick the map — the optimizer starts immediately. Try all the loss functions and compare SGD vs Adam!

Loss landscape (3D)

ready
Loss history
Click the map to start recording.
Controls
Run
Aha presets
One click sets the learning rate, resets and starts. Watch the difference.
Step0Loss1.428Best loss—Distance to min1.69|Gradient|2.020Statusready
Loss functionPicks the loss landscape the optimizer rolls over. Each shape poses a different challenge — from the simple bowl to the zigzag-inducing banana.
Two valleys — the starting point decides

Non-convex: two equally deep minima left and right of center, with a 'hill' between them. The optimizer blindly rolls into the nearest valley — it doesn't know which one is the global minimum. Real neural networks have billions of such valleys side by side; that is why different training runs of the same network end at slightly different solutions.

OptimizerSets the update rule for every step. SGD follows the gradient blindly, Momentum builds up inertia, Adam scales each axis individually.
SGDMomentumAdam
SGD — Stochastic Gradient Descent

The basic form: θ = θ − η·g. Follows the current gradient blindly, no memory, no inertia. The textbook standard and reference point — surprisingly good on smooth landscapes, zigzags heavily in flat valleys (banana). Rarely used as a default today because complex landscapes stall it.

Parameters
Scales the step size. Too small crawls forever, too large overshoots the valley and explodes — the single most important knob in any training run.
Logarithmic scale 0.001 to 3.0. Try the extremes!
LR schedule Lets η shrink over time instead of staying constant. In real trainings this stabilizes convergence toward the end.ConstantStep decayCosineReal trainings shrink η over time. Step decay halves every 40 steps, cosine decays smoothly toward zero.
Controls only the animation playback tempo, not the math. Faster shows more steps per second but does not change the path.
Escape the valley
Stuck in a local minimum? Kick just nudges, Restart rolls the dice, Best-of-8 explores systematically.

What you see here is model training as a picture: the colored map is the loss landscape (the hills and valleys of a model's error), the white path is the optimizer rolling into a valley. Pick a starting point, switch the loss function, switch the optimizer — even a different starting point or a different optimizer can lead to a very different outcome. That is the core of every modern AI training run.

What you see on the error landscape

This demo shows how a model learns: a point rolls down an error landscape into the valley. Here is what the picture shows.

What you see
A hilly, colored surface stands for the model's error — high and bright means large error, deep valleys mean small error. A marked point sits somewhere on the slope, and a star marks the deepest valley, the best result.
What happens
Step by step the point rolls downhill toward the nearest valley, leaving a trail behind it. On a steep slope it takes big steps; close to the valley floor it barely moves — then it has arrived at the bottom.
What you can do
Switch between the 1D, 2D and 3D view, pick an error landscape and an optimizer, and drag the learning-rate slider. The buttons start, pause or reset the run.
What to watch for
The learning rate is the step size. Too large and the point jumps over the valley or rocks back and forth; too small and it only creeps forward. This is how an AI model feels its way down the slope until it barely improves anymore.

Definition

What is gradient descent?

Analogy

Imagine you are standing in thick fog on a mountain and want to reach the valley. You can't see a thing — but you can feel with your feet where the slope drops most steeply.

You take a small step in that direction. Then you feel again, and repeat. Eventually the ground is flat — you reached the valley. That is exactly how a model learns.

Every iteration of a model — from a tiny regressor to a billion-parameter network — follows the same recipe of three building blocks:

  1. Loss — the model's map The loss function L(θ) measures how bad the model currently is with its parameters θ. In the demo it is the colorful map: deep blue = low loss (good), red = high loss (bad). Learning means: find a low point on this map.
  2. Gradient — the arrow uphill At every point, the gradient ∇L points in the direction of steepest ascent. But we want to go downhill — so we take the negative and step in that direction. In 1D mode the gradient is shown as a yellow tangent on the parabola.
  3. Learning rate η — how big each step is The learning rate scales each step. Too small → the model crawls forever. Too large → the step overshoots the valley and the path oscillates or even explodes. The Aha presets in the demo show both extremes.

Repeat these three building blocks a few thousand to a few million times and a trained model emerges from random initial values. That is essentially all that "learning" in a neural network really is — the magic is just that the loss landscape is high-dimensional and the gradient can still be computed automatically via backpropagation.

Three optimizers, same starting point, same learning rate, same loss landscape. The only thing that differs is the update rule each optimizer uses to decide where the next step goes. Click 'Start race' and watch the three paths run in parallel.

What is this experiment about?

We control everything except the optimizer — like a clean scientific experiment with only one variable. If the three paths still look different, that difference is caused entirely by the update rule. This makes visible what is otherwise an abstract detail in a training configuration.

What to watch for
  • SGD: follows the current gradient blindly. In flat or curved valleys it oscillates — stepping left-right across the valley instead of along it.
  • Momentum: stores past steps as 'inertia'. It takes a few steps for the flywheel to spin up — after that, it accelerates consistently in one direction.
  • Adam: scales each axis individually — directions where the gradient is noisy get damped, calm directions get amplified. On the banana it acts like a homing missile.
SGD
L: 0.522·Best loss: —
Momentum
L: 0.522·Best loss: —
Adam
L: 0.522·Best loss: —
Why the results differ

All three see the same gradient at the same point — but they process it differently. SGD uses it as-is. Momentum averages it with past gradients (β=0.9), so consistent directions get reinforced while back-and-forth swings cancel out. Adam does the same as Momentum, plus it normalizes each component by the recent variance of the gradient — noisy axes get damped, calm axes get boosted. These math details decide whether the path zigzags, flows smoothly, or overshoots the valley.

What this means for real training

On screen it looks like three lines on a map — in real training it is hours of GPU time, several percent accuracy, and sometimes the difference between model works and model fails to converge. That is why Adam (and its variants AdamW, Adafactor, Lion …) is the default for deep neural networks: it is robust to wildly different gradient scales across millions of parameters — exactly the situation in real-world networks. The demo shows in miniature what at full scale decides whether a training run succeeds.

TheoryPseudocodeStep by stepFlow diagram

The wanderer in the fog

Imagine you are standing in thick fog on a mountain and want to reach the valley. You can't see a thing — but you can feel with your feet where the slope drops most steeply, and you step in that direction. Repeat until it's flat. That is exactly what gradient descent does. It is the learning rule behind almost every trained AI model of the last 30 years.

The math in one line

Formally: $\theta_{t+1} = \theta_t - \eta \cdot \nabla L(\theta_t)$. In words: take the current parameters $\theta$, compute the gradient $\nabla L$ of the loss function at this point, and step a distance $\eta$ (learning rate) against the gradient — because the gradient points uphill, and we want to go downhill.

Three building blocks

  • Loss function L(θ): the "map" — how bad is the model with parameters θ?
  • Gradient ∇L: the arrow that points uphill at every point. We take its negative.
  • Learning rate η: the step size. Too small = painfully slow. Too large = oscillates or explodes.

SGD, Momentum, Adam — three flavors

Plain gradient descent (SGD) follows the gradient blindly. Momentum accumulates past steps like a rolling ball with inertia — it helps in flat, long valleys. Adam combines momentum with a per-parameter scaling based on gradient variance — today's default for deep networks.

Where it doesn't always work

  • Local minima: for non-convex functions (multiple valleys) the algorithm gets stuck in the nearest valley — the starting point decides.
  • Saddle points: flat regions where the gradient vanishes without being a real minimum.
  • Explosion: with a way-too-large learning rate the step overshoots and the path flies apart.

Play in the demo! Try the three loss functions (bowl = convex, two valleys = local minima, banana = flat valley) and see where SGD fails and Adam shines. The Aha presets ("Too slow", "Just right", "Explode!") show the learning-rate effect in a single click.

Check

Test your knowledge

Not completed

What does the gradient of a loss function describe?

Select one answer

Answer Key: 1) B · 2) C · 3) B · 4) C

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Demo: find the way to the valley

What does the demo show?

This demo shows gradient descent. Gradient descent means: go downhill step by step. You see a colorful landscape. Blue is low. Red is high. The goal: a ball rolls into the deepest valley. This is how an AI learns.

How you use the demo

Click on the landscape. The ball starts there. The ball rolls into the valley. You can switch between 1D, 2D and 3D. You can also change the learning rate. Learning rate means: how big is each step? Try the Aha buttons.

What you learn

You learn: a learning rate that is too big is bad. The ball jumps out of the valley. A learning rate that is too small is also bad. The ball never arrives. There are also many valleys. One valley is the deepest. This is called the global optimum.