Back to Learning Area

The Path to the Valley: Gradient Descent

How gradient descent finds the lowest point in a landscape with millions of hills — most of the time.

  • Fundamentals
  • 10 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: The path to the best result

What is gradient descent?

A neural network is an AI program. It starts with random values. The computer must find better values. This is called gradient descent. Gradient descent means: walking downhill step by step.

The learning rate

The learning rate determines how big each step is. A learning rate that is too high is dangerous. The program jumps past the target. A learning rate that is too low makes training slow. You must choose the learning rate carefully.

Three variants

There are three variants. All data at once: this is accurate but slow. This is called batch. One example at a time: this is fast but inaccurate. This is called SGD. Small groups: this is the standard. This is called mini-batch.

Every neural network starts with random weights — essentially random guesses. The entire training process is about turning those guesses into useful values. But how? The answer is an algorithm so simple it can be explained in one sentence: compute which direction is downhill, take a step.

This article explains gradient descent — the engine behind virtually all modern AI training — and why the seemingly minor detail of step size can make or break your model.

Core Thesis

Gradient descent is an optimization algorithm that iteratively adjusts model parameters by moving in the direction opposite to the gradient of the loss function. The size of each step is controlled by the learning rate — often the single most important hyperparameter in deep learning. The algorithm's behavior in the complex, high-dimensional loss landscapes of real neural networks is governed by the interplay between gradient direction, step size, and noise.

The Algorithm — Downhill in the Fog

Definition

Gradient Descent

Analogy

Imagine you are lost on a mountainside in thick fog. You cannot see the valley, but you can feel which direction the ground drops most steeply under your feet. You take a step in that direction. Then you feel again. Step by step, you descend toward the valley. This is exactly what gradient descent does — it feels the slope of the loss function and steps downhill.

Limitation: On a real mountain, you can feel the slope perfectly. In high-dimensional parameter spaces, the gradient is computed from noisy data samples, making the measurement imprecise. Also, real mountains typically have one valley — loss landscapes can have many deceptive dips and saddle ridges.

The Update Rule

Note

The Update Rule

The entire algorithm is captured in a single equation:

w_new = w_old - learning_rate × gradient

Why the minus sign? Because the gradient points uphill (direction of steepest ascent). We want to go downhill (toward lower loss). The minus reverses the direction.

Worked Example: Step by Step

Consider the loss function L(w) = (w - 3)² + 1 with starting weight w = 0 and learning rate 0.1. The gradient here is the first derivative: dL/dw = 2(w - 3). The minimum is at w = 3 where L = 1.

  1. 1. Step 1: Gradient = 2(0 - 3) = -6. Update: w = 0 - 0.1 × (-6) = 0.6. Loss: 6.76
  2. 2. Step 2: Gradient = 2(0.6 - 3) = -4.8. Update: w = 0.6 - 0.1 × (-4.8) = 1.08. Loss: 4.69
  3. 3. Step 3: Gradient = 2(1.08 - 3) = -3.84. Update: w = 1.08 - 0.1 × (-3.84) = 1.464. Loss: 3.36
  4. 4. Step 4: Gradient = 2(1.464 - 3) = -3.07. Update: w = 1.464 - 0.1 × (-3.07) = 1.771. Loss: 2.51
  5. 5. Step 5: Gradient = 2(1.771 - 3) = -2.46. Update: w = 1.771 - 0.1 × (-2.46) = 2.017. Loss: 1.97

Three Variants of Gradient Descent

Batch Gradient Descent

Uses the ENTIRE dataset to compute the gradient at each step. Advantage: stable, exact gradient. Disadvantage: slow with large datasets, lingers at saddle points for a long time because the gradient approaches zero there.

Stochastic Gradient Descent (SGD)

Uses a SINGLE random sample per step. Advantage: fast, noisy updates help escape saddle points (more on this in the Loss Landscape section). Disadvantage: very noisy — the path to the minimum zigzags wildly.

In practice, Mini-Batch Gradient Descent is the golden middle and today's standard: it uses a small batch (typically 32-256 samples; large language models such as Llama 3 even process 4 to 16 million tokens per step) per step. This combines the stability of Batch GD with the speed and noise benefits of SGD.

Interactive: Gradient Descent Step by Step

Click through the five steps and watch the ball roll down the loss curve. Notice how the steps get smaller as the ball approaches the minimum — the gradient becomes flatter.

1
2
3
4
5
6
Parameter wLoss L(w)Minimum
Step 1 / 6Starting position

The ball starts at w = 0, left of the minimum at w = 3. The loss is high: L = (0 - 3)² + 1 = 10. The gradient there is -6, so downhill is to the right. The arrow always shows the next step.

Controls
Steps

The Dial — Learning Rate

The learning rate is a number set by the developer that controls how large each step is. It is often the single most important hyperparameter in deep learning. It is NOT learned by the algorithm — the developer must choose it.

Three Scenarios — Same Function, Different Step Sizes

Too Large (lr = 1.5)

w = 0 → 9.0 → -9.0 → ... The weights oscillate wildly and diverge. The loss explodes instead of decreasing. The model has learned nothing.

Too Small (lr = 0.01)

w = 0 → 0.06 → 0.12 → ... The weights barely move. After 100 steps, w has only reached about 2.6. Training takes far longer than necessary.

Note

Just Right (lr = 0.5)

With lr = 0.5: w = 0 - 0.5 × (-6) = 3.0 — the minimum is reached in a single step. In practice, a perfect learning rate is rare, but this shows how powerful the right choice can be.

Try it yourself: as in the worked example, the ball starts at w = 0 on the loss curve L(w) = (w − 3)² + 1. Set the learning rate lr or pick one of the three scenarios and watch whether the ball rolls into the valley, overshoots it or flies off.

Typical Mini-Batch Size

32

The standard number of training samples processed per gradient descent step.

Adam Default Learning Rate

0.001

The default learning rate of the Adam optimizer (Kingma & Ba, 2014).

Adam Optimizer Published

2014

Adam combines multiple optimization techniques and adapts the learning rate per parameter automatically (Kingma & Ba).

Caution

Common Misconception

"You always have to manually tune the learning rate." — Not entirely true. Learning rate schedules reduce the rate over time (large steps early, fine-tuning later); for large models it usually first rises during a short warm-up phase. The Adam optimizer (Kingma & Ba, 2014) adapts the learning rate per parameter automatically. However, the initial learning rate still matters — Adam simplifies but does not eliminate the decision.

The Landscape — Saddle Points, Local Minima, Plateaus

The loss function of a real neural network with millions of parameters creates a complex, high-dimensional landscape. This landscape contains three types of obstacles where the gradient approaches zero — but for very different reasons.

Landscape Topology

Local Minimum All dimensions curve upward — the gradient is zero and the loss is lower than the immediate surroundings. Far rarer than saddle points in high-dimensional spaces because ALL dimensions must simultaneously curve up.
Saddle Point Mixed curvature — some dimensions curve up, others curve down. Like a mountain pass: valley in front and behind, peaks to left and right. The gradient is zero but it is NOT a minimum. Far more common than local minima.
Plateau Nearly flat region where the gradient is close to zero. Training stagnates because the algorithm receives almost no signal about which direction to move.

Key insight: In high-dimensional spaces (millions of parameters), a true local minimum requires ALL dimensions to simultaneously curve upward. This is statistically extremely unlikely, unless the loss there is already almost as low as at the global minimum. So the local minima that do exist are usually almost as good as the global one. Saddle points, where some dimensions curve up and others down, are far more common — and practically more dangerous.

Caution

Common Misconception

"Gradient descent always finds the global minimum." — Wrong. This is only guaranteed for convex functions (functions with a single valley and no other dips) and a suitable learning rate. Neural networks have non-convex loss landscapes with many dips and saddle points. In practice, finding a "good enough" minimum is sufficient — and is usually what happens.

Caution

Common Misconception

"SGD is inferior to Batch GD because it is noisier." — The opposite is often true. At a saddle point, Batch GD computes the exact gradient, which is zero — so it can stay stuck there for an extremely long time. SGD computes a noisy approximation from a random sample — the gradient is never exactly zero at a saddle point. This nudges the model off the saddle and lets it find the downhill slope. The noise is a feature, not a bug.

Adam (Adaptive Moment Estimation) maintains two running exponential averages per parameter: the first moment (average of recent gradients) and the second moment (average of recent squared gradients). Parameters that consistently receive large gradients get a smaller effective learning rate — this keeps each step roughly the size of the learning rate. Parameters with small or sparse gradients get a larger effective learning rate — they need bigger nudges. Default hyperparameters: lr = 0.001, beta1 = 0.9, beta2 = 0.999. Large language models such as Llama 3 and DeepSeek-V3 mostly train with AdamW, a refinement of Adam. Since 2025, some large models such as Kimi K2 and DeepSeek-V4 have relied on the newer optimizer Muon.

Full computation for L(w) = (w - 3)² + 1, starting at w = 0, lr = 0.1: Step 1: w = 0.00, gradient = -6.00, new w = 0.60, loss = 6.76 Step 2: w = 0.60, gradient = -4.80, new w = 1.08, loss = 4.69 ... steps 3-8 follow the same pattern — the gradient shrinks with each step, progress slows down ... Step 9: w = 2.50, gradient = -1.01, new w = 2.60, loss = 1.16 Step 10: w = 2.60, gradient = -0.81, new w = 2.68, loss = 1.10 Notice the pattern: In the first two steps, w leaps from 0 to 1.08 — a huge jump. By the end, it barely moves from 2.60 to 2.68. The steps become tiny because the valley flattens out and the gradient shrinks near the minimum.

What Comes Next

Gradient descent tells us HOW to update weights — but for a network with many layers, we need to know how errors propagate backward through the network to compute the gradient for each weight. That is backpropagation — the subject of the next article, which builds on the chain rule.

Note

Takeaways

  1. Gradient descent adjusts weights by stepping opposite to the gradient — the minus sign in the update rule is the entire point.
  2. The learning rate is often the most impactful hyperparameter: too large causes divergence, too small causes stagnation, and adaptive methods like Adam help but don't eliminate the need for a good initial choice.
  3. In real neural networks, saddle points (not local minima) are the primary obstacle, and the noise in SGD is a feature that helps escape them.

Check

Knowledge Check: Gradient Descent

Not completed

In the gradient descent update rule w_new = w_old - lr × dL/dw, why is the gradient subtracted rather than added?

Select one answer

Answer Key: 1) B · 2) B · 3) B · 4) B · 5) C

Check

Self-Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done