The Path to the Valley: Gradient Descent
How gradient descent finds the lowest point in a landscape with millions of hills — most of the time.
Every neural network starts with random weights — essentially random guesses. The entire training process is about turning those guesses into useful values. But how? The answer is an algorithm so simple it can be explained in one sentence: compute which direction is downhill, take a step.
This article explains gradient descent — the engine behind virtually all modern AI training — and why the seemingly minor detail of step size can make or break your model.
Core Thesis
Gradient descent is an optimization algorithm that iteratively adjusts model parameters by moving in the direction opposite to the gradient of the loss function. The size of each step is controlled by the learning rate — often the single most important hyperparameter in deep learning. The algorithm's behavior in the complex, high-dimensional loss landscapes of real neural networks is governed by the interplay between gradient direction, step size, and noise.
The Algorithm — Downhill in the Fog
Definition
Gradient Descent
Analogy
Definition
Limitation: On a real mountain, you can feel the slope perfectly. In high-dimensional parameter spaces, the gradient is computed from noisy data samples, making the measurement imprecise. Also, real mountains typically have one valley — loss landscapes can have many deceptive dips and saddle ridges.
The Update Rule
Note
The Update Rule
The entire algorithm is captured in a single equation:
w_new = w_old - learning_rate × gradient
Why the minus sign? Because the gradient points uphill (direction of steepest ascent). We want to go downhill (toward lower loss). The minus reverses the direction.
Worked Example: Step by Step
Consider the loss function L(w) = (w - 3)² + 1 with starting weight w = 0 and learning rate 0.1. The gradient here is the first derivative: dL/dw = 2(w - 3). The minimum is at w = 3 where L = 1.
- 1. Step 1: Gradient = 2(0 - 3) = -6. Update: w = 0 - 0.1 × (-6) = 0.6. Loss: 6.76
- 2. Step 2: Gradient = 2(0.6 - 3) = -4.8. Update: w = 0.6 - 0.1 × (-4.8) = 1.08. Loss: 4.69
- 3. Step 3: Gradient = 2(1.08 - 3) = -3.84. Update: w = 1.08 - 0.1 × (-3.84) = 1.464. Loss: 3.36
- 4. Step 4: Gradient = 2(1.464 - 3) = -3.07. Update: w = 1.464 - 0.1 × (-3.07) = 1.771. Loss: 2.51
- 5. Step 5: Gradient = 2(1.771 - 3) = -2.46. Update: w = 1.771 - 0.1 × (-2.46) = 2.017. Loss: 1.97
Three Variants of Gradient Descent
Uses the ENTIRE dataset to compute the gradient at each step. Advantage: stable, exact gradient. Disadvantage: slow with large datasets, lingers at saddle points for a long time because the gradient approaches zero there.
Uses a SINGLE random sample per step. Advantage: fast, noisy updates help escape saddle points (more on this in the Loss Landscape section). Disadvantage: very noisy — the path to the minimum zigzags wildly.
In practice, Mini-Batch Gradient Descent is the golden middle and today's standard: it uses a small batch (typically 32-256 samples; large language models such as Llama 3 even process 4 to 16 million tokens per step) per step. This combines the stability of Batch GD with the speed and noise benefits of SGD.
Interactive: Gradient Descent Step by Step
Click through the five steps and watch the ball roll down the loss curve. Notice how the steps get smaller as the ball approaches the minimum — the gradient becomes flatter.
The ball starts at w = 0, left of the minimum at w = 3. The loss is high: L = (0 - 3)² + 1 = 10. The gradient there is -6, so downhill is to the right. The arrow always shows the next step.
The Dial — Learning Rate
The learning rate is a number set by the developer that controls how large each step is. It is often the single most important hyperparameter in deep learning. It is NOT learned by the algorithm — the developer must choose it.
Three Scenarios — Same Function, Different Step Sizes
w = 0 → 9.0 → -9.0 → ... The weights oscillate wildly and diverge. The loss explodes instead of decreasing. The model has learned nothing.
w = 0 → 0.06 → 0.12 → ... The weights barely move. After 100 steps, w has only reached about 2.6. Training takes far longer than necessary.
Note
Just Right (lr = 0.5)
With lr = 0.5: w = 0 - 0.5 × (-6) = 3.0 — the minimum is reached in a single step. In practice, a perfect learning rate is rare, but this shows how powerful the right choice can be.
Try it yourself: as in the worked example, the ball starts at w = 0 on the loss curve L(w) = (w − 3)² + 1. Set the learning rate lr or pick one of the three scenarios and watch whether the ball rolls into the valley, overshoots it or flies off.
Typical Mini-Batch Size
32
The standard number of training samples processed per gradient descent step.
Adam Default Learning Rate
0.001
The default learning rate of the Adam optimizer (Kingma & Ba, 2014).
Adam Optimizer Published
2014
Adam combines multiple optimization techniques and adapts the learning rate per parameter automatically (Kingma & Ba).
Caution
Common Misconception
"You always have to manually tune the learning rate." — Not entirely true. Learning rate schedules reduce the rate over time (large steps early, fine-tuning later); for large models it usually first rises during a short warm-up phase. The Adam optimizer (Kingma & Ba, 2014) adapts the learning rate per parameter automatically. However, the initial learning rate still matters — Adam simplifies but does not eliminate the decision.
The Landscape — Saddle Points, Local Minima, Plateaus
The loss function of a real neural network with millions of parameters creates a complex, high-dimensional landscape. This landscape contains three types of obstacles where the gradient approaches zero — but for very different reasons.
Landscape Topology
Key insight: In high-dimensional spaces (millions of parameters), a true local minimum requires ALL dimensions to simultaneously curve upward. This is statistically extremely unlikely, unless the loss there is already almost as low as at the global minimum. So the local minima that do exist are usually almost as good as the global one. Saddle points, where some dimensions curve up and others down, are far more common — and practically more dangerous.
Caution
Common Misconception
"Gradient descent always finds the global minimum." — Wrong. This is only guaranteed for convex functions (functions with a single valley and no other dips) and a suitable learning rate. Neural networks have non-convex loss landscapes with many dips and saddle points. In practice, finding a "good enough" minimum is sufficient — and is usually what happens.
Caution
Common Misconception
"SGD is inferior to Batch GD because it is noisier." — The opposite is often true. At a saddle point, Batch GD computes the exact gradient, which is zero — so it can stay stuck there for an extremely long time. SGD computes a noisy approximation from a random sample — the gradient is never exactly zero at a saddle point. This nudges the model off the saddle and lets it find the downhill slope. The noise is a feature, not a bug.
Adam (Adaptive Moment Estimation) maintains two running exponential averages per parameter: the first moment (average of recent gradients) and the second moment (average of recent squared gradients). Parameters that consistently receive large gradients get a smaller effective learning rate — this keeps each step roughly the size of the learning rate. Parameters with small or sparse gradients get a larger effective learning rate — they need bigger nudges. Default hyperparameters: lr = 0.001, beta1 = 0.9, beta2 = 0.999. Large language models such as Llama 3 and DeepSeek-V3 mostly train with AdamW, a refinement of Adam. Since 2025, some large models such as Kimi K2 and DeepSeek-V4 have relied on the newer optimizer Muon.
Full computation for L(w) = (w - 3)² + 1, starting at w = 0, lr = 0.1: Step 1: w = 0.00, gradient = -6.00, new w = 0.60, loss = 6.76 Step 2: w = 0.60, gradient = -4.80, new w = 1.08, loss = 4.69 ... steps 3-8 follow the same pattern — the gradient shrinks with each step, progress slows down ... Step 9: w = 2.50, gradient = -1.01, new w = 2.60, loss = 1.16 Step 10: w = 2.60, gradient = -0.81, new w = 2.68, loss = 1.10 Notice the pattern: In the first two steps, w leaps from 0 to 1.08 — a huge jump. By the end, it barely moves from 2.60 to 2.68. The steps become tiny because the valley flattens out and the gradient shrinks near the minimum.
What Comes Next
Gradient descent tells us HOW to update weights — but for a network with many layers, we need to know how errors propagate backward through the network to compute the gradient for each weight. That is backpropagation — the subject of the next article, which builds on the chain rule.
Backpropagation Algorithm
The birth of modern machine learning through an elegant training algorithm. In October 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published the paper 'Learning representations by back-propagating errors' in Nature. This algorithm considerably changed the training of neural networks by providing an efficient method for adjusting weights in multi-layer networks. The procedure repeatedly adjusts the connection weights to minimize the difference between actual and desired output. The decisive innovation lay in the ability to train hidden layers that automatically recognize important features of the task. The mathematical foundations had already been derived earlier — by Paul Werbos (1974) and Seppo Linnainmaa (1970), among others — but it was this paper that made backpropagation widely known and convincingly demonstrated its effectiveness. Backpropagation became the workhorse of machine learning and today enables all modern deep learning applications.
Note
Takeaways
- Gradient descent adjusts weights by stepping opposite to the gradient — the minus sign in the update rule is the entire point.
- The learning rate is often the most impactful hyperparameter: too large causes divergence, too small causes stagnation, and adaptive methods like Adam help but don't eliminate the need for a good initial choice.
- In real neural networks, saddle points (not local minima) are the primary obstacle, and the noise in SGD is a feature that helps escape them.
Check
Knowledge Check: Gradient Descent
In the gradient descent update rule w_new = w_old - lr × dL/dw, why is the gradient subtracted rather than added?
Select one answer
1. In the gradient descent update rule w_new = w_old - lr × dL/dw, why is the gradient subtracted rather than added?
2. You train a neural network and observe: the loss decreases for the first 10 epochs, then starts increasing rapidly and eventually reaches values higher than the initial loss. Which learning rate scenario most likely explains this?
3. A colleague argues: "We should use Batch Gradient Descent instead of SGD because it computes the exact gradient, which must be better." Using what you know about saddle points, how would you respond?
4. Given L(w) = (w - 3)² + 1, current weight w = 1.0, and learning rate lr = 0.1: What is the new weight after one gradient descent step?
5. Why are local minima LESS of a practical problem than saddle points when training neural networks with millions of parameters?
Check
Self-Check
Can you explain this in your own words? Tick what you've got.
0 of 3 done