Every neural network starts with random weights — essentially random guesses. The entire training process is about turning those guesses into useful values. But how? The answer is an algorithm so simple it can be explained in one sentence: compute which direction is downhill, take a step.

This article explains gradient descent — the engine behind virtually all modern AI training — and why the seemingly minor detail of step size can make or break your model.

Core Thesis

Gradient descent is an optimization algorithm that iteratively adjusts model parameters by moving in the direction opposite to the gradient of the loss function. The size of each step is controlled by the learning rate — the single most important hyperparameter in deep learning. The algorithm's behavior in the complex, high-dimensional loss landscapes of real neural networks is governed by the interplay between gradient direction, step size, and noise.

The Algorithm — Downhill in the Fog

Imagine you are lost on a mountainside in thick fog. You cannot see the valley, but you can feel which direction the ground drops most steeply under your feet. You take a step in that direction. Then you feel again. Step by step, you descend toward the valley. This is exactly what gradient descent does — it feels the slope of the loss function and steps downhill.

Analogy:

Imagine you are lost on a mountainside in thick fog. You cannot see the valley, but you can feel which direction the ground drops most steeply under your feet. You take a step in that direction. Then you feel again. Step by step, you descend toward the valley. This is exactly what gradient descent does — it feels the slope of the loss function and steps downhill.

Definition:

An iterative optimization algorithm. At each step, it computes the gradient of the loss function with respect to the current weights. The gradient points in the direction of steepest ascent. The algorithm moves in the opposite direction by subtracting the gradient (scaled by the learning rate) from the current weights. The process repeats until the loss converges.

Limitation: On a real mountain, you can feel the slope perfectly. In high-dimensional parameter spaces, the gradient is computed from noisy data samples, making the measurement imprecise. Also, real mountains typically have one valley — loss landscapes can have many deceptive dips and saddle ridges.

The Update Rule

The entire algorithm is captured in a single equation:

w_new = w_old - learning_rate × gradient

Why the minus sign? Because the gradient points uphill (direction of steepest ascent). We want to go downhill (toward lower loss). The minus reverses the direction.

Worked Example: Step by Step

Consider the loss function L(w) = (w - 3)² + 1 with starting weight w = 0 and learning rate 0.1. The gradient here is the first derivative: dL/dw = 2(w - 3). The minimum is at w = 3 where L = 1.

1

Step 1: Gradient = 2(0 - 3) = -6. Update: w = 0 - 0.1 × (-6) = 0.6. Loss: 6.76

2

Step 2: Gradient = 2(0.6 - 3) = -4.8. Update: w = 0.6 - 0.1 × (-4.8) = 1.08. Loss: 4.69

3

Step 3: Gradient = 2(1.08 - 3) = -3.84. Update: w = 1.08 - 0.1 × (-3.84) = 1.464. Loss: 3.36

4

Step 4: Gradient = 2(1.464 - 3) = -3.07. Update: w = 1.464 - 0.1 × (-3.07) = 1.771. Loss: 2.51

5

Step 5: Gradient = 2(1.771 - 3) = -2.46. Update: w = 1.771 - 0.1 × (-2.46) = 2.017. Loss: 1.97

Three Variants of Gradient Descent

Batch Gradient Descent

Uses the ENTIRE dataset to compute the gradient at each step. Advantage: stable, exact gradient. Disadvantage: slow with large datasets, gets stuck at saddle points because the gradient is exactly zero there.

Stochastic Gradient Descent (SGD)

Uses a SINGLE random sample per step. Advantage: fast, noisy updates help escape saddle points (more on this in the Loss Landscape section). Disadvantage: very noisy — the path to the minimum zigzags wildly.

In practice, Mini-Batch Gradient Descent is the golden middle and today's standard: it uses a small batch (typically 32-256 samples) per step. This combines the stability of Batch GD with the speed and noise benefits of SGD.

Interactive: Gradient Descent Step by Step

Click through the five steps and watch the ball roll down the loss curve. Notice how the steps get smaller as the ball approaches the minimum — the gradient becomes flatter.

1

2

3

4

5

Step 1 / 5Starting position

The ball starts far left of the minimum. The loss is high (L=105.8). The gradient points steeply downhill to the right — the algorithm knows which direction to go.

The Dial — Learning Rate

The learning rate is a number set by the developer that controls how large each step is. It is the single most important hyperparameter in deep learning. It is NOT learned by the algorithm — the developer must choose it.

Three Scenarios — Same Function, Different Step Sizes

Too Large (lr = 1.5)

w = 0 → 9.0 → -9.0 → ... The weights oscillate wildly and diverge. The loss explodes instead of decreasing. The model has learned nothing.

Too Small (lr = 0.01)

w = 0 → 0.06 → 0.12 → ... The weights barely move. After hundreds of steps, the model is still far from the minimum. Training takes forever.

With lr = 0.5: w = 0 - 0.5 × (-6) = 3.0 — the minimum is reached in a single step. In practice, a perfect learning rate is rare, but this shows how powerful the right choice can be.

32

Typical Mini-Batch Size The standard number of training samples processed per gradient descent step.

0.001

Adam Default Learning Rate The default learning rate of the Adam optimizer (Kingma & Ba, 2014).

2014

Adam Optimizer Published Adam combines multiple optimization techniques and adapts the learning rate per parameter automatically (Kingma & Ba).

"You always have to manually tune the learning rate." — Not entirely true. Learning rate schedules reduce the rate over time (large steps early, fine-tuning later). The Adam optimizer (Kingma & Ba, 2014) adapts the learning rate per parameter automatically. However, the initial learning rate still matters — Adam simplifies but does not eliminate the decision.

Interactive: Computation Cost per Gradient Step

Move the slider to change the dataset size. Observe how the computation per step differs between SGD (single sample), mini-batch, and full-batch GD. This is why mini-batch is the standard.

Input size n = 100

110000

SGD (1 Sample)1

Mini-batch6.6

Full-batch GD100

Moderate Input

At n=100, the difference becomes visible: O(n²) requires 10.000 operations, while O(n) needs only 100. O(log n) needs just 6.6 — that's 15x less than O(n).

Ratio to O(n)

Complexity	Operations	Factor vs. O(n)
SGD (1 Sample)	1	100x faster
Mini-batch	6.6	15x faster
Full-batch GD	100	1x (Reference)

The Landscape — Saddle Points, Local Minima, Plateaus

The loss function of a real neural network with millions of parameters creates a complex, high-dimensional landscape. This landscape contains three types of obstacles where the gradient approaches zero — but for very different reasons.

Landscape Topology

Local Minimum All dimensions curve upward — the gradient is zero and the loss is lower than the immediate surroundings. Extremely rare in high-dimensional spaces because ALL dimensions must simultaneously curve up.

Saddle Point Mixed curvature — some dimensions curve up, others curve down. Like a mountain pass: valley in front and behind, peaks to left and right. The gradient is zero but it is NOT a minimum. Far more common than local minima.

Plateau Nearly flat region where the gradient is close to zero. Training stagnates because the algorithm receives almost no signal about which direction to move.

Key insight: In high-dimensional spaces (millions of parameters), a true local minimum requires ALL dimensions to simultaneously curve upward. This is statistically extremely unlikely. Saddle points, where some dimensions curve up and others down, are far more common — and practically more dangerous.

"Gradient descent always finds the global minimum." — Wrong. This is only guaranteed for convex functions (functions with a single valley and no other dips). Neural networks have non-convex loss landscapes with many dips and saddle points. In practice, finding a "good enough" minimum is sufficient — and is usually what happens.

"SGD is inferior to Batch GD because it is noisier." — The opposite is often true. At a saddle point, Batch GD computes the exact gradient, which is zero — so it gets permanently stuck. SGD computes a noisy approximation from a random sample — the gradient is never exactly zero at a saddle point. This nudges the model off the saddle and lets it find the downhill slope. The noise is a feature, not a bug.

Adam (Adaptive Moment Estimation) maintains two running exponential averages per parameter: the first moment (average of recent gradients) and the second moment (average of recent squared gradients). Parameters that consistently receive large gradients get a smaller effective learning rate — they are already heading in the right direction and need less correction. Parameters with noisy or sparse gradients get a larger effective learning rate — they need bigger nudges. Default hyperparameters: lr = 0.001, beta1 = 0.9, beta2 = 0.999.

Full computation for L(w) = (w - 3)² + 1, starting at w = 0, lr = 0.1: Step 1: w = 0.00, gradient = -6.00, new w = 0.60, loss = 6.76 Step 2: w = 0.60, gradient = -4.80, new w = 1.08, loss = 4.69 ... steps 3-8 follow the same pattern — the gradient shrinks with each step, progress slows down ... Step 9: w = 2.50, gradient = -1.01, new w = 2.60, loss = 1.16 Step 10: w = 2.60, gradient = -0.81, new w = 2.68, loss = 1.10 Notice the pattern: In the first two steps, w leaps from 0 to 1.08 — a huge jump. By the end, it barely moves from 2.60 to 2.68. The steps become tiny because the valley flattens out and the gradient shrinks near the minimum.

What Comes Next

Gradient descent tells us HOW to update weights — but for a network with many layers, we need to know how errors propagate backward through the network to compute the gradient for each weight. That is backpropagation — the subject of the next article, which builds on the chain rule.

Gradient descent adjusts weights by stepping opposite to the gradient — the minus sign in the update rule is the entire point.
The learning rate is the most impactful hyperparameter: too large causes divergence, too small causes stagnation, and adaptive methods like Adam help but don't eliminate the need for a good initial choice.
In real neural networks, saddle points (not local minima) are the primary obstacle, and the noise in SGD is a feature that helps escape them.

In the gradient descent update rule w_new = w_old - lr × dL/dw, why is the gradient subtracted rather than added?

It is an arbitrary convention that could be reversed

Because the gradient points in the direction of steepest ascent, and we want to go downhill (toward lower loss)

Because subtraction makes the computation faster

Because adding the gradient would increase the number of parameters

1. In the gradient descent update rule w_new = w_old - lr × dL/dw, why is the gradient subtracted rather than added?

☐ A) It is an arbitrary convention that could be reversed
☐ B) Because the gradient points in the direction of steepest ascent, and we want to go downhill (toward lower loss)
☐ C) Because subtraction makes the computation faster
☐ D) Because adding the gradient would increase the number of parameters

2. You train a neural network and observe: the loss decreases for the first 10 epochs, then starts increasing rapidly and eventually reaches values higher than the initial loss. Which learning rate scenario most likely explains this?

☐ A) Learning rate too small — the model is underfitting
☐ B) Learning rate too large — the model is overshooting the minimum and diverging
☐ C) The model has converged to a local minimum
☐ D) The mini-batch size is too large

3. A colleague argues: "We should use Batch Gradient Descent instead of SGD because it computes the exact gradient, which must be better." Using what you know about saddle points, how would you respond?

☐ A) Correct — exact gradients always lead to better convergence
☐ B) Incorrect — the exact gradient at a saddle point is zero, so Batch GD gets permanently stuck; SGD's noisy gradients are never exactly zero at a saddle, allowing the model to escape
☐ C) Incorrect — Batch GD is too slow, but its gradient quality is irrelevant
☐ D) Correct — but only for convex loss functions

4. Given L(w) = (w - 3)² + 1, current weight w = 1.0, and learning rate lr = 0.1: What is the new weight after one gradient descent step?

☐ A) w = 0.6
☐ B) w = 1.4
☐ C) w = 1.2
☐ D) w = 0.96

5. Why are local minima LESS of a practical problem than saddle points when training neural networks with millions of parameters?

☐ A) Because local minima have higher loss values than saddle points
☐ B) Because SGD always finds the global minimum
☐ C) Because a true local minimum requires ALL dimensions to curve upward simultaneously, which is statistically extremely unlikely in high-dimensional spaces
☐ D) Because modern optimizers like Adam eliminate local minima entirely

Answer Key: 1) B · 2) B · 3) B · 4) B · 5) C

Self-Check

Why is there a minus sign in the update rule — and what would happen if you added the gradient instead?
Why does a learning rate that is too large not simply produce a slightly less accurate result, but can cause the model to diverge entirely?
Imagine your algorithm is stuck at a saddle point. Why is it a major advantage if your compass (the gradient) wobbles slightly?

The Path to the Valley: Gradient Descent

Core Thesis

The Algorithm — Downhill in the Fog

Gradient Descent

Analogy:

Definition:

The Update Rule

The Update Rule

Worked Example: Step by Step

Three Variants of Gradient Descent

Interactive: Gradient Descent Step by Step

The Dial — Learning Rate

Three Scenarios — Same Function, Different Step Sizes

Just Right (lr = 0.5)

Common Misconception

Interactive: Computation Cost per Gradient Step

Ratio to O(n)

The Landscape — Saddle Points, Local Minima, Plateaus

Landscape Topology

Common Misconception

Common Misconception

Deep Dive: The Adam Optimizer

Deep Dive: 10-Step Computation

What Comes Next

Backpropagation Algorithm

Takeaways

Knowledge Check: Gradient Descent

In the gradient descent update rule w_new = w_old - lr × dL/dw, why is the gradient subtracted rather than added?

Self-Check

Core Thesis

The Algorithm — Downhill in the Fog

Gradient Descent

Analogy:

Definition:

The Update Rule

The Update Rule

Worked Example: Step by Step

Three Variants of Gradient Descent

Interactive: Gradient Descent Step by Step

The Dial — Learning Rate

Three Scenarios — Same Function, Different Step Sizes

Just Right (lr = 0.5)

Common Misconception

Interactive: Computation Cost per Gradient Step

Ratio to O(n)

The Landscape — Saddle Points, Local Minima, Plateaus

Landscape Topology

Common Misconception

Common Misconception

Deep Dive: The Adam Optimizer

Deep Dive: 10-Step Computation

What Comes Next

Backpropagation Algorithm

Takeaways

Knowledge Check: Gradient Descent

In the gradient descent update rule w_new = w_old - lr × dL/dw, why is the gradient subtracted rather than added?

Self-Check

Related Content

Article

Measuring Change: Derivatives

How AI Measures Its Mistakes: Loss Functions

The Artificial Neuron

The Heart of Learning

The Spark: Activation Functions

The Chain Rule: Differentiating Nested Functions

The Raw Material: Data Engineering for Machine Learning

The Network

The Machine's Knobs — Parameters vs. Hyperparameters

Linear & Logistic Regression

Data Tables & Transformations (Matrices)

Programming vs. Training

When the Model Memorizes (Overfitting)

Partial Derivatives & The Gradient

Spread of Data

Supervised Learning — Learning with a Teacher

Tensors: The Language of GPUs

Spaces and Directions (Vectors)

Demo

Evolution (Optimization)

GANs (Generative Models)

Neuroevolution

Neural Network Playground

Perceptron (Neural Networks)

Supervised Learning

Travelling Salesman: Algorithms in Competition

Glossary

Timeline