The one small kink in the line without which neural networks wouldn't learn anything at all.
Fundamentals 10 min Intermediate June 1, 2026
A neural network layer multiplies inputs by weights and adds a bias — a purely linear operation. Stack as many of these as you like, and you still have a linear model: incapable of learning curves, corners, or the XOR pattern.
Activation functions are the fix. By inserting a nonlinear transformation between layers, they give the network the power to bend decision boundaries and ultimately approximate any continuous function.
The Linear Trap
Activation Function
AnalogyDefinition
Imagine stacking tinted glass filters in front of a lamp. Each filter tints the light linearly (multiplies each colour channel by a factor). Stack ten filters and the result is still just one combined tint — you can always find a single filter that produces the same effect. But insert a prism (a nonlinear element) between the filters, and suddenly you get rainbows — structure that no single linear filter could produce.
Example
Real glass filters have small nonlinearities (absorption spectra, scattering). The analogy is idealised, but the mathematical point holds exactly for neural networks.
Analogy:
Imagine stacking tinted glass filters in front of a lamp. Each filter tints the light linearly (multiplies each colour channel by a factor). Stack ten filters and the result is still just one combined tint — you can always find a single filter that produces the same effect. But insert a prism (a nonlinear element) between the filters, and suddenly you get rainbows — structure that no single linear filter could produce.
Example
Real glass filters have small nonlinearities (absorption spectra, scattering). The analogy is idealised, but the mathematical point holds exactly for neural networks.
Definition:
A nonlinear function applied element-wise after a neuron's weighted sum. Without it, stacked layers can only produce linear decision boundaries; with it, each layer can carve the input space into increasingly complex regions.
Why depth without activation is an illusion: If layer 1 computes y = W₁x + b₁ and layer 2 computes z = W₂y + b₂, then:
Linear Collapse
z = W₂(W₁x + b₁) + b₂ = (W₂·W₁)x + (W₂·b₁ + b₂)
The two weight matrices collapse into a single product. No matter how many layers you add, the network can only draw straight decision boundaries.
The XOR problem is the canonical proof: the correct output cannot be separated by any single line. Pure linearity is insufficient.
Misconception: More Layers = More Power
More layers only increase representational capacity if nonlinear activation functions separate them. A 100-layer network without activation is mathematically identical to a 1-layer network — with different but equally linear weights.
Sigmoid, Tanh, and the Vanishing Gradient
Sigmoid (σ(z) = 1/(1+e⁻ᶻ)) squashes any input into the range (0, 1) — don't worry, you never need to compute this formula yourself. Tanh squashes into (-1, 1) and is zero-centred. Both are smooth, differentiable (meaning soft curves with no abrupt jumps), and historically important.
The real problem lies in their derivatives — the learning signal sent back through the network during training: sigmoid's maximum derivative is only 0.25 (at z = 0). During backpropagation, gradients are multiplied layer by layer.
The whisper chain: Imagine passing a message through a chain of people, but each person can only whisper at one-quarter volume. After the first person: 25% volume. After the second: 6.25%. By the tenth: 0.0001% — practically silent. In a sigmoid network, the "message" is the gradient, and each layer acts as a volume reducer.
0,25¹⁰≈ 10⁻⁶
Gradient after 10 sigmoid layers of the original signal — effectively zero. Early layers stop learning.
Misconception: Sigmoid Is Bad
Sigmoid is a bad choice for hidden layers in deep networks because of vanishing gradients. But it remains the CORRECT choice for binary classification output layers, where you specifically want a (0, 1) probability output. The function itself is not flawed — it is misapplied when used throughout a deep architecture.
ReLU and the Modern Toolkit
ReLU (Rectified Linear Unit) is almost absurdly simple: f(z) = max(0, z). For positive inputs, the output equals the input and the derivative is exactly 1. For negative inputs, both output and derivative are zero.
Think of a one-way valve in a water pipe. Positive pressure (z > 0) flows through unchanged — no friction, no loss. Negative pressure (z < 0) is completely blocked. In a long pipeline of 50 valves, the water pressure at the end equals the pressure at the start (for positive flow). With 50 sigmoid pressure regulators, each cutting flow to 25%, nothing comes out the other end.
Shape: Bend at zero • Range: [0, ∞) • Derivative: 1 (z>0) or 0 • NO systematic gradient shrinkage • Use: Default for hidden layers
In 2012, image recognition showed that a network with ReLU learned roughly six times faster than with tanh. This speed advantage was one of the key decisions that enabled training deep networks at scale.
2012 Breakthroughs
AlexNet Success
The turning point for deep learning and modern AI. On September 30, 2012, the results of the ImageNet Challenge were published — AlexNet won by such a wide margin that computer vision was fundamentally changed. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton from the University of Toronto developed a CNN architecture that beat its competition by a notable 10.9 percentage points — an improvement considered extraordinary in the scientific community. With 60 million parameters and innovative techniques such as ReLU activations and dropout layers, AlexNet demonstrated the practical superiority of deep learning impressively. That was the moment when an interesting theory became a dominant technology. Yann LeCun called it 'an undeniable turning point in the history of computer vision.' The GPU-based implementation paved the way for modern AI development.
Yet ReLU has a weakness: if a neuron's inputs become consistently negative (e.g. from a large gradient update), it outputs zero permanently — a "dead ReLU". Leaky ReLU (f(z) = max(0.01z, z)) addresses this by allowing a small gradient for negative inputs.
In Transformer architectures, GELU has become the standard activation — a smoother variant of ReLU that softens the transition at zero instead of cutting off sharply.
Signal Flow Through a Neuron
1
Input x
2
Weighted sum z = Wx + b
3
Activation f(z)
4
Output a
Misconception: ReLU Is Always Best
ReLU is the standard default for hidden layers, but the output layer requires task-appropriate activations. Binary classification: sigmoid (output in [0,1] as probability). Multi-class: softmax (outputs sum to 1). Regression: often no activation (linear output).
Interactive: Compare Activation Functions
Move the slider to test different input values x. Switch between ReLU, Sigmoid, and Tanh to see how each function transforms the same input. Pay special attention to the derivative: for Sigmoid it maxes out at 0.25 — exactly the vanishing gradient problem.
x = 1.0
Formulamax(0, 1.0) = 1.0000
1.0000Output f(x)
1.0000Derivative f'(x)
f(x)
1.00
f'(x)
1.00
Deep Dive: Universal Approximation Theorem
Cybenko showed in 1989 that a feedforward network with a single hidden layer and sigmoid activation can approximate any continuous function to arbitrary accuracy. Hornik generalised in 1991 that this holds for broad classes of activation functions. But caution: this is a pure existence statement — it says THAT such a network exists, not how large it must be. In practice, a shallow network may need impractically many neurons. Deeper networks often achieve the same approximation with fewer parameters, because each layer extracts increasingly abstract features.
Flashcards: Key Activation Functions at a Glance
Click a card to flip it. Front: name and short description. Back: formula and typical use case.
Card 1 of 5
Term
ReLU
The hidden-layer standard since 2012
Click to flip
Explanation
Passes positive values unchanged, blocks negative ones completely. Gradient = 1 for positive inputs, so no vanishing gradient.
Formula
f(x) = max(0, x)
Example
Default activation in CNNs and MLPs since AlexNet (2012)
Click to flip back
Key Takeaways
Without activation functions, a neural network of any depth is mathematically identical to a single linear transformation — depth without nonlinearity wastes parameters.
Sigmoid and Tanh saturate at their extremes, multiplying gradients by fractions that compound catastrophically across layers — this is the vanishing gradient problem.
ReLU passes positive values unchanged (gradient = 1), largely solving gradient vanishing in hidden layers. The output layer still needs task-specific activations (sigmoid for probabilities, softmax for multi-class).
Learning Goals
Why is a network without activation functions no more powerful than a single linear layer, despite having many layers?
Why does sigmoid easily lead to vanishing gradients in deep networks, and why is this problematic for learning?
Why does ReLU help with training deep networks, and why is it still not automatically the right choice for the output layer?
Quiz: Activation Functions
Question 1 / 5
Not completed
What happens when you stack multiple neural network layers that each compute only a linear transformation (y = Wx + b) without any activation function?
1. What happens when you stack multiple neural network layers that each compute only a linear transformation (y = Wx + b) without any activation function?
☐ A) The network becomes more powerful with each added layer.
☐ B) The network is mathematically equivalent to a single linear layer.
☐ C) The network can learn nonlinear patterns but trains more slowly.
☐ D) The network automatically develops nonlinear behaviour through weight interactions.
2. Why is the vanishing gradient problem particularly severe with sigmoid activation in deep networks?
☐ A) Sigmoid outputs are always close to zero, so all signals vanish.
☐ B) Sigmoid is computationally expensive, causing numerical underflow in deep networks.
☐ C) Sigmoid's maximum derivative is 0.25, and these small factors multiply across layers, causing exponential decay of the learning signal.
☐ D) Sigmoid only works with positive inputs, limiting its gradient range.
3. You are building a neural network to predict whether an email is spam (yes/no). Which activation function should you use in the hidden layers and which in the output layer?
☐ A) ReLU in the hidden layers, sigmoid in the output layer.
☐ B) Sigmoid in both hidden and output layers.
☐ C) ReLU in the hidden layers, softmax in the output layer.
☐ D) Sigmoid in the hidden layers, ReLU in the output layer.
4. What is a "dead ReLU", and why is it problematic?
☐ A) A neuron that outputs excessively large values for positive inputs, destabilising the network.
☐ B) A neuron that permanently outputs zero because its inputs are always negative — it stops learning.
☐ C) A neuron that stops computing because its bias grows too large.
☐ D) A neuron that only activates for very large inputs and ignores small signals.
5. A colleague claims: "We should use ReLU everywhere in the network — hidden layers AND output layer — because it avoids vanishing gradients." Evaluate this claim for a multi-class image classifier with 10 categories.
☐ A) Correct — ReLU's gradient of 1 is always optimal, regardless of the task.
☐ B) Partially correct — ReLU is fine for hidden layers, but the output layer needs softmax to produce a valid probability distribution over 10 classes.
☐ C) Incorrect — ReLU causes exploding gradients in all layers and should never be used.
☐ D) Partially correct — ReLU works for the output layer but sigmoid is better for hidden layers.