The Spark: Activation Functions
The one small kink in the line without which neural networks wouldn't learn anything at all.
A neural network layer multiplies inputs by weights and adds a bias — a purely linear operation. Stack as many of these as you like, and you still have a linear model: incapable of learning curves, corners, or the XOR pattern.
Activation functions are the fix. By inserting a nonlinear transformation between layers, they give the network the power to bend decision boundaries and ultimately approximate any continuous function.
The Linear Trap
Definition
Activation Function
Analogy
Example
Definition
Why depth without activation is an illusion: If layer 1 computes y = W₁x + b₁ and layer 2 computes z = W₂y + b₂, then:
The two weight matrices collapse into a single product. No matter how many layers you add, the network can only draw straight decision boundaries.
The XOR problem is the canonical proof: the correct output cannot be separated by any single line. Pure linearity is insufficient.
Caution
Misconception: More Layers = More Power
More layers only increase representational capacity if nonlinear activation functions separate them. A 100-layer network without activation is mathematically identical to a 1-layer network — with different but equally linear weights.
Sigmoid, Tanh, and the Vanishing Gradient
Sigmoid (σ(z) = 1/(1+e⁻ᶻ)) squashes any input into the range (0, 1) — don't worry, you never need to compute this formula yourself. Tanh squashes into (-1, 1) and is zero-centred. Both are smooth, differentiable (meaning soft curves with no abrupt jumps), and historically important.
The real problem lies in their derivatives — the learning signal sent back through the network during training: sigmoid's maximum derivative is only 0.25 (at z = 0). During backpropagation, gradients are multiplied layer by layer.
The whisper chain: Imagine passing a message through a chain of people, but each person can only whisper at one-quarter volume. After the first person: 25% volume. After the second: 6.25%. By the tenth: 0.0001% — practically silent. In a sigmoid network, the "message" is the gradient, and each layer acts as a volume reducer.
Gradient after 10 sigmoid layers
0,25 ¹⁰ ≈ 10⁻⁶
of the original signal, even at the maximum sigmoid derivative — effectively zero. Early layers stop learning.
Caution
Misconception: Sigmoid Is Bad
Sigmoid is a bad choice for hidden layers in deep networks because of vanishing gradients. But it remains the CORRECT choice for binary classification output layers, where you specifically want a (0, 1) probability output. The function itself is not flawed — it is misapplied when used throughout a deep architecture.
ReLU and the Modern Toolkit
ReLU (Rectified Linear Unit) is almost absurdly simple: f(z) = max(0, z). For positive inputs, the output equals the input and the derivative is exactly 1. For negative inputs, both output and derivative are zero.
Think of a one-way valve in a water pipe. Positive pressure (z > 0) flows through unchanged — no friction, no loss. Negative pressure (z < 0) is completely blocked. In a long pipeline of 50 valves, the water pressure at the end equals the pressure at the start (for positive flow). With 50 sigmoid pressure regulators, each cutting flow to 25%, nothing comes out the other end.
Shape: Soft S-curve • Range: (0, 1) • Max derivative: 0.25 • Gradient shrinks per layer • Use: Output layer (binary classification)
Shape: Bend at zero • Range: [0, ∞) • Derivative: 1 (z>0) or 0 • NO systematic gradient shrinkage • Use: Default for hidden layers
In 2012, image recognition showed that a network with ReLU learned roughly six times faster than with tanh. This speed advantage was one of the key factors that enabled training deep networks at scale.
AlexNet Success
The turning point for deep learning and modern AI. On September 30, 2012, the team submitted AlexNet to the ImageNet Challenge; when the results were published in October 2012, AlexNet had won by such a wide margin that computer vision was fundamentally changed. Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton from the University of Toronto developed a CNN architecture that beat its competition by a notable 10.9 percentage points — an improvement considered extraordinary in the scientific community. With 60 million parameters and innovative techniques such as ReLU activations and dropout layers, AlexNet demonstrated the practical superiority of deep learning impressively. That was the moment when an interesting theory became a dominant technology. Yann LeCun called it 'an undeniable turning point in the history of computer vision.' The GPU-based implementation paved the way for modern AI development.
Yet ReLU has a weakness: if a neuron's inputs become consistently negative (e.g. from a large gradient update), it outputs zero permanently — a "dead ReLU". Leaky ReLU (f(z) = max(0.01z, z)) addresses this by allowing a small gradient for negative inputs.
In Transformer architectures, GELU is widely used (newer language models such as Llama use the related variant SwiGLU) — a smoother variant of ReLU that softens the transition at zero instead of cutting off sharply.
Signal Flow Through a Neuron
- 1. Input x
- 2. Weighted sum z = Wx + b
- 3. Activation fz
- 4. Output a
Caution
Misconception: ReLU Is Always Best
ReLU is the standard default for hidden layers, but the output layer requires task-appropriate activations. Binary classification: sigmoid (output in (0, 1) as probability). Multi-class: softmax (outputs sum to 1). Regression: often no activation (linear output).
Interactive: Compare Activation Functions
Move the slider to test different input values x. Switch between ReLU, Sigmoid, and Tanh to see how each function transforms the same input. Pay special attention to the derivative: for Sigmoid it maxes out at 0.25 — exactly the vanishing gradient problem.
max(0, 1.0) = 1.0000Cybenko showed in 1989 that a feedforward network with a single hidden layer and sigmoid activation can approximate any continuous function on a closed, bounded input range to arbitrary accuracy. Hornik generalised in 1991 that this holds for broad classes of activation functions. But caution: this is a pure existence statement — it says THAT such a network exists, not how large it must be. In practice, a shallow network may need impractically many neurons. Deeper networks often achieve the same approximation with fewer parameters, because each layer extracts increasingly abstract features.
Flashcards: Key Activation Functions at a Glance
Click a card to flip it. Front: name and short description. Back: formula and typical use case.
ReLU
The hidden-layer standard since 2012
Passes positive values unchanged, blocks negative ones completely. Gradient = 1 for positive inputs, so no vanishing gradient.
f(x) = max(0, x)
Default activation in CNNs and MLPs since AlexNet (2012)
Note
Key Takeaways
- Without activation functions, a neural network of any depth is mathematically identical to a single linear transformation — depth without nonlinearity wastes parameters.
- Sigmoid and Tanh saturate at their extremes, multiplying gradients by fractions that compound catastrophically across layers — this is the vanishing gradient problem.
- ReLU passes positive values unchanged (gradient = 1), largely solving gradient vanishing in hidden layers. The output layer still needs task-specific activations (sigmoid for probabilities, softmax for multi-class).
Check
Learning Goals
Can you explain this in your own words? Tick what you've got.
0 of 3 done
Check
Quiz: Activation Functions
What happens when you stack multiple neural network layers that each compute only a linear transformation (y = Wx + b) without any activation function?
Select one answer
1. What happens when you stack multiple neural network layers that each compute only a linear transformation (y = Wx + b) without any activation function?
2. Why is the vanishing gradient problem particularly severe with sigmoid activation in deep networks?
3. You are building a neural network to predict whether an email is spam (yes/no). Which activation function should you use in the hidden layers and which in the output layer?
4. What is a "dead ReLU", and why is it problematic?
5. A colleague claims: "We should use ReLU everywhere in the network — hidden layers AND output layer — because it avoids vanishing gradients." Evaluate this claim for a multi-class image classifier with 10 categories.