Loss functions are a model's pain threshold — without them, no learning.
Fundamentals 11 min Intermediate June 1, 2026
After the forward pass (the forward computation through the network), the network has a prediction. But a prediction without a score is useless — the model cannot improve if it does not know how far off it landed.
The loss function closes this gap. It takes the model's output and the correct answer, and compresses their difference into one number. That number is the starting signal for everything that follows: gradient computation, weight updates, and ultimately, learning.
1986 Papers
Backpropagation Algorithm
The birth of modern machine learning through an elegant training algorithm. In October 1986, David Rumelhart, Geoffrey Hinton, and Ronald Williams published the paper 'Learning representations by back-propagating errors' in Nature. This algorithm considerably changed the training of neural networks by providing an efficient method for adjusting weights in multi-layer networks. The procedure repeatedly adjusts the connection weights to minimize the difference between actual and desired output. The decisive innovation lay in the ability to train hidden layers that automatically recognize important features of the task. The mathematical foundations had already been derived earlier — by Paul Werbos (1974) and Seppo Linnainmaa (1970), among others — but it was this paper that made backpropagation widely known and convincingly demonstrated its effectiveness. Backpropagation became the workhorse of machine learning and today enables all modern deep learning applications.
Mean Squared Error: Measuring Distance in Regression
Loss Function
AnalogyDefinition
Imagine scoring archers by how far each arrow lands from the bullseye. Strategy A (absolute distance): 2 cm off scores 2, 10 cm off scores 10. Strategy B (squared distance): 2 cm off scores 4, but 10 cm off scores 100. Under Strategy B, one terrible shot dominates the total score. MSE uses Strategy B — the biggest mistakes must be fixed first.
Example
In practice, you sometimes do NOT want outlier sensitivity. Alternatives like Mean Absolute Error or Huber Loss exist for such cases — MSE's outlier amplification is a deliberate trade-off.
Analogy:
Imagine scoring archers by how far each arrow lands from the bullseye. Strategy A (absolute distance): 2 cm off scores 2, 10 cm off scores 10. Strategy B (squared distance): 2 cm off scores 4, but 10 cm off scores 100. Under Strategy B, one terrible shot dominates the total score. MSE uses Strategy B — the biggest mistakes must be fixed first.
Example
In practice, you sometimes do NOT want outlier sensitivity. Alternatives like Mean Absolute Error or Huber Loss exist for such cases — MSE's outlier amplification is a deliberate trade-off.
Definition:
A loss function compresses the difference between a model's predictions and the true values into a single differentiable number. For regression problems — where the model outputs continuous values like prices or temperatures — the Mean Squared Error (MSE) is the standard. It calculates the difference between each prediction and the true value, squares it, and averages across all data points.
Mean Squared Error (MSE)
MSE = (1/n) · ∑(ypred − ytrue)²
Why square? First: without squaring, positive and negative errors cancel each other out — an overestimate of +5 and an underestimate of -5 would average to zero error. Second: squaring amplifies large errors disproportionately — an error of 10 contributes 100 to the sum, while an error of 1 contributes only 1.
Divide by n MSE = 3.54 / 5 = 0.708 — House 4 (error 1.5, squared 2.25) contributes 63% of the total error alone!
MSE forces the network to fix its worst predictions first. The quadratic penalty makes large errors disproportionately expensive.
Misconception: Just Average the Raw Errors
Without squaring, errors of +1.5 and -1.0 would partially cancel out. The simple average of raw errors here is only 0.04 — suggesting a nearly perfect model, when in fact it is badly wrong on house 4. Squaring prevents this deception.
Cross-Entropy: The Price of Confident Mistakes
Cross-Entropy
AnalogyDefinition
Imagine a weather forecaster who publicly announces confidence levels. On a day that actually rains: "90% chance of rain" — barely affects reputation. "10% chance of rain" when it pours — embarrassing, noticeable damage. "1% chance of rain" during a storm — career-ending, catastrophic damage. The logarithm in Cross-Entropy works the same way: the penalty does not grow linearly, it accelerates.
Example
Real forecasters face social dynamics, not pure log-loss. Also, Cross-Entropy assumes the true labels are correct — mislabelled training data can cause the model to be unfairly penalised.
Analogy:
Imagine a weather forecaster who publicly announces confidence levels. On a day that actually rains: "90% chance of rain" — barely affects reputation. "10% chance of rain" when it pours — embarrassing, noticeable damage. "1% chance of rain" during a storm — career-ending, catastrophic damage. The logarithm in Cross-Entropy works the same way: the penalty does not grow linearly, it accelerates.
Example
Real forecasters face social dynamics, not pure log-loss. Also, Cross-Entropy assumes the true labels are correct — mislabelled training data can cause the model to be unfairly penalised.
Definition:
For classification tasks — where the model outputs probabilities (e.g., 95% spam) — MSE is a poor fit. Cross-Entropy measures the gap between predicted probabilities and the true class labels. Its key weapon is the logarithm: -log(p) is near zero when p is close to 1, but rockets toward infinity as p approaches 0. The penalty for a confident wrong answer is not just large — it is explosively large.
Binary Cross-Entropy
L = −(1/n) · ∑[y · log(p) + (1−y) · log(1−p)]
460×
Penalty difference: -log(0.99) = 0.01 vs. -log(0.01) = 4.61
Concrete example: A spam classifier evaluates an email that is actually spam (true label = 1):
Spam Probability | Loss: -log(p) | Interpretation
0.99 | 0.01 | Correct and confident — tiny loss
0.80 | 0.22 | Correct but uncertain — small loss
0.50 | 0.69 | Coin flip — moderate loss
0.10 | 2.30 | Wrong and fairly confident — large loss
0.01 | 4.61 | Wrong and very confident — enormous loss
For problems with more than two classes, Categorical Cross-Entropy is used, typically combined with Softmax. The principle remains the same: confident wrong answers are punished disproportionately.
Interactive: How Differently Errors Are Penalised
Move the slider to change the error magnitude. Observe how differently the three penalty functions respond: the logarithmic penalty grows slowly, the linear penalty (MAE) proportionally, and the quadratic penalty (MSE) disproportionately. At error magnitude 10, the MSE value is already 100 — ten times larger than MAE.
1100
log(n)6.6
|n| (MAE)100
n² (MSE)10.000
Moderate Input
At n=100, the difference becomes visible: O(n²) requires 10.000 operations, while O(n) needs only 100. O(log n) needs just 6.6 — that's 15x less than O(n).
Ratio to O(n)
Complexity
Operations
Factor vs. O(n)
log(n)
6.6
15x faster
|n| (MAE)
100
1x (Reference)
n² (MSE)
10.000
100x slower
The Loss Landscape: Connecting Error to Optimisation
Every possible combination of the network's weights produces a specific loss value. If you could plot all these values at once, you would see a vast mathematical landscape — a surface where horizontal dimensions represent weight values and the vertical dimension represents loss magnitude.
Imagine standing blindfolded on a mountainside. Your only sensory input is the slope under your feet (the gradient). You want to reach the valley floor (minimum loss). With each step, you feel which direction slopes downward and walk that way.
A smooth, bowl-shaped mountain (a good loss landscape) makes this straightforward — every step brings you closer. A landscape with many small dips and ridges (a bad loss landscape) means you might walk into a shallow dip, feel flat ground in every direction, and stop — thinking you have arrived when the real valley is far away.
Warning: Loss = 0 Is Not a Victory
A training loss of zero almost always means overfitting — the model has memorised the training examples rather than learning generalisable patterns. On unseen test data, such a model typically performs poorly. A healthy training loss settles at a small positive value, reflecting the irreducible noise in the data.
The loss function provides the altitude, the gradient provides the direction. In the next article, you will learn how gradient descent actually navigates this landscape.
Deep Dive: Why Not MSE for Classification?
When the output uses a function that squeezes values between 0 and 1 (the so-called sigmoid function), MSE creates flat gradient zones near 0 or 1. The model receives almost no learning signal despite being completely wrong. Cross-Entropy, on the other hand, produces steep gradients in exactly these zones — the model learns faster and more reliably. Using MSE for classification is like navigating a mountain with a nearly flat map — you can barely tell which direction is downhill.
Key Takeaways
A loss function converts prediction errors into a single differentiable number — this number is the only feedback the network receives about its performance.
MSE squares errors, making large mistakes disproportionately expensive — ideal for regression. Cross-Entropy uses logarithms, making confident wrong answers catastrophically expensive — ideal for classification.
The loss value defines a point in a vast mathematical landscape. Training means navigating this landscape toward its lowest valley — and the shape of that landscape depends entirely on which loss function you chose.
Learning Goals
A model predicts [10, 20, 30] for true values [12, 20, 25]. Compute the MSE and identify which prediction contributes most to the error.
A binary classifier predicts p = 0.95 for a sample whose true label is 0. What is the Cross-Entropy loss?
Your training loss is 0.0001, your validation loss is 2.8. Explain what happened.
Test Your Knowledge
Question 1 / 5
Not completed
What is the primary role of a loss function in a neural network?
1. What is the primary role of a loss function in a neural network?
☐ A) It decides which neurons to activate during the forward pass.
☐ B) It compresses the difference between predictions and true values into a single number that guides weight updates.
☐ C) It determines the architecture of the network (number of layers and neurons).
☐ D) It selects which training examples to show the network next.
2. Why does MSE square the difference between prediction and true value instead of using the absolute difference?
☐ A) Squaring is computationally faster than taking absolute values.
☐ B) While both methods make errors positive, squaring amplifies large errors disproportionately, forcing the model to prioritise fixing its worst predictions.
☐ C) Squaring converts errors into probabilities that sum to 1.
☐ D) Squaring is required by the backpropagation algorithm and cannot be replaced.
3. You are building a model to predict apartment rental prices (continuous values in EUR). Which loss function should you use?
☐ A) Binary Cross-Entropy, because you want to minimise error.
☐ B) Categorical Cross-Entropy, because prices fall into categories.
☐ C) Mean Squared Error, because you are predicting continuous numerical values.
☐ D) Softmax Loss, because you need probabilities.
4. A spam classifier assigns a probability of 0.10 to an email that is actually spam (true label = 1). Using Binary Cross-Entropy, approximate the loss for this single sample.
☐ A) 0.10 (the predicted probability itself)
☐ B) 0.90 (one minus the predicted probability)
☐ C) About 2.3 (because -log(0.10) ≈ 2.30)
☐ D) About 0.1 (because -log(0.90) ≈ 0.11)
5. After 200 epochs, your model achieves a loss of 0.0001 on the training set but 2.8 on the validation set. A colleague celebrates the low training loss. What is the most likely problem?
☐ A) The model is underfitting — it needs more training epochs.
☐ B) The model has overfit — it found a very deep but narrow valley in the training loss landscape that does not generalise to unseen data.
☐ C) The loss function is incorrect — MSE should have been used instead of Cross-Entropy.
☐ D) The model is working perfectly — validation loss is always higher than training loss.