How AI Measures Its Mistakes: Loss Functions

After the forward pass (the forward computation through the network), the network has a prediction. But a prediction without a score is useless — the model cannot improve if it does not know how far off it landed.

The loss function closes this gap. It takes the model's output and the correct answer, and compresses their difference into one number. That number is the starting signal for everything that follows: gradient computation, weight updates, and ultimately, learning.

Mean Squared Error: Measuring Distance in Regression

Why square? First: without squaring, positive and negative errors cancel each other out — an overestimate of +5 and an underestimate of -5 would average to zero error. Second: squaring amplifies large errors disproportionately — an error of 10 contributes 100 to the sum, while an error of 1 contributes only 1.

MSE forces the network to fix its worst predictions first. The quadratic penalty makes large errors disproportionately expensive.

Cross-Entropy: The Price of Confident Mistakes

Concrete example: A spam classifier evaluates an email that is actually spam (true label = 1):

Spam Probability  |  Loss: -log(p)  |  Interpretation
0.99               |  0.01           |  Correct and confident — tiny loss
0.80               |  0.22           |  Correct but uncertain — small loss
0.50               |  0.69           |  Coin flip — moderate loss
0.10               |  2.30           |  Wrong and fairly confident — large loss
0.01               |  4.61           |  Wrong and very confident — enormous loss

For problems with more than two classes, Categorical Cross-Entropy is used, typically combined with Softmax. The principle remains the same: confident wrong answers are punished disproportionately.

Interactive: How Differently Errors Are Penalised

Try it yourself: move the prediction for house 4 from the worked example and watch how the absolute and the squared error grow and what that does to the MSE of all five houses. Under “Classification” you set the spam probability of the spam email from the table: the more confidently the model is wrong, the more steeply cross-entropy climbs, while the squared error never goes above 1.

The Loss Landscape: Connecting Error to Optimisation

Every possible combination of the network's weights produces a specific loss value. If you could plot all these values at once, you would see a vast mathematical landscape — a surface where horizontal dimensions represent weight values and the vertical dimension represents loss magnitude.

Imagine standing blindfolded on a mountainside. Your only sensory input is the slope under your feet (the gradient). You want to reach the valley floor (minimum loss). With each step, you feel which direction slopes downward and walk that way.

A smooth, bowl-shaped mountain (a good loss landscape) makes this straightforward — every step brings you closer. A landscape with many small dips and ridges (a bad loss landscape) means you might walk into a shallow dip, feel flat ground in every direction, and stop — thinking you have arrived when the real valley is far away.

The loss function provides the altitude, the gradient provides the direction. In the next article, you will learn how gradient descent actually navigates this landscape.