Back to Learning Area

How Good Is Your Model? Metrics That Actually Matter

Evaluating models without self-deception — metrics that do more than just look good.

  • Fundamentals
  • 10 min
  • Intermediate

Plain Language

This preview is in Plain Language. The whole portal is also available in Plain Language. You can change the language at the top. Or use the button below. Our Plain Language texts are AI-assisted and editorially reviewed. They follow the core principles of easy-to-read language, but they are not certified Easy Read: a review panel of people with learning difficulties does not test them.

Preview: Evaluating AI models correctly

The problem with accuracy alone

An AI model has an accuracy of 99 percent. Sounds good. But maybe it just says: everything is healthy. Then it overlooks all sick patients. Accuracy alone sometimes lies.

The confusion matrix

A confusion matrix shows all types of errors. A model can be right or wrong. Good is: sick recognized as sick, and healthy recognized as healthy. Bad is: sick overlooked or healthy classified incorrectly.

Precision, recall, and F1

Precision means: how many of the found cases are really positive? Recall means: how many of the real cases did the model discover? The F1-score combines precision and recall into one value. This is how you choose the right metric.

A model predicts "healthy" for every patient and achieves 99% accuracy. Impressive? No, because the 1% it misses are exactly the patients who actually have the disease. Accuracy alone can be a dangerous lie — and the Confusion Matrix reveals the truth.

A single metric never tells the full story. In this article, you will learn to use the Confusion Matrix as a diagnostic tool and derive Accuracy, Precision, Recall, and the F1-Score from it. Which metric matters most is not a math question but a design decision.

The Confusion Matrix: The Foundation

Definition

Confusion Matrix

Analogy

Think of a COVID test. "Positive" means the test says you are sick. "Negative" means the test says you are healthy. The test can be right in two ways (True Positive, True Negative) and wrong in two ways (False Positive = false alarm, False Negative = missed case). The Confusion Matrix is this test's report card.

Imagine: 1,000 patients are tested. 50 are actually sick, 950 are healthy. The test produces the following results:

The COVID test analogy precisely shows how errors can go in two fundamentally different directions. The limitation: real-world classification problems can have more than two classes, and the four equally weighted cells can suggest equal error costs when they are often asymmetric.

True Positive (TP)

45

Sick and correctly identified as sick — 45 patients

False Negative (FN)

5

Sick but incorrectly classified as healthy — 5 patients

False Positive (FP)

47

Healthy but incorrectly classified as sick — 47 patients

True Negative (TN)

903

Healthy and correctly identified as healthy — 903 patients

The standard metrics in this article — Accuracy, Precision, Recall, F1-Score — are simply different views of these four numbers. The Confusion Matrix is the diagnostic surface of your model.

Caution

Common Misconception: "High accuracy means I don't need the Confusion Matrix"

Two models with identical accuracy can have completely different error profiles. One produces many false alarms; the other misses many real cases. Only the Confusion Matrix shows HOW your model is wrong — not just HOW OFTEN.

Three Lenses: Accuracy, Precision, and Recall

  1. 1. Collect predictions
  2. 2. Fill the Confusion Matrix
  3. 3. Calculate metrics

Accuracy = (TP + TN) / Total = (45 + 903) / 1,000 = 94.8%. Accuracy asks: "What fraction of ALL predictions was correct?" Sounds good — but it hides a problem.

Precision = TP / (TP + FP) = 45 / 92 ≈ 48.9%. Precision asks: "Of all POSITIVE predictions, how many were actually correct?" Less than half of the positive test results are actually true.

Recall = TP / (TP + FN) = 45 / 50 = 90%. Recall asks: "Of all ACTUALLY sick patients, how many did the test find?" 90% of sick patients are detected — that is good.

Precision: 48.9%

Of 92 positive test results, only 45 were actually sick. More than half are false alarms.

Recall: 90%

Of 50 actually sick patients, the test found 45. Only 5 were missed.

Analogy: The Fishing Net

A net that catches only fish with no bycatch has high Precision. A huge net that guarantees catching every fish in the lake (high Recall) also drags in a lot of debris (lower Precision).

Where the analogy breaks: it suggests Precision and Recall are mechanically linked (same net). In reality, a model can sometimes improve both simultaneously — for example, by using better features.

Caution

Warning: The Accuracy Paradox

On a dataset with 99% class A and 1% class B, a model that ALWAYS predicts "A" achieves 99% accuracy. Impressive? The model has zero Recall for class B — it is completely useless for the minority class.

The Accuracy Paradox is why you should never report Accuracy alone on imbalanced data. On imbalanced datasets, Accuracy lies.

Interactive: Match the Metrics

You now know TP, FP, FN, TN as well as Accuracy, Precision, and Recall. Test your knowledge with this memory game — match the terms to their definitions.

0/ 8 Pairs
0Attempts
0:00Time

The F1-Score: Balancing Precision and Recall

Definition

F1-Score

Analogy

Think of a school report card: a student scores 100% in math and 0% in language. The arithmetic average is 50% — a pass. But would you call this student well-educated? The harmonic mean gives 0%, correctly reflecting that one dimension is completely missing. The F1-Score works the same way — it punishes one-sided weakness mercilessly.

Extreme example: Model A has Precision = 99% and Recall = 1%. Arithmetic mean = 50% — looks acceptable. F1-Score = 2 x 0.99 x 0.01 / (0.99 + 0.01) = approximately 2%. The F1-Score immediately exposes the one-sided failure.

Our COVID test: Precision = 48.9%, Recall = 90%, F1-Score = approximately 63.4%. Much lower than the 94.8% Accuracy — the F1-Score shows the true picture.

Many classifiers output probabilities, not direct predictions. A threshold determines the decision: above 50%? Then "positive." Below 50%? Then "negative."

Lowering the threshold: more predictions become positive, so Recall goes up while Precision usually goes down. Raising the threshold: fewer predictions become positive, so Precision usually goes up while Recall goes down. It is like turning a volume knob on a stereo — except here you are adjusting the balance between two types of error.

The right metric depends on the real-world cost of errors:

  • Medical screening: prioritize Recall, while keeping false alarms at a bearable level. Missing a sick patient (FN) can be fatal.
  • Spam filter: maximize Precision. Losing a real email to the spam folder (FP) is a costly mistake.
  • Balanced datasets with similar error costs: Accuracy is fine.
  • General purpose: F1-Score as a balanced default when Precision and Recall are equally important.

Caution

Common Misconception: "The F1-Score is always the best metric"

The F1-Score weights Precision and Recall equally. But in medical screening, missing a disease (low Recall) is far worse than a false alarm (low Precision). In such cases, a metric that weights Recall more heavily fits better, such as the F2-Score, or a fixed minimum Recall below which the model is not deployed.

Threshold Lab: Confusion Matrix & ROC

Interactive: Explore the Metrics

The calculator works through the COVID example as a confusion matrix. Change the values: how does a small share of actual positives affect precision, accuracy and the F1 score? “Accuracy Paradox” loads a test that always says “healthy”.

Note

Key Takeaways

  1. The Confusion Matrix is the diagnostic surface of your model — it shows not just how often the model is wrong, but HOW it is wrong.
  2. Accuracy can mislead on imbalanced datasets. Precision and Recall reveal what Accuracy hides: the character of errors.
  3. The F1-Score balances Precision and Recall. Which metric matters most depends on the real-world cost of False Positives vs. False Negatives in your specific application.

Check

Quiz: Model Evaluation

Not completed

A medical test classifies a healthy person as sick. What type of error is this?

Select one answer

Answer Key: 1) C · 2) C · 3) C · 4) C

Check

Comprehension Check

Can you explain this in your own words? Tick what you've got.

0 of 3 done