How Good Is Your Model? Metrics That Actually Matter
Evaluating models without self-deception — metrics that do more than just look good.
Fundamentals 10 min Intermediate May 18, 2026
A model predicts "healthy" for every patient and achieves 99% accuracy. Impressive? Not if the 1% it misses are the patients who actually have the disease. Accuracy alone can be a dangerous lie — and the Confusion Matrix reveals the truth.
A single metric never tells the full story. In this article, you will learn to use the Confusion Matrix as a diagnostic tool and derive Accuracy, Precision, Recall, and the F1-Score from it. Which metric matters most is not a math question but a design decision.
The Confusion Matrix: The Foundation
Confusion Matrix
AnalogyDefinition
Think of a COVID test. "Positive" means the test says you are sick. "Negative" means the test says you are healthy. The test can be right in two ways (True Positive, True Negative) and wrong in two ways (False Positive = false alarm, False Negative = missed case). The Confusion Matrix is this test's report card.
Analogy:
Think of a COVID test. "Positive" means the test says you are sick. "Negative" means the test says you are healthy. The test can be right in two ways (True Positive, True Negative) and wrong in two ways (False Positive = false alarm, False Negative = missed case). The Confusion Matrix is this test's report card.
Definition:
The Confusion Matrix is a 2x2 table that classifies every prediction into four categories: True Positive (correctly predicted positive), True Negative (correctly predicted negative), False Positive (incorrectly predicted positive — Type I error), False Negative (incorrectly predicted negative — Type II error). All standard classification metrics are derived from these four numbers.
Imagine: 1,000 patients are tested. 50 are actually sick, 950 are healthy. The test produces the following results:
The COVID test analogy precisely shows how errors can go in two fundamentally different directions. The limitation: real-world classification problems can have more than two classes, and the medical framing can suggest equal error costs when they are often asymmetric.
45
True Positive (TP) Sick and correctly identified as sick — 45 patients
5
False Negative (FN) Sick but incorrectly classified as healthy — 5 patients
47
False Positive (FP) Healthy but incorrectly classified as sick — 47 patients
903
True Negative (TN) Healthy and correctly identified as healthy — 903 patients
All standard metrics — Accuracy, Precision, Recall, F1-Score — are simply different views of these four numbers. The Confusion Matrix is the diagnostic surface of your model.
Common Misconception: "High accuracy means I don't need the Confusion Matrix"
Two models with identical accuracy can have completely different error profiles. One produces many false alarms; the other misses many real cases. Only the Confusion Matrix shows HOW your model is wrong — not just HOW OFTEN.
Three Lenses: Accuracy, Precision, and Recall
1
Collect predictions
2
Fill the Confusion Matrix
3
Calculate metrics
Accuracy = (TP + TN) / Total = (45 + 903) / 1,000 = 94.8%. Accuracy asks: "What fraction of ALL predictions was correct?" Sounds good — but it hides a problem.
Precision = TP / (TP + FP) = 45 / 92 ≈ 48.9%. Precision asks: "Of all POSITIVE predictions, how many were actually correct?" Less than half of the positive test results are actually true.
Recall = TP / (TP + FN) = 45 / 50 = 90%. Recall asks: "Of all ACTUALLY sick patients, how many did the test find?" 90% of sick patients are detected — that is good.
Precision: 48.9%
Of 92 positive test results, only 45 were actually sick. More than half are false alarms.
Recall: 90%
Of 50 actually sick patients, the test found 45. Only 5 were missed.
Analogy: The Fishing Net
A net that catches only fish with no bycatch has high Precision. A huge net that guarantees catching every fish in the lake (high Recall) also drags in a lot of debris (lower Precision).
Where the analogy breaks: it suggests Precision and Recall are mechanically linked (same net). In reality, a model can sometimes improve both simultaneously — for example, by using better features.
Warning: The Accuracy Paradox
On a dataset with 99% class A and 1% class B, a model that ALWAYS predicts "A" achieves 99% accuracy. Impressive? The model has zero Recall for class B — it is completely useless for the minority class.
The Accuracy Paradox is why real-world ML papers rarely report only Accuracy. On imbalanced datasets, Accuracy lies.
Interactive: Match the Metrics
You now know TP, FP, FN, TN as well as Accuracy, Precision, and Recall. Test your knowledge with this memory game — match the terms to their definitions.
0/ 8 Pairs
0Attempts
0:00Time
The F1-Score: Balancing Precision and Recall
F1-Score
AnalogyDefinition
Think of a school report card: a student scores 100% in math and 0% in language. The arithmetic average is 50% — a pass. But would you call this student well-educated? The harmonic mean gives 0%, correctly reflecting that one dimension is completely missing. The F1-Score works the same way — it punishes one-sided weakness mercilessly.
Analogy:
Think of a school report card: a student scores 100% in math and 0% in language. The arithmetic average is 50% — a pass. But would you call this student well-educated? The harmonic mean gives 0%, correctly reflecting that one dimension is completely missing. The F1-Score works the same way — it punishes one-sided weakness mercilessly.
Definition:
The F1-Score is the harmonic mean of Precision and Recall: F1 = 2 x (Precision x Recall) / (Precision + Recall). Unlike the arithmetic mean, the harmonic mean punishes extreme imbalance. If either Precision or Recall is near zero, the F1-Score drops to near zero.
Extreme example: Model A has Precision = 99% and Recall = 1%. Arithmetic mean = 50% — looks acceptable. F1-Score = 2 x 0.99 x 0.01 / (0.99 + 0.01) = approximately 2%. The F1-Score immediately exposes the one-sided failure.
Our COVID test: Precision = 48.9%, Recall = 90%, F1-Score = approximately 63.4%. Much lower than the 94.8% Accuracy — the F1-Score shows the true picture.
Deep Dive: The Threshold Slider
Many classifiers output probabilities, not direct predictions. A threshold determines the decision: above 50%? Then "positive." Below 50%? Then "negative."
Lowering the threshold: more predictions become positive, so Recall goes up but Precision goes down. Raising the threshold: fewer predictions become positive, so Precision goes up but Recall goes down. It is like turning a volume knob on a stereo — except here you are adjusting the balance between two types of error.
Deep Dive: When to Use Which Metric?
The right metric depends on the real-world cost of errors:
Medical screening: maximize Recall. Missing a sick patient (FN) can be fatal.
Spam filter: maximize Precision. Losing a real email to the spam folder (FP) is a costly mistake.
Balanced datasets with similar error costs: Accuracy is fine.
General purpose: F1-Score as a balanced default when Precision and Recall are equally important.
Common Misconception: "The F1-Score is always the best metric"
The F1-Score weights Precision and Recall equally. But in medical screening, missing a disease (low Recall) is far worse than a false alarm (low Precision). In such cases, optimizing directly for Recall is more appropriate than the F1-Score.
Threshold Lab: Confusion Matrix & ROC
Lowering the threshold: more predictions become positive, so Recall goes up but Precision goes down. Raising the threshold: fewer predictions become positive, so Precision goes up but Recall goes down. It is like turning a volume knob on a stereo — except here you are adjusting the balance between two types of error.
Loading interactive demo ...
Interactive: Explore the Metrics
Adjust the parameters and observe how the Confusion Matrix values and metrics change. How does a low base rate affect Precision?
Scenario: Medical Test
A disease affects a certain proportion of the population. A test detects the disease with a certain hit rate, but also produces false-positive results in healthy people. How likely is the disease really when the test comes back positive?
Input Values
%
%
%
Examples:
Bayes' Calculation
P(B)= P(B|A) × P(A) + P(B|¬A) × P(¬A)
P(B)= 0.9500 × 0.0100 + 0.0500 × 0.9900 = 0.0590
P(A|B)= P(B|A) × P(A) / P(B)
P(A|B)= 0.9500 × 0.0100 / 0.0590 = 0.1610
Result
16.1%
P(A|B) — Probability of disease given a positive test
Before (Prior)
1%
After (Posterior)
16.1%
Bayes factor: The test has increased by a factor of 16.1 the probability (from 1% to 16.1%).
The Bayes Paradox
Although the test has 95% sensitivity, the probability given a positive result is only 16.1%. This is due to the low base rate (1%): Among 1,000 tested people, there are 50 false alarms but only 10 true hits. The false positives overwhelm the real cases.
Visualized: 1,000 People Tested
10
Sick & tested positive
(True Positive)
1
Sick & tested negative
(False Negative)
50
Healthy & tested positive
(False Positive)
941
Healthy & tested negative
(True Negative)
Of 60 positive tests, only 10 are actually sick. This gives P(A|B) = 10/60 = 16.1%.
Key Takeaways
The Confusion Matrix is the diagnostic surface of your model — it shows not just how often the model is wrong, but HOW it is wrong.
Accuracy can mislead on imbalanced datasets. Precision and Recall reveal what Accuracy hides: the character of errors.
The F1-Score balances Precision and Recall. Which metric matters most depends on the real-world cost of False Positives vs. False Negatives in your specific application.
Quiz: Model Evaluation
Question 1 / 4
Not completed
A medical test classifies a healthy person as sick. What type of error is this?
1. A medical test classifies a healthy person as sick. What type of error is this?
☐ A) True Positive
☐ B) True Negative
☐ C) False Positive
☐ D) False Negative
2. A Confusion Matrix shows: TP=80, FN=20, FP=10, TN=890. What is the Precision?
☐ A) 97%
☐ B) 80%
☐ C) 88.9%
☐ D) 90%
3. A fraud detection dataset contains 0.1% fraud cases. A model predicts "no fraud" for every transaction and achieves 99.9% accuracy. What is its Recall for the fraud class?
☐ A) 99.9%
☐ B) 50%
☐ C) 0%
☐ D) Cannot be determined
4. A hospital uses a screening model. Missing a disease (False Negative) is far more dangerous than a false alarm (False Positive). Which optimization strategy is most appropriate?
☐ A) Maximize Accuracy
☐ B) Maximize Precision
☐ C) Maximize Recall
☐ D) Maximize F1-Score
Answer Key: 1) C · 2) C · 3) C · 4) C
Comprehension Check
A COVID test has Accuracy = 94.8% but Precision = 48.9%. Explain why Accuracy paints a misleading picture here.
Calculate the F1-Score for a model with Precision = 80% and Recall = 40%. Compare the result with the arithmetic mean — why do the two values differ?
Name a concrete scenario where you would prioritize Recall over Precision, and explain why.