Bayes & Conditional Probability
Conditional probability: the tool that identifies statisticians — they do the math instead of guessing.
A medical test is 95% accurate. The disease affects 1% of people. You test positive. How likely are you to actually be sick? The most common answer: "95%." The real answer: about 16%. This is not a trick — it is Bayes' theorem in action.
Your gut feeling is often way off with probabilities — especially when rare events are involved. This article walks you through three tools: conditional probability (what "given that" means mathematically), Bayes' theorem (the formula for reversing probabilities), and Bayesian updating (how AI systems learn from data).
Conditional Probability — "Given That..."
Definition
Conditional Probability
Analogy
Example
Definition
Example: The Medical Test
1% of the population has disease X. The test detects sick people 95% of the time (sensitivity). In healthy people, it falsely shows positive 5% of the time. Take 1,000 people:
- 1. 1,000 people: 10 sick, 990 healthy (completed)
- 2. Of 10 sick: 9 test positive95% would be 9.5 (completed)
- 3. Of 990 healthy: 50 false positives5% would be 49.5 (completed)
- 4. Total positive: 9 + 50 = 59 (completed)
- 5. P(sick | positive) = 9/59 ≈ 15%without rounding to whole people: 16% (current step)
You test positive. The probability of actually being sick: only 16%. The 990 healthy people produce 50 false alarms that flood the 9 true positives. The base rate (1% prevalence) dominates.
Caution
The Base Rate Fallacy
Do not confuse P(positive | sick) with P(sick | positive). Test accuracy (how well the test detects the sick) and diagnostic probability (how likely you are sick given a positive test) are completely different things. The base rate — how rare the disease is — makes all the difference.
Caution
Misconception: 95% Accurate Test = 95% Probability of Being Sick
95% describes P(positive | sick) — how well the test finds sick people. But you want P(sick | positive) — how likely you are sick. Without the base rate (how common the disease is), the accuracy number is meaningless. For rare diseases, even excellent tests produce mostly false alarms.
Bayes' Theorem — The Formula for Updating Beliefs
Definition
Bayes' Theorem
Analogy
Example
Definition
The Four Building Blocks
Worked Example
P(sick | positive) = P(positive | sick) x P(sick) / P(positive)
= 0.95 x 0.01 / 0.059
= 0.0095 / 0.059
≈ 0.161 → about 16.1%
P(positive) = P(positive | sick) x P(sick) + P(positive | healthy) x P(healthy)
= 0.95 x 0.01 + 0.05 x 0.99
= 0.0095 + 0.0495
= 0.059Count 1,000 people, divide positive-tested sick by all positives. Intuitive, but only practical with small numbers.
Prior x Likelihood / Evidence. Same answer (16%; the table shows 15% only because it rounds to whole people), but universally applicable — even when you cannot count 1,000 people.
Caution
Misconception: Strong Evidence Overrides Everything
Even if P(B|A) is high — the likelihood alone does not determine the result. With an extremely low prior (e.g., a disease affects 0.01%), the posterior stays low even with a 99% test. The prior always pulls the result in its direction. Only with moderate priors or multiple independent pieces of evidence can the evidence overcome the prior.
Interactive: Compute Bayes' Theorem
Adjust the prior, sensitivity, and false positive rate and watch how the posterior changes. Try the predefined scenarios from the article (rare disease, spam filter) and experiment with your own values.
Bayesian Updating — Learning Step by Step
Definition
Bayesian Updating
Analogy
Example
Definition
Example: Spam Filter (Naive Bayes)
A spam filter classifies emails using word frequencies. Prior: P(Spam) = 0.4, P(Ham) = 0.6. The email contains the word "winner":
P("winner" | Spam) = 0.8
P("winner" | Ham) = 0.05
P(Spam | "winner") = (0.8 x 0.4) / (0.8 x 0.4 + 0.05 x 0.6)
= 0.32 / 0.35
≈ 0.914 → 91.4% spam probabilityIf the email also contains "click," another Bayes update further increases the spam probability. Each word is a piece of evidence. This is Naive Bayes — "naive" because it assumes words are independent given the class. A simplification that works remarkably well in practice.
Python: Naive Bayes
from sklearn.naive_bayes import MultinomialNB
# Training features (word frequencies)
X_train = [[5, 1, 0], [4, 2, 0], [0, 1, 3], [1, 0, 4]]
y_train = ['spam', 'spam', 'ham', 'ham']
clf = MultinomialNB()
clf.fit(X_train, y_train)
# Classify a new email
clf.predict([[3, 1, 0]]) # → ['spam']Caution
Misconception: The Prior Does Not Matter — Just Let the Data Speak
With large datasets, yes — the prior's influence fades. But with limited data (common in practice: medical imaging, rare events, small user bases), the prior can dominate the posterior. Choosing a thoughtful prior is not unfair bias — it is informed reasoning.
Training an AI model follows a similar pattern: the prior is the model's random initial weights. The evidence is the training data. The posterior is the trained model. Each training batch updates the weights — much like Bayes updates the posterior with each new piece of evidence. Strictly speaking, though, this is not Bayesian: ordinary training looks for a single good set of weights rather than a probability distribution, and a real prior lives in regularization (such as weight decay) rather than in the random weights. So Bayes helps you understand how AI learns — as a way of thinking about it, not as a blueprint.
Try it: The Coin Detective
The calculator above shows a single Bayes computation. Here you live the process: flip a hidden coin and watch your belief update flip by flip — from prior knowledge, through the evidence, to a new conviction (the posterior).
Note
Key Takeaways
- P(A|B) ≠ P(B|A) — confusing the direction of conditioning is one of the most common probability errors; ignore the base rate as well and you fall for the Base Rate Fallacy.
- Bayes' formula P(A|B) = P(B|A) x P(A) / P(B) combines prior knowledge (prior) with new observation (likelihood) to produce the updated belief (posterior).
- Bayesian updating is iterative: each posterior becomes the new prior. With little data, the prior dominates; with lots of data, the evidence dominates. Spam filters and medical diagnostics use exactly this principle; AI training resembles it.
Check
Quiz: Bayes & Conditional Probability
What does P(A|B) mean?
Select one answer
1. What does P(A|B) mean?
2. Why are most positive test results wrong for rare diseases?
3. A spam filter has prior P(Spam) = 0.5. An email contains "winner": P("winner" | Spam) = 0.6, P("winner" | Ham) = 0.1. What is P(Spam | "winner")?
4. An AI model starts with random weights and is trained with data. What role do the initial weights play in the Bayesian picture from the deep dive?
Check
Checkpoint: Bayes & Conditional Probability
Can you explain this in your own words? Tick what you've got.
0 of 3 done