Bayes' Theorem and Conditional Probability

🧠

Bayes' Theorem is arguably the single most important mathematical formula for understanding how we should update our beliefs in the face of new evidence. It forms the foundation of modern machine learning and artificial intelligence.

The Core Concept: Conditional Probability

Before Bayes, we must understand Conditional Probability. It asks: "What is the probability of event A happening, given that we already know event B has happened?"

We write this as $P(A \mid B)$ (read as "Probability of A given B").

P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}

Enter Bayes' Theorem

Bayes' Theorem provides a rigorous way to reverse conditional probabilities. It allows us to compute $P(A \mid B)$ if we know $P(B \mid A)$ .

⚠️

Crucial Distinction: $P(A \mid B)$ is not the same as $P(B \mid A)$ .

The probability of having a cough given you have a cold $P(\text{Cough} \mid \text{Cold})$ is extremely high.
The probability of having a cold given you have a cough $P(\text{Cold} \mid \text{Cough})$ is much lower (you could have allergies, dust, asthma, etc.).

The beautiful formula is:

P(A \mid B) = \frac{P(B \mid A) \cdot P(A)}{P(B)}

Where:

$P(A)$ - The Prior: Your initial belief before seeing any evidence.
$P(B \mid A)$ - The Likelihood: How probable is the evidence, given that your belief is true?
$P(B)$ - The Evidence (Marginal Likelihood): The total probability of observing the evidence under all possible circumstances.
$P(A \mid B)$ - The Posterior: Your newly updated belief after seeing the evidence.

The Base Rate Fallacy (Medical Testing Example)

The most mind-blowing application of Bayes' rule happens in medical testing.

Imagine a disease that affects 1% of the population (The Prior). We have a highly accurate test for it:

If you have the disease, the test is 90% likely to correctly catch it (Sensitivity).
If you are completely healthy, the test is only 9% likely to incorrectly say you have it (False Positive).

You take the test. It comes back POSITIVE.

😨

Intuition Trap! Most doctors and patients guess that you have a 90% chance of having the disease. Let's see what Bayes' rule actually says...

P(\text{Disease} \mid \text{Positive Test}) = \frac{P(\text{Positive} \mid \text{Disease}) \cdot P(\text{Disease})}{P(\text{Positive Total})}

= \frac{0.90 \cdot 0.01}{(0.90 \cdot 0.01) + (0.09 \cdot 0.99)} = \frac{0.009}{0.009 + 0.0891} = \frac{0.009}{0.0981} \approx 9.17\%

Wait, what?! Even with a positive test, you only have a 9.17% chance of actually having the disease!

Why? Because the disease is so rare (1%) compared to the false positive rate (9%). The vast majority of positive tests will be healthy people triggering false positives.

See It To Believe It

Use the interactive simulation below. Drag the "Disease Rarity" slider to see how heavily the Base Rate impacts the actual truth of your test results!

Bayes’ theorem (medical testing)

If you test positive, what’s the chance you actually have the disease?

Positive predictive value

9.2%

Positive test rate: 9.81% of the population

Disease prevalence

1.0%

Base rate in the population.

Sensitivity

90%

If sick, chance of a positive test.

False positive rate

If healthy, chance of an incorrect positive.

Test Your Knowledge

Example: The Spam Filter (Bayes' Rule)

Suppose 20% of all emails are spam. A spam filter correctly flags 90% of spam emails as "Spam", but also incorrectly flags 5% of regular emails as "Spam". If you see an email in your Spam folder, what is the probability it is actually spam?

View Step-by-Step Solution

Let $S$ = is spam, $F$ = flagged as spam. We want to find $P(S|F)$ .

$P(S) = 0.20$ (Prior)
$P(\neg S) = 0.80$
$P(F|S) = 0.90$ (True Positive Rate)
$P(F|\neg S) = 0.05$ (False Positive Rate)

Total probability of being flagged: $P(F) = P(F|S)P(S) + P(F|\neg S)P(\neg S) = (0.90 \times 0.20) + (0.05 \times 0.80) = 0.18 + 0.04 = 0.22$

Apply Bayes: $P(S|F) = \frac{P(F|S)P(S)}{P(F)} = \frac{0.18}{0.22} \approx 0.818$

Even though the filter is 90% accurate, an email in the spam folder only has an 81.8% chance of being actual spam.

Independence Math Descriptions