DEV Community

Astra Quant
Astra Quant

Posted on

Bayes in 15 Lines: Why a 99% Accurate Test Can Still Be Mostly Wrong

A classic puzzle: a condition affects 1 in 1,000 people. A test catches 99% of real cases and wrongly flags 1% of healthy people. You test positive. How likely is it that you actually have the condition?

Most people guess around 99%. The real answer is under 10%. Here is the whole thing in a few lines of Python.

def posterior(prior, sensitivity, false_positive_rate):
    true_pos = prior * sensitivity
    false_pos = (1 - prior) * false_positive_rate
    return true_pos / (true_pos + false_pos)

p = posterior(prior=0.001, sensitivity=0.99, false_positive_rate=0.01)
print(f"P(condition | positive) = {p:.1%}")   # about 9.0%

# A second, independent positive test: use the first result as the new prior
p2 = posterior(prior=p, sensitivity=0.99, false_positive_rate=0.01)
print(f"After two positives = {p2:.1%}")       # about 90.8%
Enter fullscreen mode Exit fullscreen mode

Why it happens

Picture 100,000 people:

  • 100 have the condition, and the test flags 99 of them.
  • 99,900 are healthy, and the test still flags 1% of them: 999 people.

So out of 1,098 positives, only 99 are real. That is about 9%. The test is not bad. The condition is just rare, and false positives from the huge healthy group swamp the true ones.

The takeaway for engineers

The same trap shows up everywhere we build detectors: fraud alerts, anomaly monitors, spam filters, flaky test triage. A "99% accurate" alert on a rare event will mostly page you for nothing. Before trusting a signal, ask two questions:

  1. How rare is the thing I'm looking for (the base rate)?
  2. How often does the detector fire when nothing is wrong?

Then run the 15 lines. Your intuition will thank you.

What is the most surprising base-rate trap you have hit in production?

Top comments (0)