Extraordinary Claims

“A wise man… proportions his belief to the evidence.” – David Hume

“Extraordinary claims require extraordinary evidence.” – Carl Sagan

BASE RATE NEGLECT

Outcomes

Yes No

Test = Yes True Positive False Positive

Test = No False Negative True Negative

Base Rates

The base rate for any condition is simply the proportion of people who have the condition.

Base rate of dog owners in Hong Kong:

# of Dog Owners in Hong Kong÷

# of Hong Kongers

Base Rate

Yes No

Test = Yes True Positive False Positive

Test = No False Negative True Negative

As the base rate decreases, the number of true positives decreases.

Base Rate

Yes No

Test = Yes True Positive False Positive

Test = No False Negative True Negative

Importance of Base Rates

Therefore, the proportion of false positives out of total positives increases:

False Positive÷

False Positive + True Positive

Importance of Base Rates

And the proportion of true positives out of total positives decreases:

True Positive÷

False Positive + True Positive

What Does It Mean?

It means that even very good tests (low false positive rate, low false negative rate) cannot reliably detect conditions with low base rates.

Example

Suppose the police have a test for telling whether someone is driving drunk:• If you are drunk, the test returns “positive”

100% of the time.• If you are not drunk, the test returns

“positive” 95% of the time.The police randomly stop 1,000 drivers and test them.

The base rate of drunk drivers is:

# of drunk drivers÷

# of drunk drivers + # of sober drivers

High Base Rate

The base rate of drunk drivers is:

# of drunk drivers =500÷

# of drunk drivers =500 + # of sober drivers =500

Base rate = 50%

Outcomes

Drunk Not Drunk

Test = Yes True Positive False Positive

Test = No False Negative True Negative

Outcomes

Drunk Not Drunk

Test = Yes 500 x 100% 500 x 5%

Test = No 500 x 0% 500 x 95%

Outcomes

Drunk Not Drunk

Test = Yes 500 25

Test = No 0 475

Good Test!

Now we have 500 true positives and only 25 false positives:

False Positive÷

False Positive + True Positive

Good Test!

Now we have 500 true positives and only 25 false positives:

25÷

25 + 5005% =

Low Base Rate

The base rate of drunk drivers is:

# of drunk drivers =5÷

# of drunk drivers =5 + # of sober drivers =995

Base rate = 0.5%

Outcomes

Drunk Not Drunk

Test = Yes True Positive False Positive

Test = No False Negative True Negative

Outcomes

Drunk Not Drunk

Test = Yes 5 x 100% 995 x 5%

Test = No 5 x 0% 995 x 95%

Outcomes

Drunk Not Drunk

Test = Yes 5 50

Test = No 0 945

Bad Test!

Now we have 5 true positives and 50 false positives:

False Positive÷

False Positive + True Positive

Bad Test!

Now we have 5 true positives and 50 false positives:

50÷

50 + 591% =

We saw this same phenomenon when we learned that most published scientific research is false.

Low base rate of truths.

High proportion of false positives.

Base Rate Neglect Fallacy

The base rate neglect fallacy is when we ignore the base rate.

The base rate is very low, so our tests are very unreliable… but we still trust the tests.

It Matters

The Hong Kong police stop people 2 million times every year.

Only 22,500 of these cases (1%) are found to be offenses.

Even if the police have a very low false positive rate it’s clear that the base rate of offenses does not warrant these stops.

PRIOR PROBABILITIES

Probabilities

A base rate is a rate (a frequency). # of X’s out of # of Y’s. # of drunk drivers out of # of total drivers.

Sometimes a rate or a frequency doesn’t make sense. Some things only happen once, like an election. So here we talk about probabilities instead of rates.

Probabilities

For things that do happen a lot the probabilities are (or approximate) the frequencies:

Probability of random person being stopped by police

=# of people stopped by police ÷ total # of people

Things that Only Happened Once

• The 2012 Chief Executive Election• The 1997 Handover• The great plague of Hong Kong• The second world war• The extinction of the dinosaurs• The big bang

Evidence

Drunk tests are a type of evidence. Testing positive raises the probability that you are drunk. But how much does it raise the probability? That depends on the base rate.

Evidence

You can also have evidence for or against one-time events. For example, you might have a footprint at the scene of a crime. This raises the probability that certain people are guilty. How much? That depends on the prior probability.

Priors and Posteriors

Prior probability = probability that something is true before you look at the new evidence.

Posterior probability = probability that something is true after you look at the new evidence.

Relativity

Priors and posteriors are relative to evidence. Once we look at new evidence, our posteriors become our new priors when we consider the next evidence.

Bayes’ Theorem

P (data/ hyp.) x P(hyp.)÷

P(data)P(hypothesis/ data) =

PosteriorPrior

Prior/ Posterior Positively Correlated

P(hypothesis/ data) [P(data/ hyp.) x P(hyp.)]∝

What Does It Mean

If the prior probability of some event is very low, then even very good evidence for that event does not significantly increase its probability.

“Extraordinary claims require extraordinary evidence.” – Carl Sagan

HUME ON MIRACLES

“A wise man… proportions his belief to the evidence.” – David Hume

Novel Testimony

Suppose that we get testimony concerning something we have never experienced.

Hume imagines someone from the equatorial regions being told about frost, and snow, and ice. They have never experienced anything like that before.

It’s Strange!

Hume thinks this person would have reason to disbelieve stories about a white powder that fell from the sky, covered everything by several inches, and then turned to water and went away.

It’s not that they should believe the stories are not true, just that they don’t have to believe they are true. We need more evidence, because the prior is so low.

But nowsuppose someone tells us an even stranger story.

It’s like the snow-story, in that we’ve never experienced anything like it before. But it’s even stranger, because we have always experienced the opposite before.

Miracles

For Hume, this is the definition of a miracle. A miracle is a violation of the laws of nature. Every event or process in the world conforms to the laws of nature (for example, the laws of physics like the law of gravity)– except, if there are any, miracles.

Example

There are about 100 billion people who have lived and died in the history of humanity (and there are 7 billion more who are alive now).

As far as we know, none of the 100 billion people who have ever died and were dead for four days, later came back to life. It’s a law of nature that when you die, that’s the end, there’s no more.

Lazarus

Although there is testimony, in at least one religious book– the Christian bible– that such an event occurred at least once in history, when Jesus raised Lazarus from the dead, after he had been dead for four days.

What Should We Believe?

According to Hume, we should be wise and apportion our belief to the evidence.

Since on the one hand we have 100 billion people who died and never came back, and on the other hand we have an old legend from a book intended to make people believe its religious views, it’s most probable that the raising of Lazarus never happened.

Hume on Miracles

“No testimony is sufficient to establish a miracle, unless the testimony be of such a kind that its falsehood would be more miraculous than the fact which it endeavors to establish.”

Seeing and Believing

So, for example, Hume would even say that if you saw someone die and come back to life, you should not believe that it really happened.

Seeing and Believing

Because it’s always possible that what you saw was a trick, or the person was never really dead, or you were on drugs or…

Since none of those suppositions are miraculous, you should believe them instead of believing in the miracle. They’re more likely than a violation of nature’s laws.

NO MORE PHILOSOPHY!

OK, Back to Science…

There’s a debate among scientists about Evidence Based Medicine vs. Science Based Medicine.

They sound the same, but they’re very different!

Modern Medicine

In current modern medicine the following is (one) best estimate:• 37% of treatments are based on Randomized

Controlled Trials• 76% of treatments are based on good

evidence (RCTs, observational studies)• The rest should be based on scientific theory

(reasonable extension of what we know).

Evidence Based Medicine

One idea is that the 76% of tested-treatments are the “real” evidence based medicine and the rest is no better than untested alternative medicine. These are equal:• Treatments based on scientific theory

(reasonable extension of what we know).• Untested pre-scientific or otherwise

alternative treatments (e.g. homeopathy).

Difficult Tests

Some alternative treatments are difficult to test.

Homeopaths claim that their treatments are individualized. So it’s not enough to give everyone suffering from a disease the same magic water… they have to come into the shop for a personalized experience.

Can’t Placebo a Whole Shop!

False Equivalence

This means we should let the homeopaths “get away with it.” Sure, their treatments aren’t supported by science, but neither are 24% of modern treatments.

Science Based Medicine

Science based medicine, on the other hand, says we should take into account prior probability.

We have lots of scientific knowledge of water. Nothing about it says that chemically pure water that in the past contained other chemicals and was then shaken should behave any differently than regular chemically pure water

Science Based Medicine

And, science based medicine says that the 24% of treatments that are not evidence based, while they should still be tested, are much better because of prior probability.

If science tells us why they should work, then we should believe the science even if we haven’t tested them (yet) or can’t test them.

Example

It’s immoral not to perform blood transfusions on people who have lost lots of blood.

So we can’t do a RCT on blood transfusions.

But that doesn’t mean they’re as silly as homeopathy.