v i r g i n i a c o m m o n w e a l t h u n i v e r s i t y the scientific method and statistics...

V I R G I N I A C O M M O N W E A L T H U N I V E R S I T Y

The Scientific Method and Statistics

Critical Appraisal Skills depend upon an understanding of the scientific method

and the role statistics plays

Al Best, PhDPerkinson 3100B

[email protected]


Critical Appraisal Skills Are the results of the

study valid? What are the results? Will the results help

locally?


My goals for you

Be able to answer four questions: Based on the study design, what is

the level of evidence? How were threats to validity

addressed? Based on the goals of the study, How

do you describe the results? To justify the conclusions, were

comparisons done appropriately?


Stats 2: Outline Science: Using data to answer questions Case-control study (example)

– Estimate prevalence in a population Sampling, measurement, randomness Estimates and Confidence Intervals

– Compare cases and controls Hypothesis testing P-value, confidence interval

– Interpret the results Prevalence difference


A Scientist’s Quandary

Are the results of the study valid?– Most experiments are highly local but

have general aspirations.– How can findings generalize to other

people, in other settings, with comparable interventions, and other outcomes.

How will you assess whether the paper’s findings will generalize to your situation? – This is the question of external validity.


Solution?

Use a process where sample data does generalize to the units, treatments, variables and settings not directly observed.

Follow the process called Statistical Inference using the Scientific Method.


Statistical Inference

Inference

Measurement

Population

Sampling

Sample

Data


Example: Barasch’s “Risk Factors for

Osteonecrosis of the Jaws” “We conducted a case-control study in

dental practices to determine the risk associated with bisphosphonates and to identify other risk factors for ONJ,…”

From the introduction of Barasch, et al. (2011) J Dent Res 90(4), 439-444. pubmed/21317246

http://www.ncbi.nlm.nih.gov/pubmed/21317246

http://www.ncbi.nlm.nih.gov/pubmed/21317246


Classic Steps: Inference using Statistics

1. What’s the question? (Introduction)– Conceptualize the population– State the question

2. How will you answer the question? (Methods)– The sample– The measurements– Analysis technique

4. What does it mean? (Discussion)

3. Answer the question (Results)– The sample– The measurements– Analysis technique


“a case… study … of the risk associated with BP and ONJ”

Conceptualize the population– BP or ONJ?

State the question: Which?– In ONJ cases, estimate the prevalence of

BP use.Or

– In BP cases, estimate the prevalence of ONJ.

Abbreviations: BP=bisphophonates, ONJ=osteonecrosis of the jaw



Main question: Compare– the prevalence of BP use higher in ONJ

casesto

– the prevalence of BP use higher in controls.

Design a measurement system– ONJ = case or control– BP = use or non-use



Design a measurement system– ONJ case

Section C. Osteonecrosis of the Jaw: Natural History I am going to ask you a few questions regarding the condition we are studying in this research. The name of this condition is osteonecrosis of the jaw and information about it was mailed to you with the consent form you signed to agree to participate in this interview.

(FOR CASES) Your dentist informed us that you have Osteonecrosis of the Jaw, but we need to confirm this information with you. (FOR CONTROLS) Your dentist informed us that you do not have Osteonecrosis of the Jaw. We need to confirm this information with you.

1. Do you currently or have you ever had an area of exposed bone in your mouth? This condition, where the bone becomes exposed and does not heal, is known as osteonecrosis of the jaw. This condition was described in the informational brochure that was mailed to you and you may refer to this information if needed. _ Yes _ No


Segue: Data

Data is information in context.

That is, data is the set of information—usually numbers—arising out of measurements of individuals.

Part of the information in data is also its context—how did this information come about?

Tonight we’re going to let the statistics speak for themselves

Ed Koren, © The New Yorker, 9 December 1974.


Segue: Data classification

Type of data

Distinguishing Characteristics Examples

Categoricalor qualitative

Observations grouped into distinct classes

Nominal Classes without a natural order or rank

Sex, treatment group, presence or absence

Ordinal Classes with a predetermined or natural order

Disease severity, bone density, plaque accumulation, bleeding


Segue: Data classificationType of data

Distinguishing Characteristics Examples

Continuous or quantitative (numeric)

Observations may assume any value on a continuous scale

Interval Numeric value with equal unit differences

Temperature, GPA, age, duration of disease, income amount, serum cholesterol, number of decayed teeth

Synonyms: continuous, interval



Design a measurement system– ONJ case

What type of data is this?– Nominal or

Ordinal orInterval (continuous)?



Measure the subjects– BP use

Section H. MedicationsNow I would like to ask you about some of the medications that you have taken during your lifetime. This is the last section of the interview. It would be helpful to use the sheets that we sent you in our last letter.1. Have you EVER taken any of the following drugs orally or BY MOUTH?


Looking Ahead

BPexposure Cases Controls Total %Yes BIG N small n ? ?No small n BIG N ? ?Total 170 3*170 680 100.0

ONJ



Measure the subjects– BP use

What type of data is this?– Nominal or

Ordinal orInterval?


Backing up: State the question In ONJ cases, estimate the prevalence of

BP use. The population has a parameter—call it π

—we are trying to estimate this using data.

Inference

Measurement

Population

Sampling

Sample

Data


The population has a parameter—call it π

Conceptualization: The population True:nONJ = true count of everyone who has

ONJ True:nBP = true count of everyone who has

ONJ and also used BP π = true prevalence proportion of BP in ONJ

patients.π = True:nBP / True:nONJ


Estimation of population parameter using sample

statistic Definition: A statistic is a single descriptive number computed from the data.

Conceptualization: The sample nONJ = count in sample who have ONJ

nBP = count in sample who have ONJ and also used BP

p = estimated prevalence proportion of BP

in ONJ patients.p = nBP / nONJ


Backing up: State the question In ONJ cases, estimate the prevalence of

BP use.

D

Inference

Measurement

Population

Sampling

Sample nnOONNJJ

Data pp == nnBBPP // nnOONNJJ

π = True:nBP / True:nONJ


Classic Steps

What’s the question? (Introduction)– Conceptualize the population– State the question

How will you answer the question? (Methods)– The sample– The measurements– Analysis technique

What does it mean? (Discussion)

Answer the question (Results)– The sample– The measurements– Analysis technique


Actual nONJ? Actual nBP?

Analyze the data– BP use in ONJ patients– nONJ = count in sample who have ONJ

– nBP = count in sample who have ONJ and also used BP

p = estimated prevalence proportion of BP in ONJ patients.p = nBP / nONJ


Estimate of p?

Analyze the data– BP use

– From sample nONJ=191, in these nBP=113p = 113/191 ?= 83% = proportion 0.83

– Point estimate of p = 0.83


Segue: estimation error In ONJ cases, estimate the prevalence of

BP use.

Inference

Measurement

Population

Sampling

Sample NNOONNJJ

Data pp == nnBBPP // nnOONNJJ

Variability due to sampling:308 cases down to 191

Variability due to measurement:nBP = 113nNoBP = 24Unknown = 24

.825=113/137


Answer the question In ONJ cases, estimate the prevalence of

BP use. The population has a parameter—call it π—

we are trying to estimate—using data.

Inference

Measurement

Population

Sampling

Sample

Data

In ONJ patients, 83% reported BP use


Classic Steps






Defining the research question:Testable consequence?

Conceptual progression from general to specific

General question– Is Bisphosphonate use a risk factor for

ONJ? Specific hypothesis

– Is the prevalence of bisphosphonate use higher in ONJ cases than in controls?

Testable consequence– Prediction of a relationship– Potentially refutable by data


Defining the research question: Prediction

Prediction: a statistical relationship between exposure and outcome

–BP prevalence will be higher in ONJ cases than in controls

How do we arrive at this? Using a refutable hypothesis


Defining the research question: Formalization

A refutable hypothesis Statistical formalization: Ho: proportion BP (ONJ) =

proportion BP (controls)– Which may be disproved beyond a

reasonable doubt through falsification by data via statistical hypothesis testing, in favor of:

Ha: proportion BP (ONJ) > proportion BP (controls)


Critical Appraisal

Is it a testable, research question? How did they try to rule out bias,

confounding, chance? How did they consider multiple outcome

measures and multiple predictors? Did they disclose what was done with

enough detail so others may replicate?


How Science Advances Clinical Knowledge

Science forms a question and brings data to bear to answer the question.

Informally:1. Frame a clinical research question.2. State its testable consequences as either

“just random variability” or “unusual outcomes”.

3. Compare the actual data with these two choices and decide which to believe.

4. Discuss our present understanding.


Testing Hypotheses

Or, linking the four steps to the standard IMRD organization of a paper:1. What’s the question? (Introduction)2. How do you answer the question?

(Methods)3. Answer the question. (Results)4. What does it mean? (Discussion)


What’s the Question?

“We conducted a case-control study in dental practices to determine the risk associated with bisphosphonates and to identify other risk factors for ONJ,…”

From the introduction of Barasch, et al. (2011) J Dent Res 90(4), 439-444.


How do you answer the question?

Outline– Propose two states of nature– Use the rule of simplicity– Take into account that “noise happens”– Use a test statistic to decide: Signal or

Noise?


Two states; Two hypotheses We begin by conceiving the true state of

nature as being either: – no difference or – a difference.

We always start by assuming that nothing is going on—that any apparent differences are purely because of chance. Our preference, as scientists, is to believe the simplest explanation for a phenomenon.– Assume: no difference (AKA null

hypothesis).


Rule of Simplicity When you have two competing theories

which make exactly the same predictions, the one that is simpler is the better.

The simplest explanation for some phenomenon is more likely to be accurate than more complicated explanations.

The explanation requiring the fewest assumptions is most likely to be correct.

AKA “Occam’s Razor”


Statistical World View Thou shalt not interpret randomness. Chance happens. Noise exists.

Making an interpretation that goes beyond this requires justification.

If random noise, measurement error, or chance occurrence can account for variations (differences) in the observations, then there is no need to formulate a more complicated explanation.– We embody this preference in the

statement of the null hypothesis.


Null Hypothesis We evaluate this proposition using

statistical techniques. The null hypothesis is the statement that

is tested. It’s abbreviated H0:

A null-hypothesis is the simplest explanation of events: There is no difference. There is no change. There is no improvement. Nothing unusual is occurring.

A null-hypothesis is the statement we hope to contradict with data. That is, we usually hope to reject the null hypothesis.


Assume: Nothing is going on

Prevalence of bisphosphonate use within those who do have ONJ is equal to the prevalence of bisphosphonate use within those who do not have ONJ.

HO: πcases = πcontrols


Two Hypotheses Prevalence of bisphosphonate use within those

who have ONJ is equal to the prevalence of bisphosphonate use within those who do not have ONJ.

HO: Pcases = Pcontrols

Can we reject the above, in favor of: Prevalence of bisphosphonate use within those

who have ONJ is different than the prevalence of bisphosphonate use within those who do not have ONJ.

HA: Pcases ≠ Pcontrols So: Done with step 1: We’ve stated the question.Next: How will you answer the question?


Proof By Contradiction Nature is either in one state or the other.

– We prefer to believe the simplest explanation.

Collect data from the real world. Assess the likelihood of observing this data

under the null hypothesis. Choose to believe:

– If the data is within what we would expect then we retain our preference for the null-hypothesis as the best explanation.

– If the data is very unlikely, then we reject the null hypothesis in favor of its alternative.


What if: HO is true? In ONJ cases, estimate the prevalence of

BP use. In controls, estimate the prevalence of BP

use.

Inference

Measurement

Population

Sampling

nnOONNJJ nnccoonnttrroollss

ppOONNJJ == nnBBPP,,OONNJJ // nnOONNJJ

ppccoonnttrrooll == nnBBPP,,ccoonntt.. // nnccoonntt

π = πONJ= πcontrol


What if?

Assess the likelihood of observing various data possibilities under the null hypothesis.

Assume this is true:– HO: Pcases = Pcontrols

Then the sample estimate of Pcases will be “close to” the sample estimate of Pcontrols.

– By “close to” we mean that, because of sampling variability and measurement error we expect them to be somewhat different.


Contingency Table

“Results: … Therefore, 191 cases together with 573 controls were included in the analyses….Bisphosphonate use was reported by 113 cases (83%) and 71 controls (15%),…” Note: 113+71=184.

BPexposure Cases Controls Total %Yes 184 30.2No 426 69.8Yes 137 473 610 100.0

ONJ

473137

Total


Contingency Table

Assume this is true:– HO: Pcases = Pcontrols = 0.302

BPexposure Cases Controls Total %Yes ? ? 184 30.2No ? ? 426 69.8Yes 137 473 610 100.0

ONJ

Total


Contingency Table

Assume this is true:– HO: Pcases = Pcontrols = 0.302– 30.2% of 137= 41

BPexposure Cases Controls Total %Yes 184 30.2No 426 69.8Yes 137 473 610 100.0

ONJ

Total


Contingency Table

Assume this is true:– HO: Pcases = Pcontrols = 0.302– 30.2% of 473 = 143

BPexposure Cases Controls Total %Yes 41(30%) 184 30.2No 426 69.8Yes 137 473 610 100.0

ONJ

Total


Contingency Table

Assume this is true:– HO: Pcases = Pcontrols = 0.302– 137 – 41 = 96– 473 – 143 = 330

BPexposure Cases Controls Total %Yes 41(30%) 143(30%) 184 30.2No 96(70%) 330(70%) 426 69.8Yes 137 473 610 100.0

ONJ

Total


Test Statistic

Test stat: Difference in prevalence – HO: Pcases – Pcontrols = 0.0– Expected difference = 0%


ONJ

Total


Testing Hypotheses

Recall:– Frame a clinical research question.– State its testable consequences as

either “just random variability” or “unusual outcomes”.

– Compare the actual data with these two choices and decide which to believe.

– Discuss our present understanding.


I presume the null-hypothesis is true, do the data support

this? Observed difference = 0% P-value = 1.0


ONJ

Total



this? Observed difference = 0.6% P-value = 0.885


ONJ

Total



this? Observed difference = 1.6% P-value = 0.724

BPexposure Cases Controls Total %Yes 43(31%) 141(29.8%) 184 30.2No 94(69%) 332(70.2%) 426 69.8Yes 137 473 610 100.0

ONJ

Total



this? Observed difference = 90% P-value < 0.001


ONJ

Total


Trade offs

Alpha = Type I error = prob. of rejecting a true null hypothesisBeta = Type II error = prob. of not finding a true difference

Conclusion

Truth

Do not reject null-hypothesis

(p-value > .05)

Reject null-hypothesis (p-value < .05)

Null-hypothesis (no difference) correct Type I error

Alternative hypothesis (difference) Type II error correct


Significance Level

The significance level is represented by the Greek symbol “alpha”, α.

It is the probability of rejecting a true null hypothesis.

The researcher chooses the risk of making this error: concluding that the null hypothesis is false when it really is true. – The most typical values are α = .05, .01,

or .10.


Universal Decision Rule

Choose to believe: HO: null-hypothesis

(For non-extreme values of the test statistic)

– Choose this if p-value ≥ α (usually 0.05) HA: alternative-hypothesis

(For extreme values of the test statistic)

– Choose this if p-value < α (usually 0.05)


! Are we done yet? !







this? Observed difference = 67.5% P-value < .0001

BPexposure Cases Controls Total %Yes 113(82.5%) 71(15%) 184 30.2No 24(17.5%) 402(85%) 426 69.8Yes 137 473 610 100.0

ONJ

Total


Chi-square test Expected data

Observed data (chi-square, P-value < .0001)


ONJ

BPexposure Cases Controls Total %Yes 113(82.5%) 71(15%) 184 30.2No 24(17.5%) 402(85%) 426 69.8Yes 137 473 610 100.0

ONJ

Total

Total


State a Conclusion Prevalence of bisphosphonate use within

those who have ONJ is equal to the prevalence of bisphosphonate use within those who do not have ONJ.– HO: Pcases = Pcontrols

Prevalence of bisphosphonate use within those who have ONJ is different than the prevalence of bisphosphonate use within those who do not have ONJ.– HA: Pcases ≠ Pcontrols

Evidence: 82.5% prevalence vs 15%, p-value <.0001


P-value

The p-value is the probability that the data occurred by chance, assuming the null hypothesis is true.

The p-value is NOT the probability that the null-hypothesis is true.


The p-value is NOT the probability that the null-hypothesis

is true.

and:1−pvalue is NOT the probability that the

alternative hypothesis is true


Trade offs


Conclusion

Truth


(p-value > .05)


Null-hypothesis (no difference) correct Type I error

Alternative hypothesis (difference)

Type II error correct


Actuality


Results

Truth


(p-value > .05)


Null-hypothesis (no difference) Blind alley ?

Alternative hypothesis (difference) ? Discovery!


P-value

A modest reality:The p-value is simply the probability that the data occurred by chance. Big leap:A significant p-value is a license to make up a story.


Results

The Prevalence of bisphosphonate use within those who have ONJ is different than the prevalence of bisphosphonate use within those who do not have ONJ (p-value < .0001).

Discussion “In conclusion, this case-control study

supports a causal link between bisphosphonates and ONJ”*

*See page 443, the last Discussion paragraph.


Review To assess validity

– In the study, What’s the question?– Where did the data come from?

Sampling and measurement.– What would you expect if “nothing is

going on”?– Is the observed data different than that?

But other factors could account for the observed difference– Bias, confounding, multiplicity


Next Time

Be able to answer four questions: Based on the study design, what is the

level of evidence? How were threats to validity addressed? Based on the goals of the study, How do

you describe the results? To justify the conclusions, were

comparisons done appropriately?

v i r g i n i a c o m m o n w e a l t h u n i v e r s i t y the scientific method and statistics...

Documents

o n w e

v e r s i t y example

v e r s i t y stats

v e r s i t y solution

t h u n

study valid

study design

scientific method