ECON 4360-01ECON 4360-01 Health Economic Policy Lab

Kem P. Krueger, Pharm.D., Ph.D.Kem P. Krueger, Pharm.D., Ph.D.

Anne Alexander, M.S., Ph.D.Anne Alexander, M.S., Ph.D.

University of WyomingUniversity of Wyoming

Overview of Study DesignsOverview of Study Designs& Threats to & Threats to

Internal and External ValidityInternal and External Validity

ObjectivesObjectives

Define internal and external validity Define internal and external validity and explain the relationship and explain the relationship between the twobetween the two

Describe each type of extraneous Describe each type of extraneous variables that threaten the internal variables that threaten the internal validity of a studyvalidity of a study

ObjectivesObjectives

Describe each of the following study Describe each of the following study designs and explain the strengths and designs and explain the strengths and weaknesses of each one:weaknesses of each one: an experimental studyan experimental study a quasi-experimental studya quasi-experimental study a case-control epidemiology studya case-control epidemiology study a cohort epidemiology studya cohort epidemiology study a correlational studya correlational study a descriptive studya descriptive study

Internal ValidityInternal Validity

The minimum requirements that are The minimum requirements that are necessary for a study to produce necessary for a study to produce

results that are believable.results that are believable.

Threats to Internal ValidityThreats to Internal Validity

HistoryHistory

MaturationMaturation

TestingTesting

InstrumentationInstrumentation

Statistical regressionStatistical regression

Selection biasSelection bias

Experimental mortalityExperimental mortality

HistoryHistory

A Specific event or events that A Specific event or events that occur between the first and last occur between the first and last observation in an experiment that observation in an experiment that could produce a change in the could produce a change in the dependent variable.dependent variable.

This change could be erroneously This change could be erroneously attributed to the intervention (the attributed to the intervention (the independent variable).independent variable).

MaturationMaturation

A change occurring in the study A change occurring in the study subjects as a function of time rather subjects as a function of time rather than a function of a specific event.than a function of a specific event.

If this change impacts the If this change impacts the dependent variable, it may be dependent variable, it may be erroneously attributed to the erroneously attributed to the intervention (the independent intervention (the independent variable).variable).

TestingTesting

When subjects are tested or measured When subjects are tested or measured multiple times during an experiment, multiple times during an experiment, their scores on subsequent tests or their scores on subsequent tests or measures may improve because they measures may improve because they remember previous answers or they remember previous answers or they are more skilled at the task they are are more skilled at the task they are performing because due to practice.performing because due to practice.

This improvement may be erroneously This improvement may be erroneously attributed to the intervention (ind var).attributed to the intervention (ind var).

InstrumentationInstrumentation

Changes in the calibration of a Changes in the calibration of a measurement instrument or measurement instrument or changes in observers during a study changes in observers during a study may produce changes in the may produce changes in the independent variable that are independent variable that are erroneously attributed to the erroneously attributed to the intervention.intervention.

Statistical RegressionStatistical Regression

If people in a group have extremely If people in a group have extremely high scores they will most likely high scores they will most likely “regress to the mean” on their own. “regress to the mean” on their own.

The change in the score may be The change in the score may be erroneously attributed to the erroneously attributed to the intervention.intervention.

Selection BiasSelection Bias

Differential selection of subjects Differential selection of subjects If severely ill subjects are If severely ill subjects are

differentially selected for the differentially selected for the placebo group and mildly ill subjects placebo group and mildly ill subjects are selected for the intervention are selected for the intervention group, the difference between the group, the difference between the groups could be erroneously groups could be erroneously attributed to the intervention.attributed to the intervention.

Experimental MortalityExperimental Mortality

The differential loss of subjects from The differential loss of subjects from one group may affect the independent one group may affect the independent variable.variable.

If severely ill patients drop out of the If severely ill patients drop out of the intervention group because they cannot intervention group because they cannot tolerate the side effects of the study tolerate the side effects of the study drug, the average health status of that drug, the average health status of that group will improve because the sick group will improve because the sick people are gone. This improvement people are gone. This improvement may be erroneously attributed to the may be erroneously attributed to the drug (independent variable).drug (independent variable).

External ValidityExternal Validity

External validity deals with the External validity deals with the generalizability of the results of the generalizability of the results of the study. To what population and study. To what population and setting can the results of this study setting can the results of this study be applied?be applied?

Study DesignsStudy Designs

Levels of EvidenceLevels of Evidence

Experimental DesignsExperimental Designs Randomized Controlled StudiesRandomized Controlled Studies Crossover StudiesCrossover Studies

Quasi-Experimental DesignsQuasi-Experimental Designs Case ControlCase Control Cohort StudiesCohort Studies

Correlational StudiesCorrelational Studies Desciptive StudiesDesciptive Studies

Case ReportsCase Reports Cross Sectional SurveysCross Sectional Surveys

Experimental DesignsExperimental Designs

Randomized controlled trialRandomized controlled trial Pretest-posttest control group Pretest-posttest control group

designdesign

RR OO XX OO

RR OO OO Double or single blindDouble or single blind

Experimental DesignsExperimental Designs

AdvantagesAdvantages Control for all threats to internal validityControl for all threats to internal validity Random selection or assignment is used so Random selection or assignment is used so

there is an unbiased distribution of there is an unbiased distribution of confoundersconfounders

Blinding usually occurs in RCTs (double or Blinding usually occurs in RCTs (double or single)single)

Prospective data collectionProspective data collection Can be used to determine cause and effectCan be used to determine cause and effect

Experimental DesignsExperimental Designs

DisadvantagesDisadvantages Expensive (time and money)Expensive (time and money) If volunteers are recruited they may If volunteers are recruited they may

not represent the population.not represent the population. Placebo controlled studies may be Placebo controlled studies may be

unethicalunethical

Crossover StudiesCrossover Studies (An Experimental Design) (An Experimental Design)

Randomized trial, each subject serves as Randomized trial, each subject serves as his / her own controlhis / her own control

RR OO XX11 OO X X22 OO

RR OO XX22 OO X X11 OO Double or single blind with a washout Double or single blind with a washout

period after first treatmentperiod after first treatment Subject’s response XSubject’s response X11 is compared to his / is compared to his /

her response to Xher response to X22

Crossover StudiesCrossover Studies (An Experimental Design) (An Experimental Design)

AdvantagesAdvantages Since the subjects serve as their own Since the subjects serve as their own

control, intra-group variation is lower and control, intra-group variation is lower and fewer subjects are neededfewer subjects are needed

Random selection or assignment is used so Random selection or assignment is used so there is an unbiased distribution of there is an unbiased distribution of confoundersconfounders

Blinding usually occurs (double or single)Blinding usually occurs (double or single) Threats to internal validity are controlledThreats to internal validity are controlled Can determine cause and effectCan determine cause and effect

Crossover StudiesCrossover Studies (An Experimental Design) (An Experimental Design)

DisadvantagesDisadvantages Carry over / order effectsCarry over / order effects Order of treatment may matterOrder of treatment may matter Cross over timingCross over timing Subjects have more opportunities to Subjects have more opportunities to

drop outdrop out If volunteers are recruited they may If volunteers are recruited they may

not represent the population.not represent the population.

Quasi-Experimental DesignsQuasi-Experimental Designs

AdvantagesAdvantages Randomization may not always be Randomization may not always be

possible, so this is the next strongest possible, so this is the next strongest study designstudy design

Controls for most threats to internal Controls for most threats to internal validityvalidity

Confounding variables can be Confounding variables can be controlled by matching the groups on controlled by matching the groups on key variableskey variables

Quasi-Experimental DesignsQuasi-Experimental Designs

DisadvantagesDisadvantages Cause and effect cannot be establishedCause and effect cannot be established Selection bias may occur if the Selection bias may occur if the

researcher is not carefulresearcher is not careful

Cohort StudiesCohort StudiesExposureExposure DiseaseDisease

??

ExposureExposure DiseaseDisease??

Prospective Cohort StudyProspective Cohort Study

Retrospective Cohort StudyRetrospective Cohort Study

Measures of Disease AssociationMeasures of Disease Association

Relative Risk (RR): estimates the Relative Risk (RR): estimates the magnitude of an association magnitude of an association between exposure and disease and between exposure and disease and indicates the likelihood of indicates the likelihood of developing the disease in the developing the disease in the exposed group relative to those who exposed group relative to those who are not exposedare not exposed

Cohort StudiesCohort Studies

DisadvantagesDisadvantages Identifying controls is difficultIdentifying controls is difficult Exposure may be linked to a hidden Exposure may be linked to a hidden

confounderconfounder Blinding is difficultBlinding is difficult No randomizationNo randomization Large sample size is needed for rare Large sample size is needed for rare

diseasesdiseases

Case-Control StudiesCase-Control Studies

ExposureExposure DiseaseDisease??

Measures of Disease AssociationMeasures of Disease Association

Odds Ratio (OR): in a case-control study it is Odds Ratio (OR): in a case-control study it is not possible to estimate the RR, so it must be not possible to estimate the RR, so it must be estimated using an odds ratioestimated using an odds ratio

Measures of Disease AssociationMeasures of Disease Association

Odds Ratio (OR): in a case-control study it is not possible to estimate the RR, so it must be estimated using an odds ratio

Case-Controlled StudiesCase-Controlled Studies

AdvantagesAdvantages Relatively easy to conduct (save time & $)Relatively easy to conduct (save time & $) Few ethical problemsFew ethical problems

DisadvantagesDisadvantages Cannot prove causalityCannot prove causality Study is susceptible to recall bias and Study is susceptible to recall bias and

selection biasselection bias Confounders may not be controlledConfounders may not be controlled

Correlational Studies

Look for associations or relationships between variables

Advantages Often use secondary data Large samples

Disadvantages Limited by available data Cannot determine cause and effect

relationship

Descriptive StudiesDescriptive Studies

Describe the situation at a given point in time and may identify things for further study

Advantages May be easier to conduct Sample may be easy to identify

Disadvantages Data collection instrument must be valid and

reliable Cannot determine cause and effect relationship

Questions?Questions?