Experimental design

Experimental design

ITS classDecember 2, 2004

ITS classDecember 2, 2004

Quantitative researchQuantitative research

Quantify relationships between variables

Measured on a sample of subjects

Relationships expressed through effect statistics Correlations, relative

frequencies, diff. between means

Quantify relationships between variables

Measured on a sample of subjects

Relationships expressed through effect statistics Correlations, relative

frequencies, diff. between means

Types of StudyTypes of Study

Descriptive Measure things as they are

Experimental Take measurements Try an intervention

(treatment) Take measurements

Descriptive Measure things as they are

Experimental Take measurements Try an intervention

(treatment) Take measurements

Descriptive studiesDescriptive studies

Case Case series Cross-sectional Cohort or prospective or

longitudinal Case-control or

retrospective

Case Case series Cross-sectional Cohort or prospective or

longitudinal Case-control or

retrospective

Experimental studiesExperimental studies

AKA longitudinal, repeated-measures, interventions

Without a control group Time series Crossover

With a control group

AKA longitudinal, repeated-measures, interventions

Without a control group Time series Crossover

With a control group

Control group studiesControl group studies

Necessary if treatment effect cannot be “washed out”

Random assignment: min. chance that either group is not typical of population

Single-blind: subjects Double-blind: experimenter

too Best studies: data analyzed

blind

Necessary if treatment effect cannot be “washed out”

Random assignment: min. chance that either group is not typical of population

Single-blind: subjects Double-blind: experimenter

too Best studies: data analyzed

blind

Ethical issuesEthical issues

Randomized controlled study may not be ethical

E.g.: Heart disease study over 10 years

Access to treatment Inform subjects of

randomization

Randomized controlled study may not be ethical

E.g.: Heart disease study over 10 years

Access to treatment Inform subjects of

randomization

Quality of designsQuality of designs

Quality of evidence for causal relationships

Least for case, case-series Cross-sectional, case-

control Prospective Experimental best

evidence

Quality of evidence for causal relationships

Least for case, case-series Cross-sectional, case-

control Prospective Experimental best

evidence

ConfoundingConfounding

Two variables may be related for other reasons

Control for potential confounding factors

Two variables may be related for other reasons

Control for potential confounding factors

SamplesSamples

Generalizability Representative Safest: Random sample

Balance: Stratified random (%s) Balance on pretest!

Selection bias: Not representative Sources: age, socioeconomic

status Self-selection into groups High compliance

Generalizability Representative Safest: Random sample

Balance: Stratified random (%s) Balance on pretest!

Selection bias: Not representative Sources: age, socioeconomic

status Self-selection into groups High compliance

Sample sizeSample size

Statistical significance Big enough to be sure you will

detect the smallest worthwhile effect To be sure =

detect 80% of time Detect =

95% of time, expect a smaller value if there is no effect, p<0.05

Smallest worthwhile effect = Smallest to make a diff. to subjects’

lives

Statistical significance Big enough to be sure you will

detect the smallest worthwhile effect To be sure =

detect 80% of time Detect =

95% of time, expect a smaller value if there is no effect, p<0.05

Smallest worthwhile effect = Smallest to make a diff. to subjects’

lives

Statistical significanceStatistical significance

P-value: the probability of getting

something more extreme than your result, when there is no effect in the population

P-value: the probability of getting

something more extreme than your result, when there is no effect in the population

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

SourcesSources

http://www.sportsci.org/jour/0001/wghdesign.html: Good resource for basic stats

http://www.softwareevaluation.de/ : Website for EASy-D: a database of evaluation of adaptive systems, also links to research-based usability guidelines

http://www.sportsci.org/jour/0001/wghdesign.html: Good resource for basic stats

http://www.softwareevaluation.de/ : Website for EASy-D: a database of evaluation of adaptive systems, also links to research-based usability guidelines