Experimental Control& Design

Psych 231: Research Methods in Psychology

Controlling Variability

Methods of Experimental Control Constancy/Randomization Comparison Production

Methods of Controlling Variability

Constancy/Randomization If there is a variable that may be related to the DV that you can’t (or don’t want to) manipulate• Control variable: hold it constant• Random variable: let it vary randomly across all of the experimental conditions

But beware of potential confounds, variables that co-vary with both the IV and DV but aren’t controlled

Methods of Controlling Variability

Comparison An experiment always makes a comparison, so it must have at least two groups• Sometimes there are control groups

• This is typically the absence of the treatment

Traininggroup

No training (Control) group

• Without control groups if is harder to see what is really happening in the experiment

• It is easier to be swayed by plausibility or inappropriate comparisons

Methods of Controlling Variability

Comparison An experiment always makes a comparison, so it must have at least two groups• Sometimes there are control groups

• This is typically the absence of the treatment

1 week of Training group

2 weeks of Training group

• Sometimes there are a range of values of the IV

3 weeks of Training group

Methods of Controlling Variability

Production The experimenter selects the specific values of the Independent Variables

1 week of Training group

2 weeks of Training group

3 weeks of Training group

• Need to do this carefully• Suppose that you don’t find a difference in the DV across your different groups

• Is this because the IV and DV aren’t related?• Or is it because your levels of IV weren’t different enough

Experimental designs

So far we’ve covered a lot of the about details experiments generally

Now let’s consider some specific experimental designs. Some bad designs Some good designs

• 1 Factor, two levels• 1 Factor, multi-levels• Factorial (more than 1 factor)• Between & within factors

Poorly designed experiments

Bad design example 1: Does standing close to somebody cause them to move? “hmm… that’s an empirical question. Let’s see what happens if …”

So you stand closely to people and see how long before they move

Problem: no control group to establish the comparison group (this design is sometimes called “one-shot case study design”)

Poorly designed experiments

Bad design example 2: Testing the effectiveness of a stop smoking relaxation program

The participants choose which group (relaxation or no program) to be in

Poorly designed experiments

Non-equivalent control groups

participants

Traininggroup

No training (Control) group

Measure

Measure

Self Assignment

Independent Variable

Dependent Variable

RandomAssignment

Problem: selection bias for the two groups, need to do random assignment to groups

Problem: selection bias for the two groups, need to do random assignment to groups

Bad design example 2:

Poorly designed experiments

Bad design example 3: Does a relaxation program decrease the urge to smoke? Pretest desire level – give relaxation program – posttest desire to smoke

Poorly designed experiments

One group pretest-posttest design

participantsPre-test Training group

Post-testMeasure

Independent Variable

Dependent Variable

Dependent Variable

Problems include: history, maturation, testing, and more

Pre-test No Training group

Post-testMeasure

Add another factor

Bad design example 3:

1 factor - 2 levels

Good design example How does anxiety level affect test performance?• Two groups take the same test

• Grp1 (moderate anxiety group): 5 min lecture on the importance of good grades for success

• Grp2 (low anxiety group): 5 min lecture on how good grades don’t matter, just trying is good enough 1 Factor (Independent variable), two levels

• Basically you want to compare two treatments (conditions)• The statistics are pretty easy, a t-test

1 factor - 2 levels

participants

Low

Moderate Test

Test

Random Assignment

Anxiety

Dependent Variable

Good design example How does anxiety level affect test performance?

anxiety

low moderate

8060

low moderatetest performance

anxiety

One factor

Two levels

Use a t-test to see if these points are statistically different

T-test = Observed difference between conditions

Difference expected by chance

Good design example How does anxiety level affect test performance?

1 factor - 2 levels

Advantages: Simple, relatively easy to interpret the results

Is the independent variable worth studying?• If no effect, then usually don’t bother with a more complex design

Sometimes two levels is all you need• One theory predicts one pattern and another predicts a different pattern

1 factor - 2 levels

low moderatetest performance

anxiety

What happens within of the ranges that you test?Interpolation

Disadvantages: “True” shape of the function is hard to see

• Interpolation and Extrapolation are not a good idea

1 factor - 2 levels

Extrapolation

low moderate

test performance

anxiety

What happens outside of the ranges that you test?

Disadvantages: “True” shape of the function is hard to see

• Interpolation and Extrapolation are not a good idea

1 factor - 2 levels

high

1 Factor - multilevel experiments

For more complex theories you will typically need more complex designs (more than two levels of one IV)

1 factor - more than two levels Basically you want to compare more than two conditions

The statistics are a little more difficult, an ANOVA (Analysis of Variance)

Good design example (similar to earlier ex.) How does anxiety level affect test performance?• Two groups take the same test

• Grp1 (moderate anxiety group): 5 min lecture on the importance of good grades for success

• Grp2 (low anxiety group): 5 min lecture on how good grades don’t matter, just trying is good enough

1 Factor - multilevel experiments

• Grp3 (high anxiety group): 5 min lecture on how the students must pass this test to pass the course

1 factor - 3 levels

participants

Low

Moderate Test

Test

Random Assignment

Anxiety

Dependent Variable

High Test

1 Factor - multilevel experiments

anxiety

low mod high

8060 60

low mod

test performance

anxiety

high

1 Factor - multilevel experiments

Advantages Gives a better picture of the relationship (function)

Generally, the more levels you have, the less you have to worry about your range of the independent variable

Relationship between Anxiety and Performance

low moderate

test performance

anxiety

2 levels

highlow mod

test performance

anxiety

3 levels

1 Factor - multilevel experiments

Disadvantages Needs more resources (participants and/or stimuli)

Requires more complex statistical analysis (analysis of variance and pair-wise comparisons)

Pair-wise comparisons

The ANOVA just tells you that not all of the groups are equal. If this is your conclusion (you get a “significant ANOVA”) then you should do further tests to see where the differences are• High vs. Low• High vs. Moderate• Low vs. Moderate