causal theory and research design - lordsutch.comlordsutch.com/pol251/shively-6.pdfcausal theory and...

Causal Theory and ResearchDesign

Chapter 6 of The Craft of Political Research

Chris Lawrence

[email protected]

Causal Theory and Research Design – p.1/28

Introduction

In this chapter, we will discuss:

The nature of causality

Research design: how to study socialphenomena

Introduction

In this chapter, we will discuss:

The nature of causality

Research design: how to study socialphenomena

The Nature of Causality

We can say two variables are related ifcertain values of one variable coincide withcertain variables of another variable.

To put it another way, two variables arerelated if they vary together in a pattern.

However, it is quite a leap to say that thechanges in one variable occur as a directresult of the influence of another variable.

The Nature of Causality

We can say two variables are related ifcertain values of one variable coincide withcertain variables of another variable.

To put it another way, two variables arerelated if they vary together in a pattern.

However, it is quite a leap to say that thechanges in one variable occur as a directresult of the influence of another variable.

Causal Relationships

When the values of one variable produce thevalues of the other variable, the relationship isa causal relationship.

In theory-driven research, we are almostexclusively concerned with causalrelationships.

Examples: changes in party identificationcause changes in vote choice; changes inmilitary spending cause changes in the levelof tension between states.

Causal vs. Non-Causal Relationships

Not all relationships are necessarily causal.

For example, some argue that the MichiganModel is wrong: that something “else” causeschanges in both party identification and votechoice. (This might be “ideology.”)

Causal vs. Non-Causal Relationships

Not all relationships are necessarily causal.

For example, some argue that the MichiganModel is wrong: that something “else” causeschanges in both party identification and votechoice. (This might be “ideology.”)

How Far Can Objectivity Take Us?

The question of whether or not a relationshipexists is inherently objective and can beanswered by empirical observation: eitherthere is a relationship or there isn’t one.

However, determining if a relationship iscausal is more subjective, and must be drivenby the use of theory.

For example: winter does not “cause” spring.

Why Worry About Causality Then?

If we can’t objectively determine causality,why shouldn’t we just concern ourselves withfinding relationships?

As Shively puts it, causality gives us“leverage” on reality: it helps us understandwhy the world works the way it does, andmakes it possible for us to change things.

For Example. . .

For example, just because people whosmoked were more likely to get lung cancerdidn’t mean that smoking caused lungcancer. Proving the causal link throughresearch and eliminating other possibilitiesmade it possible for us to change public policyto reduce the incidence of lung cancer.

Interpreting Relationships

When we see a relationship between twovariables, there are two possibilities:

Causation is not involved. This is exceedinglyunlikely, but it is possible. For example, bysheer coincidence it could occur that twounrelated events would appear to be related:Punxatawney Phil could predict the durationof winter well for a few years.

Causation is involved, somehow . . .

Interpreting Relationships

When we see a relationship between twovariables, there are two possibilities:

Causation is not involved. This is exceedinglyunlikely, but it is possible. For example, bysheer coincidence it could occur that twounrelated events would appear to be related:Punxatawney Phil could predict the durationof winter well for a few years.

Causation is involved, somehow . . .

What if causation is involved?

There are two possibilities, and we need toinvestigate further to decide which is the case:

The covariance we observe is the result ofoutside factors that drive both phenomena:therefore, one variable does not cause theother, but rather a third variable caused both.

One phenomenon could actually cause theother, but we still need to decide whichcauses which: does A cause B, or does Bcause A? (Temporal sequence can helphere!)

What if causation is involved?

There are two possibilities, and we need toinvestigate further to decide which is the case:

The covariance we observe is the result ofoutside factors that drive both phenomena:therefore, one variable does not cause theother, but rather a third variable caused both.

One phenomenon could actually cause theother, but we still need to decide whichcauses which: does A cause B, or does Bcause A? (Temporal sequence can helphere!)

Experimental Designs

The basis for all scientific research comes from

the experiment. There are two major types of

scientific experiment that Shively discusses: a

“natural” experiment and a “pure” experiment.

Natural Experiments

Natural experiments include two groups:

An “experimental” group

A “control” group

In a natural experiment, the experimentalgroup is exposed to a stimulus or treatment(representing our independent variable),while the control group is not.

After the experiment, we take a measurementof our dependent variable in both groups.

Natural Experiments

Natural Experiments (continued)

Any difference between the two groups weassume is the result of the stimulus.

To reduce the possibility of other causalfactors, we randomize assignment to the twogroups, and try to use as many subjects aspossible.

Natural Experiments (continued)

Any difference between the two groups weassume is the result of the stimulus.

To reduce the possibility of other causalfactors, we randomize assignment to the twogroups, and try to use as many subjects aspossible.

Pure Experiments

As in the natural experiment, we have twogroups: the control group and experimentalgroup. We also still have two variables, anindependent variable and a dependent variable.Unlike the natural experiment, we measure thedependent variable twice.

We measure it first before both groups areexposed to the stimulus.

We measure it again after both groups areexposed to the stimulus.

This approach allows us to find any hidden

differences between the two groups before the

stimulus, so we don’t accidentally attribute them

to the stimulus.

Pure Experiments

As in the natural experiment, we have twogroups: the control group and experimentalgroup. We also still have two variables, anindependent variable and a dependent variable.Unlike the natural experiment, we measure thedependent variable twice.

We measure it first before both groups areexposed to the stimulus.

We measure it again after both groups areexposed to the stimulus.

This approach allows us to find any hidden

differences between the two groups before the

stimulus, so we don’t accidentally attribute them

to the stimulus.

Why Not Always Use Pure Experiments?

There are some drawbacks to the pureexperimental approach, however:

We might have some instrument reactivity :the pre-test may affect the respondents insome way. It may “clue” them in to what isgoing on, drawing attention to things we maynot want them to pay close attention to.

It could be more expensive (though if youhave people in the room already, the expensefactor is minimized).

In both types of experiments, we will also have

to concern ourselves with measurement error,

particularly issues of reliability and validity. (This

explains why we already talked about these

issues!)

Why Not Always Use Pure Experiments?

There are some drawbacks to the pureexperimental approach, however:

We might have some instrument reactivity :the pre-test may affect the respondents insome way. It may “clue” them in to what isgoing on, drawing attention to things we maynot want them to pay close attention to.

It could be more expensive (though if youhave people in the room already, the expensefactor is minimized).

In both types of experiments, we will also have

to concern ourselves with measurement error,

particularly issues of reliability and validity. (This

explains why we already talked about these

issues!)

Non-Experimental Designs

Unfortunately, we don’t often conductexperiments in political science, for two majorreasons:

We can’t control peoples’ behavior over along period of time

We can’t control the introduction of stimuli(people already know something aboutpolitics by the time we find them)

So, we have to rely on other designs that sacrifice

the presence of a pure control group. Causal Theory and Research Design – p.16/28

Non-Experimental Designs

Unfortunately, we don’t often conductexperiments in political science, for two majorreasons:

We can’t control peoples’ behavior over along period of time

We can’t control the introduction of stimuli(people already know something aboutpolitics by the time we find them)

So, we have to rely on other designs that sacrifice

the presence of a pure control group. Causal Theory and Research Design – p.16/28

The Cross-Sectional Design

By far the most common design in politicalscience is cross-sectional.

Measurements of the independent anddependent variables are taken at the sametime.

The researcher has no control over whether aparticular respondent has been exposed tothe independent variable.

The Cross-Sectional Design

By far the most common design in politicalscience is cross-sectional.

Measurements of the independent anddependent variables are taken at the sametime.

The researcher has no control over whether aparticular respondent has been exposed tothe independent variable.

Cross-Sectional Designs (continued)

The researcher can’t assign respondents tocontrol or experimental groups.

The researcher can’t affect the conditionsunder which people are exposed to thestimulus.

Instead of experimental controls, theresearcher must use data analysis to drawconclusions about the effects of theindependent variable.

Advantages

While this presentation suggests thatcross-sectional designs are inferior toexperiments, in fact they can help in certainsituations:

We can observe phenomena in a naturalsetting, instead of a lab situation.

We can study larger and more representativepopulations.

We can test hypotheses that can’t be easilytested through experimentation.

Advantages

Experiments Don’t Always Work. . .

The final point above is perhaps the mostimportant. For example, say we hypothesize thatgreater education leads to higher income:

We can randomly ask adults about theireducation and income, divide therespondents into groups based on their levelof education, and compare the averageincomes of the groups.

We have no way to control who gets moreeducation and who gets less.

Experiments Don’t Always Work. . .

The final point above is perhaps the mostimportant. For example, say we hypothesize thatgreater education leads to higher income:

We can randomly ask adults about theireducation and income, divide therespondents into groups based on their levelof education, and compare the averageincomes of the groups.

We have no way to control who gets moreeducation and who gets less.

Experiment Problems (continued)

As a practical matter, it would be grosslyunethical to take preschoolers and onlyeducate some of them!

Often, experimental research is toounrealistic to study political phenomena. Wecan’t have “fake elections” and “fake wars” inthe lab and expect realistic results.

Experiment Problems (continued)

As a practical matter, it would be grosslyunethical to take preschoolers and onlyeducate some of them!

Often, experimental research is toounrealistic to study political phenomena. Wecan’t have “fake elections” and “fake wars” inthe lab and expect realistic results.

Panel Studies

Another non-experimental design is called a“panel study.” Panel studies are likecross-sectional designs, but we introduce a timeelement.

The researcher measures the variables ofinterest with the same units of analysis(people, countries, whatever) at multiplepoints in time.

Uses of Panel Studies

We can use this technique to examinechanges over time and to provide a pre-testbefore the stimulus is naturally introduced intothe population.

For example, we might want to find outwhether exposure to television advertisingincreases voters’ ability to identify relevantissues in a campaign.

Uses of Panel Studies

We can use this technique to examinechanges over time and to provide a pre-testbefore the stimulus is naturally introduced intothe population.

For example, we might want to find outwhether exposure to television advertisingincreases voters’ ability to identify relevantissues in a campaign.

An Example Panel Study

An experiment would seem ideal, since wecould take a pre-test, make the control groupnot watch the ads, and then do a post-test.

However, this wouldn’t work because we haveno way to ensure that the control groupdoesn’t see any of the ads unless we lockthem away for a while or get their parents toground them.

An Example Panel Study

An experiment would seem ideal, since wecould take a pre-test, make the control groupnot watch the ads, and then do a post-test.

However, this wouldn’t work because we haveno way to ensure that the control groupdoesn’t see any of the ads unless we lockthem away for a while or get their parents toground them.

Voilà! A solution!

Solution: a panel design. Test the subjectsbefore the campaign begins, then after thecampaign starts, and measure bothawareness of the issues and (self-reported)exposure to the ads.

Time Series Design

Another approach we can use is the time seriesdesign. Like a panel design, we measure thevariables multiple times.

The more times we measure, the easier it isto rule out other possible causes for changebetween time points.

For example, if we measured Bill Clinton’spopularity at the beginning and end of 1999,we would observe some change, but literallythousands of factors could have caused it.

Time Series Design

Another approach we can use is the time seriesdesign. Like a panel design, we measure thevariables multiple times.

The more times we measure, the easier it isto rule out other possible causes for changebetween time points.

For example, if we measured Bill Clinton’spopularity at the beginning and end of 1999,we would observe some change, but literallythousands of factors could have caused it.

What’s The Frequency, Bill?

To get around this problem, we couldmeasure Clinton’s popularity monthly, weekly,or even daily. Thus if Clinton’s popularityincreased by ten percent in a day, we couldfairly attribute that change to one or twoevents.

Other Uses

A time series can also allow us to see if

changes affect different groups in different ways;

for example, more politically sophisticated voters

may have more stable opinions than less

sophisticated voters.

causal theory and research design - lordsutch.comlordsutch.com/pol251/shively-6.pdfcausal theory and...

Documents