Evaluating Evidence and Wrap-up

Political Analysis, Week 8 (MT 2015)Oxford Q-Step Centre

Prof. David Kirk

Outline

Tradeoffs with experiments

Steps in Evaluating Research

Tradeoffs of Experiments

Generalizability/External Validity (see Week 7 lecture)

Ethical dilemmas with randomly allocating treatmente.g., withholding a potentially beneficial drug treatment from a sick patient

e.g., skewing an election through political advertisement

Cost

Causal Description versus Causal Explanation

Causal Description vs. Causal Explanation

Causal description: randomized experiments are generally regarded as the gold standard research design for assessing the effect of a deliberately introduced treatment

Causal explanation: experiments do less well at explaining why the cause and effect are related (i.e., the mechanism)

Understanding the mechanism is important not only for determining why an intervention might be effective, but also for assessing the validity of a treatment (example to follow)

Recall, Bond et al. (2012): A 61-million-person experiment in social influence and political mobilization

RM Bond et al. Nature 489, 295-298 (2012) doi:10.1038/nature11421

But what is the mechanism?

Recall: Miguel, Satyanath, and Sergenti (2004)

Miguel, Edward, Shanker Satyanath, and Ernest Sergenti. 2004. “Economic shocks and civil conflict: An instrumental variables approach.” Journal of Political Economy 122:725-53.

Examined the impact of economic conditions on the likelihood of civil conflict

Hypothesis: economic growth is negatively related to civil conflict

But what is the mechanism? The authors find that economic decline increases the likelihood of civil conflict, but why is that?

Deworming Treatment and Educational Attainment

• Hugely influential study that prompted a push for more randomized experiments in developmental economics as well as substantial investment in deworming interventions worldwide

On debates about the quality of deworming research, see (7:45 to 16:00): http://www.bbc.co.uk/programmes/b0659q1f#play

Mechanism underlying deworming

Miguel and Kremer argue that deworming treatment reduces school absenteeism (the outcome) by reducing illness (the mechanism)

Some of the findings of Miguel and Kremer have recently been called into question (see replication by Calum Davey et al. 2015)

• One claim is that deworming treatment does not improve specific health outcomes as Miguel and Kremer proposed (this claim is supported by the Davey et al replication as well as a systematic Cochrane review of other randomized experiments of deworming interventions)

• So if deworming programs do not improve health, why do they appear to increase school attendance?

It could be a Hawthorne Effect

The Hawthorne Effect refers to systematic error from respondents reacting to the fact they are participants in research

This term stems from a study done by Roethlisberger and Dickson (1939) examining the effect of improved lighting on worker productivity at Western Electric’s Hawthorne site.

The authors conclude that productivity increased not because of better lighting, but because workers felt like they should be extra productive because they were being observed by researchers

Maybe school attendance increased in the Miguel and Kremer study because the pupils knew that the researchers were watching to see if they attended school (and not because of worm reduction)

Steps in Evaluating Research

Steps in Evaluating Research

1) Identify and assess the research question and hypothesis• Relation to prior research; fit between question and hypothesis

2) Identify research design

3) Assess measurement of concepts

4) Assess internal validity (discussed in Weeks 4 – 7)• Refers to the validity with which one can conclude that the observed

relationship (covariation) between an independent and dependent variable reflects a causal relationship (as opposed to spurious)

5) Assess external validity (discussed in Week 7)

From question to theory to hypotheses

Independent Variable X

(e.g. wealth)

Dependent Variable Y

(e.g. democracy)

X= GDP per capita Y= Freedom House Index

“the higher the level of GDP per capita, the higher FH Index”

+

Explanation (How/Why)“no democracy without

bourgeoisie”

Observable ComponentHypothesis

ResearchQuestion

?

Theory

Step 1) Identify Research Question and Hypotheses

It may surprise you that some research articles do not clearly state a specific research question; rather, the author may state a general research aim without offering a specific question to answer

Moreover, when a specific question is stated, it may not be completely congruent with the broader research aim

Similarly, sometimes hypotheses are offered which do not actually address the stated research question

Hence, it is vital to identify the question and hypothesis in a study, and assess whether the latter actually provides a test of the former

Example Study (PS reading list)

Campbell, Rosie, Sarah Childs, and Joni Lovenduski. 2010. “Do Women Need Women Representatives?” British Journal of Political Science 40(1): 171-194.

The title is important: per the title, the puzzle here is to determine if female MPs do a better job than male MPs of representing female political interests.

Argument: “In so far as women and men have different preferences, increasing the number of women in parliament should increase the average presence of attitudes favourable to women’s preferences.” p.175

Research Aim and Question

• “Our research contributes to the elite/mass debate by considering whether there is a congruence in the political attitudes of women politicians and women citizens.” P.175

• “The guiding research question of this study is whether women in the political elite share political attitudes with women in the mass public.” P.178

Are these research aims consistent with the title of the article: Do Women Need Women Representatives?

Put differently, even if female politicians and women in the mass public share political attitudes, does this necessarily mean that women need women representatives? What if men in the political elite have the same attitudes toward gender equality as women in the political elite?

Hypotheses

• “We hypothesize that evidence of a commonality between the political attitudes of women at the mass and elite level is suggestive of uncrystallized interests” P.174

• “We hypothesize that more women than men (across a variety of social groups) favour greater equality for women” P.175

• “We hypothesize that women will be more likely than men to think that such attempts to achieve equality have not gone far enough and more likely to support the strongest measures to achieve equality.” P.178

Research Question

“The guiding research question of this study is whether women in the political elite share political attitudes with women in the mass public.”

Hypothesis

“We hypothesize that more women than men (across a variety of social groups [including political elites]) favour greater equality for women”

Will testing the hypothesis provide a direct answer to the research question?

Step 2) Identify Research Design

• Randomized experiment

• Natural and Quasi-Experiments

• Observational study: may be interested in identifying a causal effect, but does not include random assignment or the direct manipulation of the independent/treatment variable

• Survey data• Government data• Also: ethnographies, archival research, case studies

3) Variables and Measurement

Measurement -> the process of assigning numbers/labels to units of analysis in order to represent a concept.

This entails linking concepts to observable data.

In other words, the goal of measurement is to move the research process from abstract concepts to concrete measures.

Conceptualization

A concept is an abstract element or characterization that represents a phenomenon (e.g., democracy)

Conceptualization is the process of formulating and clarifying concepts, and the end product is a conceptual definition

Conceptual Definition: a working definition of a conceptFor example, are political rights and civil liberties part of the conceptual

definition of democracy?

What about the presence of elections and a constitution?

Operationalization

Involves identifying the observable manifestations of abstract conceptsOperations: the procedure which spells out exactly how a concept

will be measured. A key step is to select the source of information on a concept (i.e., surveys, archival records).Note that many concepts are operationalized different ways across

studies (e.g., democracy), but one common strategy is to use operational definitions used in previous studies (e.g., the Freedom House Index or the Democracy Index)

Reliability and Validity of Measurement

Reliability refers to the consistency (precision) with which we can measure a variable

Validity refers to the fit between the concept and the operational definition (i.e. did we actually measure the concept we intended to measure?) Just because someone says they are measuring democracy does not mean they

actually did. Therefore, if s/he finds a correlation between GDP and an invalid measurement of democracy, we actually cannot conclude anything about the relationship between GDP and democracy

The reliability of a measure is essential in order for that measure to be valid; however reliability is necessary but not sufficient for validity

Systematic Review of Research Literature

On any given research topic, there is often an enormous number of studies of varying quality with varying conclusions. While it is imperative for you to develop the skills to draw your own judgments about research, there are also research groups that systematically review the quality of evidence on given topics:

Cochrane Colloboration (http://www.cochrane.org/): network of researchers that gathers and assesses the quality of evidence about medical research and health treatments

Campbell Collaboration (http://www.campbellcollaboration.org/): a social science version of Cochrane; this group systematically reviews and assesses research evidence on social science questions (e.g., education, crime, international development, welfare)

What Works Clearinghouse (Education): http://ies.ed.gov/ncee/wwc/

Reminder: Assessment

Essay, no more than 2000 words; due by noon on Friday of week 2 of Hilary Term

Choose an empirical finding that appears in an article from the reading list of your core paper (i.e., CG, IR, PS). (You may choose one of the articles replicated in the labs.)

• If you are taking more than one core paper, you must do an essay for each

See Weblearn for the full assignment

If we have time…

http://www.ted.com/talks/ben_goldacre_battling_bad_science.html