survey experiments: illuminating theoretical and methodological controversies harold clarke epps,...
TRANSCRIPT
Survey Experiments: Illuminating Theoretical and
Methodological Controversies
Harold Clarke
EPPS, University of Texas at Dallas
email: [email protected]
Utility of Survey Experiments
• Survey experiments – increasingly popular over past decade
• Sold as way to study causal processes that cannot be investigated with traditional cross-sectional & panel surveys
• Randomization as cure for omitted variable biases & endogeneity problems that bedevil causal inference in observational studies
• Survey experiments – external validity - sophomores in labs are not representative!
• Other problems – truncated variance, small N’s, functional form forgotten by ANOVA, model specification
Survey Experiment Vehicles• In-person probability surveys – ANES, BES, GSS• Mail-back surveys• RDD telephone surveys – impact of TESS – NSF
funds• Internet surveys – general populations - BES
Continuous Monitoring Survey (CMS), BES Rolling Campaign Panel Survey (RCPS), now ANES
• Internet surveys – specific populations – Survey Monkey, Qualtrics -> e.g., students, employees, patients, party members
• Amazon Mechanical Turk too!
Internet Survey Experiments
• Internet methodology – exciting possibilities
• Huge N’s – massive statistical power
• Inexpensive – vastly expand research opportunities – e.g., CMS
• Technical tools – video, audio, feedback to respondents, intricate randomizations
Theory & Method• Often think of experiments as theory or, at least,
hypothesis testing• Hypotheses of interest may concern measurement of
key concepts• Survey measures may be affected by context in which
questions are asked• Internal – question ordering effects• External – political/economic conditions• Gaines, Kuklinski & Quirk (2007) discuss the latter in
terms of survey experiments, but it is of general interest• Use survey experiments to investigate how external
context affects survey responses or relationships among variables – cross-level interactions
External Context Survey Experiment: Inglehart and Post-Materialism
• Testing the impact of macro-economic context on responses to Inglehart’s Materialism-Postmaterialism question battery in the Euro-Barometer surveys
• Does Inglehart’s battery really measure what he says it does?
• Or, is it contaminated by changing economic context
• Why does this matter?
Inglehart’s Value Shift Thesis & His Values Battery
Inglehart famously argued there has been a “value shift” from materialist to postmaterialism driven by rising affluence in post-war Western democracies. Everybody jumped on this bandwagon!
Four-Item Values Battery administered in bi-annual Euro-Barometer surveys
Items:• Fight Rising Prices• Maintain Order in the Nation• Give People More Say in Government• Protect Freedom of Speech
Respondents asked to select the most important, then select the next most important of the 3 remaining items.
Figure 1. Why The Battery Might Be Affected by Macro-Economic Context
0
2
4
6
8
10
12
14
0
4
8
12
16
20
24
28
74 76 78 80 82 84 86 88 90 92 94 96
Pe
r ce
nt P
er ce
nt
Unemployment
Inflation
Year
Figure 1. Inflation and Unemployment Rates,Britain January 1974 - June 1997
r = -.70
<-Thatcher Elected
Figure 2. Germany – Canonical Case
-1
0
1
2
3
4
5
6
7
8
-40
-30
-20
-10
0
10
76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92
Infla
tion
& U
ne
mp
loym
en
t - P
erc
en
tag
eB
ala
nce
Po
st-Ma
teria
lists - Ma
teria
lists - Pe
rcen
tag
e
Unemployment
Inflation
Balance Post-MaterialistsMinus Materialists
Correlation: Balance & Unemployment = +76 Balance & Inflation = -.79
Figure 2. Inflation, Unemployment & Balance of Inglehart Post-Materialistsand Materialists, Germany 1976 -1992
Values Battery Survey Experiment
• Clarke’s Chance Encounter at Zuma• Dinner With Max Kaase• What are you working on Harold?• Studying how Macro-Economic Conditions Affect
Responses to Inglehart’s Values Battery• Nice aggregate results, but need to nail it at the
individual level• Survey experiment!• Parallel work being done by Kaase in Germany• Let’s work together!
The Experiment
• National Samples of Canadian & German Electorates
• Split-Sample Design• Half-One gets traditional Values Battery; Half-
Two gets Values Battery with Unemployment item substituted for Inflation Item
• At end of Canadian survey, ask other battery – Do people change their value classification in a 15-minute survey?!
• Results?
Figure 3. Alternative Values Batteries, Responses - Canada
Figure 3. Responses to Alternative Versions of Inglehart Values Battery
18
9
52
3
17
35
25
15
7
18
0
10
20
30
40
50
60
Maintain Order More Say Jobs/Prices Free Speech MultipleMentions
Per
cen
t
Unemployment Version Inflation Version
Figure 4. Materialists & Post-MaterialistsRival Values Batteries - Canada
FIgure 4. Materialists and Post-Materiailists - Alternative Versions of Inglehart Values Battery
38
52
11
-27
25
58
17
-8
-40
-30
-20
-10
0
10
20
30
40
50
60
70
Materialist Mixed Post-Materialist Balance
Per
cen
t
Unemployment Version Inflation Version
Results & Implications
• Same results in Germany!
• Significant portion of the hypothesized value shift an artifact of interaction between his measure and changing macroeconomic context
• Example of Gaines’ et al. caveat in their Political Analysis article
• Simple experiments can have big payoffs
Three Other Examples of Survey Experiments
• The Dynamics of Partisanship – Does party identification have an individual dynamic? One aspect of controversy concerns question wording – A task for survey experiments!
• Does Mode Matter? - Do Traditional In-Person and Internet Election Studies Tell the Same Story?
• The Endogeneity of Ideal Points? – Leveraging the Power of the Internet With a Feedback-to-Respondent Experiment to determine if “De Gustibus” is a viable psephological anthropology.
Dynamics of Partisanship
• Longstanding Controversy!• Campbell et al. v. Fiorina et al.• Green et al.• In Canada, Richard Johnston & Andre Blais said
if only we changed the “or what?” in first ANES party id question to “or none of these” we would find:
• 1. fewer identifiers• 2. more stable identifiers
Survey Experiment on Stability of Party Identification in Canada
• Split Half Samples in 2000-2002 Political Support in Canada Panel Surveys
• One Half Gets Traditional Wording; Other Half Gets Revised Wording
• Results: • Slightly fewer identifiers with revised wording• Rates of partisan stability almost exactly the same • Lots of partisan instability in both versions• Survey experiment results reinforce MMLC analyses of
US, UK, Canadian and German multi-wave panel data Clarke & McCutcheon POQ 2009 and Stegmueller, Neundorf and Scotto, POQ 2011.
Figure 8. Dynamics of Party Identification, 2000-2002 PSC Panel: Stable Identifiers
Traditional and Revised Question Wordings
59.6 60.259.4 58.6
0
10
20
30
40
50
60
70
80
90
100
1st Question Both Questions
Perc
ent
Traditional Revised
Does Mode Matter? Are Ideal Points Endogenous?
• Mode – see modecomp1.ppt
• Ideal Points – see endogjepoprev.pdf
Conclusions I
• Survey experiments are useful tools!
• Survey experiments can address big theoretical & methodological controversies
• Internet surveys put survey experiments “in reach” for many researchers
• CCAP, CCES, CMS – example vehicles
• Qualtrics – sophisticated platform – many universities subscribe – no cost
Conclusions II• Survey experiments not a panacea• Example of ongoing controversies over
implications of endogenity of economic evaluations in party support and voting models
• Never resolve controversy with traditional cross-sectional or panel survey designs!
• Survey experiments very unlikely to simulate real economic conditions
• Way forward - use CMS monthly surveys 1997- 2010 & VEC modeling – see Snark Paper
• Data Driven Solutions • Not Necessarily Expensive