surveyresearch @ surveymonkey surveymonkey @...
TRANSCRIPT
![Page 1: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/1.jpg)
SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPOR
![Page 2: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/2.jpg)
Revisiting Sample Frame and Mode Effects: A Comparison of Point Estimates
Nancy A. Mathiowetz, University of Wisconsin-Milwaukee J. Michael Brick, Westat Sarah Cho and Jon Cohen, SurveyMonkey Ruth Igielnik, Scott Keeter, Andrew Mercer, and Kyley McGeeney, Pew Research Center
![Page 3: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/3.jpg)
Benchmarks • American Community Survey • Current Population Survey
Internet Supplement Supplement on Volunteering
• National Health Interview Survey
![Page 4: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/4.jpg)
Estimates and Benchmarks: Health Items
0
10
20
30
40
50
60
70
80
90
100
Excellent/ Very Good Health
No Dr due to Costs Dr Visits: 2-3 Never Smoked
ABS:Mail
Prob:Web
Non-Prob:Web
Benchmark
![Page 5: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/5.jpg)
Estimates and Benchmarks: Volunteer, Active Duty, Home Tenure, and Food Insecurity
0
10
20
30
40
50
60
70
80
90
100
Volunteer Active Duty Tenure 5+ Years Food Insecure
ABS: Mail
Prob:Web
Non-Prob: Web
Benchmark
![Page 6: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/6.jpg)
Internet Use Every Day: Expected Differences; Differences not eliminated by weighting
0
10
20
30
40
50
60
70
80
90
100
Internet Every Day: Total Population Internet Every Day: Online Population
ABS: Mail
Prob: Web
Non-Prob: Web
CPS Internet Supplement, 2013
![Page 7: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/7.jpg)
Summary: Point Estimates • Median Difference between sample and benchmark
ABS: 5.4 percentage points Probability-based Web: 9.1 percentage points Non-probability based Web: 6.5 percentage points
![Page 8: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/8.jpg)
Do the samples lead to different predicted probabilities?
• Logistic regression models for each of the samples • Independent Variables
Gender, age, Race, Education, Income, Currently Employed Not necessarily great models, but consistent!
![Page 9: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/9.jpg)
Predicted Probabilities: Illustrative case Female, 25-34 years of age, white, some college, $40,000-$99,999, employed
0
10
20
30
40
50
60
70
80
90
100
Volunteer Internet: Every Day Food Own Home
ABS:Mail
Prob:Web
Non-Prob:Web
![Page 10: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/10.jpg)
Predicted Probabilities—health items
0 10 20 30 40 50 60 70 80 90
100
Ex/VG Health No Dr: Costs Never Smoked
ABS:Mail
Prob:Web
Non-Prob: Web
![Page 11: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/11.jpg)
Conclusions • Point estimates:
No one design consistently superior when compared to benchmarks Variability we would expect The findings suggest that estimates from the ABS differs from both the non-
probability and the probability web surveys • Coverage and mode confounded
• Multivariate models (not shown) show surprisingly few differences with respect to direction or significance
Although the size of the coefficients in these models were substantively different—evident in the predicted probabilities
![Page 12: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/12.jpg)
SurveyResearch@SurveyMonkey Elec4on Polling
![Page 13: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/13.jpg)
Everyone is hard to reach
SOURCE: Pew Research Center Assessing the Representa4veness of Telephone Surveys
90%
77% 79%
73% 72%
62%
43% 40% 34%
31%
21%
14%
36%
28% 25%
21%
15%
9% 0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1997 2000 2003 2006 2009 2012
Contact rate Coopera4on rate Response rate
![Page 14: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/14.jpg)
TALKING TURKEY
![Page 15: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/15.jpg)
![Page 16: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/16.jpg)
14%
23%
36% 36%
51%
57% 59%
63% 64% 68%
71% 73% 74%
77% 76% 79%
82% 85%
89%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1995 1996 1997 1998 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014
Percent who use the Internet at least occasionally
Breaking news …
SOURCE: Internet penetra4on data (percent who at least occasionally use the internet, read or check email, or use the internet on their mobile device) from Pew Research Internet Project
![Page 17: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/17.jpg)
SurveyMonkey by the numbers
![Page 18: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/18.jpg)
Not just “surveys”
![Page 19: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/19.jpg)
Geographic diversity
SurveyMonkey Monthly Traffic Density
U.S. Population Density
![Page 20: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/20.jpg)
So what now?
![Page 21: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/21.jpg)
• Conducted Oct. 3 to Nov. 6 • 199,699 total interviews; 171,909 completes
• 34,405 “already voted,” including 13,828 before Elec4on Day
• Click-‐thru rate of 7 to 14 percent across geographies
• 8,296 completes with teens under 18
2014 Elec4on project overview
![Page 22: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/22.jpg)
What was worrisome
![Page 23: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/23.jpg)
UPDATED / Public story of 2014
![Page 24: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/24.jpg)
Unique insight / expansive state-‐level data
Total Udall Gardner
White 76 43 52
Black 4 64 25
Hispanic or Latino 16 58 39
Asian 1 66 33
Other 4 42 49
![Page 25: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/25.jpg)
UK – May 2015
![Page 26: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/26.jpg)
• Conducted April 30 to May 6
• Reported on interviews with 18,131 voters • Average click-‐thru rate of 16 percent • Compelling campaign storyline
-‐ Unpopular PM, economy as No. 1, SNP
• Conserva4ves +6 over Labour
Bri4sh elec4on project overview
![Page 27: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/27.jpg)
![Page 28: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/28.jpg)
Impact Of Images On Survey Participation, Respondents, And Online Panel Recruitment
Mingnan Liu
![Page 29: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/29.jpg)
• U.K. SurveyMonkey end page • March 12-24, 2015 • Views = 218,846 • Completes = 13,354
Experiment
![Page 30: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/30.jpg)
Experimental condition 1 - Downing Street
![Page 31: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/31.jpg)
Experimental condition 2 – Voting map
![Page 32: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/32.jpg)
Experimental condition 3 - Rosettes
![Page 33: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/33.jpg)
Findings – Click rate
n=72,474 n=72,963 n=73,459
5.9%
9.3% 8.2%
0.0%
1.0%
2.0%
3.0%
4.0%
5.0%
6.0%
7.0%
8.0%
9.0%
10.0%
Downing St Map Rosettes
Clic
k ra
te
![Page 34: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/34.jpg)
Findings – Completion rate
89.94% 89.85% 90.86%
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Downing St Map Rosettes
Com
plet
ion
rate
n=3641 n=5909 n=5250
![Page 35: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/35.jpg)
Findings – Political questions
Χ2 = 36.4, p = 0.006 Χ2 = 39.9, p = 0.002
30% 26% 27%
0%
10%
20%
30%
40%
50%
Downing St Map Rosettes
% C
onse
rvat
ive
Vote: Conservative party
32% 27% 28%
0%
10%
20%
30%
40%
50%
Downing St Map Rosettes
% C
onse
rvat
ive
Party ID: Conservative party
![Page 36: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/36.jpg)
Findings – Media questions
χ2 = 33.0, p < .001
41.7 48.4 46.9
0.0 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0
100.0
Downing St Map Rosettes
%
Social media: Several times a day
18.0 15.6 16.5
0.0
20.0
40.0
60.0
80.0
100.0
Downing St Map Rosettes
%
Watch TV: Several times a day
χ2 = 7.4, p<.05
![Page 37: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/37.jpg)
Non-probability surveys online: Does “empanelment” affect results?
Sarah Cho
![Page 38: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/38.jpg)
Survey Thanks Page
![Page 39: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/39.jpg)
Survey Thanks Page - Recruitment into One Survey
![Page 40: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/40.jpg)
Methods • Conducted same survey on SurveyMonkey “survey
thanks” page and SurveyMonkey Audience (panel)
• Thanks page survey - Conducted October 10-November 2, 2014 - 90,932 completes throughout the US, oversamples in
competitive races
• Audience survey - Conducted October 27-November 10, 2014 - 3,393 completes in US, oversamples in HI, CA, NY
![Page 41: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/41.jpg)
Opinions on Issues Similar, Party ID Different Weighted results Thanks Page Audience Right direction 28% 32%
Wrong track 70 66
Most important issue
Economy 45% 48%
Health care 21 24
Illegal immigration 17 14
Foreign policy 15 13
Have twitter account 27% 30%
Democrat 31% 35%
Independent 30 34
Republican 30 20
Other 7 8
NOTE: No answer not shown
![Page 42: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/42.jpg)
Differing “Call to Action”
85%
82%
53%
41%
RV
LV/Already voted
Follow what's going on in government most of the time
Very interested in midterm election
NOTE: No answer not shown
79%
74%
41%
30%
RV
LV/Already voted
Follow what's going on in government most of the time
Very interested in midterm election
Thank You Page Audience
![Page 43: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/43.jpg)
Potential Additional Weighting Variables
Weighted demographics Thanks Page Audience Have children under 18 in HH 31% 22%
Currently married 56% 47%
Religious attendance More than once a week 11% 9% Once a week 22 16 A few times a month 12 9 A few times a year 27 26 Never 25 39
Identify as having no religion 16% 24%
NOTE: No answer not shown
![Page 44: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/44.jpg)
Predictors of Completion Rates in Online Surveys
Noble Kuriakose
![Page 45: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/45.jpg)
25,080 customer surveys
• Administered on SurveyMonkey’s online panel, Contribute
• Conducted between May 2011 and April 2015 • At least 100 respondents per survey
Data
![Page 46: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/46.jpg)
Findings – Start simple
6,607 1,072 1,381 15,950
Mean Completion Percentage by Type of Opening Question
84.9 86.9
83.5
88.7
60
65
70
75
80
85
90
95
100
Introductory Text/Image Matrix/Rating Scale Open Ended Simple Multiple Choice
![Page 47: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/47.jpg)
Findings – As with any survey, limit questions Mean Completion Percentage by Number of Questions
60
65
70
75
80
85
90
95
100
1 4 7 10 13 16 19 22 25 28 31 34 37 40
At least 100 surveys in each category.
![Page 48: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/48.jpg)
Findings – Avoid multiple tough questions Mean Completion Percentage by Number of Questions
At least 100 surveys in each category.
60
65
70
75
80
85
90
95
100
0 1 2 3 4 5 6 7 8 9 10
Matrix/Rating
Open Ended
![Page 49: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/49.jpg)
OLS Regression, Highest (Absolute) Standardized Beta Coefficients
Findings – What matters most?
Variable (ref. group) Beta # of Open Ends- 5 or more (1) -0.19 # of Words in Question Text (continuous) -0.11 # of Answer Options (continuous) -0.11 # of Words in Opening Question Text (continuous) -0.10 # of Open Ends- 3 (1) -0.09 # of Open Ends- 0 (1) +0.09 # of Open Ends- 4 (1) -0.07 # of Matrix Qs- 5 or more (1) -0.07 # of Matrix Qs- 0 (1) +0.05
All Coefficients are stat. sig. at <.0001
![Page 50: SurveyResearch @ SurveyMonkey SurveyMonkey @ AAPORpapor.ipower.com/wp-content/uploads/2015/08/SurveyMonkey... · 2016-06-16 · thanks” page and SurveyMonkey Audience (panel) •](https://reader035.vdocuments.net/reader035/viewer/2022081502/5f04bba67e708231d40f6eba/html5/thumbnails/50.jpg)
• Survey meta and para data explain 20% (R2) of the variance in completion percentages– ignoring the type and content of the survey.
• As with all surveys, aim to reduce respondent burden • Respondents penalize surveys for making them use their
keyboards • Avoid blocks of text in the question
Conclusions