survey design 201 getting from “doing a survey” to “doing a survey right.” andy zehner...
TRANSCRIPT
![Page 1: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/1.jpg)
Survey Design 201
Getting from “Doing a survey” to “Doing a survey right.”
Andy ZehnerOffice of AssessmentYoung Hall, Rm 521
![Page 2: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/2.jpg)
Basic tips
1. Don’t ask what you already know2. Avoid biased or leading questions3. Make choice options:
1. exhaustive2. exclusive3. balanced
4. Avoid double-barrelled questions5. Don’t assume knowledge of terms
6. Avoid jargon7. Be brief 8. Be specific9. Do/Don’t include a “neutral” option10. Avoid complex negative constructions11. Order questions strategically12. Save controversial questions for last
![Page 3: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/3.jpg)
Spot the flaw
1. “Rate the quality and variety of the food at Wiley Dining Court”
2. “CCO Career Services are:”• Excellent Outstanding Very good Good Average
3. “Have you participated in any flipped courses during the past year?”• Yes No
4. Were the clerical staff who served you in [office] courteous?1. Yes2. Neutral3. No
5. What portion of the material did you master by the end of the training session?• All Most Some Very little None
![Page 4: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/4.jpg)
New considerations
• Vague quantifiers• Individual response style• Dunning-Kruger Effect
![Page 5: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/5.jpg)
Dunning-Kruger Effect
![Page 6: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/6.jpg)
Kruger & Dunning, Journal of Personality and Social Psychology, 1999
![Page 7: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/7.jpg)
What evidence of Dunning Kruger Effect looks like
Very poor Poor Fair Good Very good Excellent0%
10%
20%
30%
40%
50%
60%
Freshmen 1Freshmen 2Seniors 1Seniors 2
2014 Purdue SERU survey
![Page 8: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/8.jpg)
Vague quantifiers
![Page 9: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/9.jpg)
Vague quantifiers
![Page 10: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/10.jpg)
How often have you…
Had a class where the professor knew your name
Never
Rarely
Occasio
nally
Somewhat o
ftenOften
Very often
0%
5%
10%
15%
20%
25%
30%
Communicated with an instructor by email/in person
Never
Rarely
Occasio
nally
Somewhat o
ftenOften
Very often
0%
5%
10%
15%
20%
25%
30%
2014 SERU Survey, Purdue data
![Page 11: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/11.jpg)
Vague quantifiers
• “Always” almost always means always• “Never” almost always means never• Any other vague quantifier is . . . vague• Vague quantifiers convey relative position only• Meaning varies from one person to another• Meaning varies from one question to another• Grammatical opposites aren’t perceived equally
![Page 12: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/12.jpg)
“Sometimes”
• “asking questions in class” 2 times per week• “working with other students outside of class” 1 time per month• “making a class presentation” 2 times per
semester• “participating in a community-based project” 1 time per year
How Often is Often Revisited: the Meaning and Linearity of Vague Quantifiers Used on the National Survey of Student Engagement -- Laird, et al., Indiana University, 2008
![Page 13: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/13.jpg)
Vague quantifiers of behavioral frequency, Marincic, 2011
![Page 14: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/14.jpg)
“Stay away from using words like “often,” “usually,” “generally,” etc. Each person’s thought process is different and some people may infer a different meaning.”
-- Smart Survey Design, Survey Monkey, 2011
Retrieved 3/11/2015
![Page 15: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/15.jpg)
Consider using idiosyncratic terms!
2013 Purdue Student Satisfaction Survey
![Page 16: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/16.jpg)
Response style
![Page 17: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/17.jpg)
Moderate & Extreme Response
Bursar’s drop-box
Very unim
portant/d
issati
sfied
Unimporta
nt/diss
atisfi
ed
Neither
Importa
nt/Sati
sfied
Very im
portant/s
atisfi
ed0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Importance Satisfaction
Registering for classes
Very unim
portant/d
issati
sfied
Unimporta
nt/diss
atisfi
ed
Neither
Importa
nt/Sati
sfied
Very im
portant/s
atisfi
ed0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Importance Satisfaction
2010 Purdue Student Importance & Satisfaction Survey
![Page 18: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/18.jpg)
Quotes from focus groups, Purdue, 2012-2013
“I feel like here in the US people use stronger
language, like ‘Oh, this is so wonderful.’ But for me,
I’m not so willing to say it.”
“Americans use a lot of hyperbole: ‘It’s super, it’s
wonderful!’ That is not language we use where
I come from.”
![Page 19: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/19.jpg)
Factors that determine acquiescence
• Individualism -.445 *• Power distance .372 *• Extraversion .446 **• Uncertainty avoidance .533 *
Response styles in cross-national survey research: a 26-country study, Harzing, et al, 2006
![Page 20: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/20.jpg)
Response styles by nationality
30 35 40 45 50 55 60 655
10
15
20
25
30
Acquiescence
Extr
eme
Resp
onse
Sty
le
Japan
India
Germany
Holland
Mexico
Turkey
China
![Page 21: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/21.jpg)
Some words don’t translate well
“good” v. 良い / いい
“normal” v. нормальный
![Page 22: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/22.jpg)
Does America spend too little, about the right amount, or too much money on…
Too little About right Too much0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
"Welfare" "Assistance to the poor"
General Social Survey, 2014
![Page 23: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/23.jpg)
Again: Does America spend too little, about the right amount, or too much money on…
Liberal respondents
Too little About right Too much0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
"Welfare" "Assistance to the poor"
Conservative respondents
Too little About right Too much0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
"Welfare" "Assistance to the poor"
![Page 24: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/24.jpg)
Order of choice options matters
Purdue’s favorite pizza:
Hot Box / Mad Mushroom / Villa@PMU / Domino’s
Domino’s / Villa@PMU / Mad Mushroom / Hot Box
Biasing Effects of Scale-Checking Style in Response to a Likert Scale, American Statistical Assn, Friedman, et al., 1994
![Page 25: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/25.jpg)
Numbers on the scale matter, as much as the words do
0 1 2 3 4 56 7 8 9 10
Not at allExtremely
Successfulsuccessful
-5 -4 -3 -2 -1 01 2 3 4 5
Not at allExtremely
Successfulsuccessful
Rating the Rating Scales, Journal of Marketing Management, Friedman & Amoo, 1999
![Page 26: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/26.jpg)
You can get reliable results with a moderate sample size or response rate
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 5000
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 1000
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 500
Stronglydisagree
Disagree Neutral Agree Stronglyagree
0%5%
10%15%20%25%30%35%40%
True distribution Sample of 50
![Page 27: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/27.jpg)
![Page 28: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/28.jpg)
After the shot went off, my
training took over.
![Page 29: Survey Design 201 Getting from “Doing a survey” to “Doing a survey right.” Andy Zehner Office of Assessment Young Hall, Rm 521 alzehner@purdue.edu](https://reader030.vdocuments.net/reader030/viewer/2022032518/56649cc65503460f9498fdb3/html5/thumbnails/29.jpg)
Resources and links:
A good overall lesson in 106 slides
Survey Monkey’s guide
Cross–national response styles
Rating the Rating Scales
Vague quantifiers: How Often is Often?
Vague quantifiers: Marincic dissertation
More about importance/performance analysis
Tex Grebner’s video (language)