online data collection: tips for effective visual design online data collection bosr... · 2018....
TRANSCRIPT
-
Online Data Collection: Tips for Effective Visual Design
Lindsey Witt-Swanson
Assistant Director
Bureau of Sociological Research
-
Bureau of Sociological Research
• Created in 1964 as the data collection vehicle for the Sociology Department
• Directed by a member of the Sociology faculty
• Served all UNL colleges in the past five years
• Fee-for-service organization
• Currently have 63 projects in the field
2
-
• Research Design Consultation• Survey, questionnaire, and sample design• Cost/Budget estimates
• Data Collection• Mail, phone, and web surveys• In-person and cognitive interviewing• Focus group facilitation• Program evaluation
• Data Processing• Entry/Verification, coding, and analysis• Transcription• Technical report preparation
• Research Support• Recruitment• IRB preparations• Trainings
3
-
Expertise through continuous research and education• Director: Jolene Smyth, PhD
• Co-Author of Internet, Phone, Mail, and Mixed-Mode Surveys with Don Dillman
• Active research agenda looking at minimizing survey error
• Staff• Four staff members have MS in survey research• Continuous education through UNL courses, short
courses at conferences, webinars, and internal, professional development
• Participation in professional meetings to stay informed of the latest developments in survey research methodologyand add to it through our original research
• Practical experience
4
-
Visual Design Matters
• Visual design impacts if and when respondents see the parts of our questionnaire.
• If a question, response option, instruction, or answer space is unseen, it will not be processed.
• If it is seen out of order, it may be processed in unintentional ways.
• It also impacts how respondents proceed through the rest of the response process.
• How do they comprehend the question?• How do they retrieve information?• How do they formulate a judgment?• How do they map their response to our questionnaire?
5
-
We need to be cooperative communicators
• In a self-administered survey, the visual design/layout is part of what we (the researchers) are saying.
• Respondents assume it is not irrelevant; rather, it has meaning.
• So they try to make sense of it – to read meaning into it.
• The meaning they read into it will impact their response process.
• This is… • Bad if our visual design is arbitrary and done without thought or strategy
• Good if we can harness visual design as an additional way to communicate to respondents (e.g., to help them comprehend a question).
6
-
An example of visual design impacting data
7
Source: Couper, Mick P., Michael W. Traugott, and Mark J. Lamias. 2001. “Web Survey Design and Administration.” Public Opinion Quarterly. 65(2):230-253.
-
8
Invalid responses included
things like “about 3” or
“between 4 and 5.”
11.3% provided invalid
responses
20.7% provided invalid
responses
The size of the answer box impacted respondents’ comprehension of the response task, particularly, how exact their answer should be.
-
Graphics can impact data
9
How would you rate your health?
Excellent
Very Good
Good
Fair
Poor 33% said excellent or very
good
41% said
excellent or
very good
The images likely impacted how respondents defined the vague word “healthy” and the information they retrieved and used to form their judgment.
Source: Couper, Mick P., Frederick G. Conrad, and Roger Tourangeau. 2007. “Visual Context Effects in Web Surveys.” Public Opinion Quarterly. 71(4):623-
634.
-
Picking the right question format
• If we are serious about data quality, functionality and usability should always come before branding and entertainment.
• Use the format that gets you the data you need• NOT what looks fun or entertaining
• NOT what is the most aesthetically pleasing
10
-
11
-
12
-
Sliders
• “Fun” features can have an impact on the data you collect
• Sliders are interactive and different
• Is this the real answer or did they not answer the question?
13
-
Side-by-Side Questions
• A way to get a lot of items in a small amount of space
• Very burdensome (multiple questions at once) and can train respondents to skip questions
• Will not work on mobile
14
-
Other Visual Design Concepts
15
• Processing Concepts• Bottom-up processing
• Top-down processing
• Preattentive processing
• Attentive processing
• Foveal view
• Useful field of view
• Figure/ground composition
• Visual Elements• Words
• Numbers
• Symbols
• Graphics
• Visual Properties• Size• Font• Brightness/contrast• Color• Shape• Orientation• Static vs. motion• Etc.
• Gestalt Grouping Principles• Proximity• Similarity• Pragnanz• Closure• Common Region• Continuity• Elemental Connectedness
-
Nonsubstantive Answer Options
• Don’t Know, Not Applicable, No Opinion
• Programs will automatically treat nonsubstantive answer options just like the substantive answer options
• Can impact data when it shifts the visual midpoint
16
-
Programming Matters: Skip and Loops
• Example Skip: Females get breast cancer questions, Males get prostate cancer questions
• Example Loop: Ask the same questions for each doctor visit in the last 30 days
• If they are wrong during administration, people will be asked the wrong questions leading to the very least missing data but worse possible breakoffs
• Skips and loops need to be checked, rechecked, and checked one more time
17
-
Radial buttons versus Check-all boxes• Radial buttons indicate one answer
• Boxes indicate more than one answer
• They need to be used consistently to train the respondent of what you want
18
-
Answer Option Values
• Best to get them right during programming to save a lot of time recoding on the back end
• Example: Lime automatically makes answer options A1, A2, etc.• Have to change to 1, 2, and so on to be usable in later analyses
• If you don’t do this before hand, you have to recode all of your questions before you can run any statistical tests
19
-
Check-All Programming
• Enter and then look at test data before fielding to uncover errors in the programming missed any other way
20
-
Checks
• Read through
• Check skip patterns and loops
• Answer the questions• Helps notice things missed in the read through
• Export the data you entered to make sure it is saving and exporting correctly.
21
-
Other Issues to consider
• You need to know what the program automatically collects• Qualtrics automatically collects IP addresses, and won’t let you shut it off
• Matters to IRB and respondent confidentiality
• Who owns and has access to the data?• Qualtrics: The data stays on the Qualtrics servers even after you export it
• Lime: The client owns the data and we can erase it from the Lime server because we own and maintain the server
22
-
Summary
• Visual Design matters• We need to think about it or our design may have unintended consequences
on our data
• Programming matters• Yes, you can fix some errors/issues on the back end, but certainly not all of
them
• Being educated about your web survey platform matters• It can dictate data ownership, confidentiality matters, and more
23
-
This is the tip of the iceberg
• Call BOSR• This is what we do
• We stay up-to-date on all the quirky, software update changes
• We work through these quirks with the IRB regularly
• We offer researchers two to three hours of free consultation
• Jolene will do another SBSRC short course on visual design in the spring
• Watch for announcements!
24
-
Thank you!
Lindsey Witt-Swanson
Bureau of Sociological Research
University of Nebraska-Lincoln
25
mailto:[email protected]