Survey Research in HCIHendrik Müller, Aaron Sedley and Elizabeth Ferrall-Nunge

Neha Sharma and Mingkun Gao

Outline

● Background○ History○ When to Use○ How to do Good Survey

Research● Examples of Research● Summary

Background

What is a survey?

● Gathering information ● From a subset of people● Generalized to the target

population

History of Using Survey

When to Use

● Attitudes: Measure and represent attitudes.● Intent: Collect people’s reasons for using an application.● Task success: Quantify levels of success.● User experience feedback: Collect open-ended feedback

about a user’s experience.

When to Use (Cont’)

● User characteristics: Figure out users’ characteristics.● Interactions with technology: Understand how they interact

with technologies.● Awareness: Understand awareness of features.● Comparisons: Compare attitudes across different

applications.

When Avoid to Use

● Precise Behaviors○ Use Log data

● Underlying motivations○ Use contextual inquiry.

● Usability evaluations○ Use Task-based observational research and interview methods, such as

usability studies.

Using Survey with other Methods

● Follow previous qualitative studies to help quantify specific observations

● Initially identify high-level insights that can be followed by in-depth research through more qualitative methods

How to make a good survey?

● Research goals and constructs

● Population and sampling

● Questionnaire design and biases

● Review and survey pretesting

● Implementation and launch

● Data analysis and reporting

Research Goals and Constructs

● Define a clear research goal and constructs

○ Example of Research Goal and constructs

■ Research Goal: Understand users’ happiness with an online application

■ Happiness is a multidimensional concept, separate it into measurable pieces (Constructs)

■ Constructs: overall satisfaction, perceived speed and perceived utility

Population and Sampling

● Population: People to whom researchers wish to generalize their results.● Sampling frame: the set of people who the researcher is able to contact for the

survey.● Sample: the people from the sampling frame who are invited to take the survey.● Respondents: samples who answer the survey.

Sampling Methods

● Sampling Methods

○ Probability (Random) Sampling

■ Gold standard because every person in the sampling frame has an equal chance

■ Avoid selection bias

○ Non-Probability Sampling

■ Impossible to reach and randomly select from the target population, e.g. small population, sensitive and rare behavior

Determining the Appropriate Sample Size● Determine the size of population

being studied.● Choose the required level of

precision.○ Margin of error○ Confidence level○ E.g. In a study [CL = 95%, MoE=5%],

80% samples use computer everyday

● Estimate how many people to invite.○ Response rate

● Invitation Methods: Mail, Phone, In-person and Internet.

Questionnaire Design and Biases

● Open-ended questions: Ask survey respondents to write in their own answers.

● Closed-ended questions: provide a set of predefined answers to choose from.

Questionnaire Bias

● Satisficing Bias: use a suboptimal amount of cognitive effort to answer questions.

● When this will happen:

○ Cognitive ability to answer is low○ Motivation to answer is low○ Question difficulty is high

To minimize Satisficing bias:

● Avoid too complex questions

● Avoid options like “No Opinion”, “Don’t know” or “Unsure”

● Avoid long questionnaire for increasing fatigue

● Show importance of the survey to increase respondents motivation

● Add some trap questions (e.g. Please input 5 in the following text box) to identify satisficers

(Satisficing)

● Acquiescence Bias: When presented with agree/disagree, yes/no, or true/false statements, some respondents are more likely to concur with the statement independent of its substance.

● When this will happen:

○ Agreeableness Personality ○ Social conventions, saying ‘yes’ is

polite○ Only think about why it is true○ Assume the survey administrator

agrees with the statement

To minimize Acquiescence bias

● Avoid yes/no, agree/disagree or true/false questions

● Ask construct-specific questions (i.e., questions that ask about the underlying construct in a neutral, non-leading way) instead of agreement statements

● Use reverse-keyed constructs; i.e., the same construct is asked positively and negatively in the same survey. The raw scores of both responses are then combined to correct for acquiescence bias.

Questionnaire Bias (Acquiescence)

● Social Desirability Bias: respondents answer questions in a manner they feel will be positively perceived by others

● When this will happen:

○ Provide information for sensitive topics

○ Perceive a threat/consequence of answering truthfully

○ True identity was captured○ The data is collected by another

person directly

To minimize Social Desirability bias

● Answer anonymously.

● The survey should be self-administered.

Questionnaire Bias (Social Desirability)

● Response Order Bias: Select the items toward the beginning (i.e., primacy effect) or the end (i.e., recency effect) of an answer list or scale.

● Question Order Bias: the potential to bias subsequent questions by priming respondents.

To minimize Response Order Bias:

● Unrelated answer options should be randomly ordered across respondents

To minimize Question Order Bias:

● Questions should be ordered from broad to specific, similar to our conversation convention

● Early questions should be easy to answer and directly related to the survey topic

● The questionnaire should be divided into multiple pages with distinct sections labeled for easier cognitive processing

Questionnaire Bias (Order Related Bias)

Other Types of Questions to Avoid

● Leading question

○ E.g. This application was recently ranked as number one in customer satisfaction. How satisfied are you with your experience today?

● Double-barreled question:

○ E.g. How satisfied are you with your smartphone and tablet computer?

● Recall question:

○ E.g. How many times did you use an Internet search engine over the past 6 months?

● Prediction question:

○ Over the next month, how frequently will you use an Internet search engine?

Visual Design Considerations

● Visual design of a survey, such as use of image, may affect participants’ performance

○ E.g. Please rate your level of health. Image of patient vs. Image of Strong man

● Larger text fields - increased amount of information but may also intimidate respondents and can result in higher break-offs.

Leveraging Established Questionnaire

An alternative to constructing a brand new questionnaire is utilizing questionnaires developed by others.

Commonly used Questionnaires in HCI:

NASA Task Load Index (NASA TLX): Originally developed for aircraft cockpits, this questionnaire allows researchers to subjectively assess the workload of operators working with human–machine systems. It measures mental demand, physical demand, temporal demand, performance, effort, and frustration.

Launch Survey

● Review and Survey Pretesting: At this point in the survey life cycle, it is appropriate to have potential respondents take and evaluate the survey in order to identify any remaining points of confusion.

○ Cognitive testing

○ Field testing

● Implementation and Launch: Google surveys, survey monkey

○ How to maximize response rate: Make the survey as short as possible and provide incentives to participants.

Data Analysis and Report

● Data Preparation and Cleaning○ Duplicate responses, Speeders, Straight-liners, Inadequate open-ended responses, handle missing

data and break-offs.

● Analysis of Closed-Ended Responses○ Descriptive statistics○ Inferential statistics

■ Estimation statistics: margin of error, confidence interval.■ Hypothesis testing.

● Analysis of Open-Ended Comments○ Coding the responses: intra-coding reliability and inter-coding reliability.

● Report result○ Answer the research question.○ Describe original research goals and used survey methodology.

Examples of Research

“Why did you enroll in this course?” Developing a Standardized Survey Question for Reasons to Enroll

Developed for Massive Open Online Courses (MOOCS).

Survey research used to scale and personalize courses based on user’s motivations behind enrollment.

Responses were open-ended questions from previous courses and coded.

Social Presence in Web Surveys

Study to prove the merits of social interface theory on computer administered surveys (which are used to cancel social desirability effects).

Asked questions about socially desired traits and socially undesired traits.

Social interface effects could not be replicated.

Social Presence in Web Surveys (Cont’d)

Keep on Lockin’ in the Free World: A Multi-National Comparison of Smartphone Locking

Research was conducted to understand the differences in attitude towards smartphone unlocking and indirectly information security.

Questions were translated in multiple languages.

Keep on Lockin’ in the Free World: A Multi-National Comparison of Smartphone Locking

Amazon Mechanical Turk and Survey Research

“Thus, consistent with other research that has used MTurk, most workers are from the U.S. and India (Horton et al. 2011; Ipeirotis et al. 2010). While the studies examined here did not rule out other countries (e.g., Canada), the evidence at least suggests that requestors will be the most successful when recruiting workers from either India or the U.S.”

Amazon Mechanical Turk and Survey Research (Cont’d)

“In July of 2013, a HIT for a survey that was similar in length and also limited to U.S. residents was created. Workers were paid $0.50 for this HIT and it was completed in approximately five hours. The average time per assignment was nine minutes and 28 seconds. An identical HIT was created for residents of India with an average time per assignment of nine minutes and 27 seconds, virtually identical to the U.S. population.”

Summary

Make sure you have clear research goals and constructs, population and sampling, questionnaire design and biases, review and survey pretesting, implementation and launch, data analysis and reporting.

“Surveys are useful, but often misused by people who don't know the basics (where to show them, what to ask, who to ask, how to ask it). Often, people just want validation for a decision they've already made, or help making a decision they don't want to put real work into (experiments, qualitative user research).”

- Anonymous UX Researcher

ReferencesEmily Schneider and René F. Kizilcec. 2014. "Why did you enroll in this course?": developing a standardized survey question for reasons to enroll. In Proceedings of the first ACM conference on Learning @ scale conference (L@S '14). ACM, New York, NY, USA, 147-148. DOI=http://dx.doi.org.proxy2.library.illinois.edu/10.1145/2556325.2567852

Marian Harbach, Alexander De Luca, Nathan Malkin, and Serge Egelman. 2016. Keep on Lockin' in the Free World: A Multi-National Comparison of Smartphone Locking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 4823-4827. DOI: https://doi-org.proxy2.library.illinois.edu/10.1145/2858036.2858273

Mick P. Couper, Roger Tourangeau, and Darby M. Steiger. 2001. Social presence in Web surveys. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '01). ACM, New York, NY, USA, 412-417. DOI=http://dx.doi.org.proxy2.library.illinois.edu/10.1145/365024.365306

ReferencesDupius, Marc, Barbara Endicott-Popovsky, and Robert Crossler. An Analysis Of The Use Of Amazon’S Mechanical Turk For Survey Research In The Cloud. University of Washington, 2017. Web. 1 Apr. 2017.http://faculty.washington.edu/marcjd/articles/An%20Analysis%20of%20the%20Use%20of%20Amazon%E2%80%99s%20Mechanical%20Turk%20for%20Survey%20Research%20in%20the%20Cloud%20-%20v.%203.pdf