Asking Users and ExpertsAsking Users and Experts

Changsung Moon

Jaeyoung Lee

The Main AimsThe Main AimsDiscuss when it is appropriate to use different

types of interviews and questionnaires.

Teach you the basics of questionnaire design.

Describe how to do interviews, heuristic evaluation, and walkthroughs.

Describe how to collect, analyze, and present data.

Discuss the strengths and limitations of these techniques.

The Types of one-on-one The Types of one-on-one InterviewsInterviews

Unstructured (or open-ended)◦ Questions or topics are open-ended, interviewee is

free to answer in any manner, and topics do not have to be covered in any particular order. Ex) Suggest ways of collecting the interview data.

Structured◦ Most controlled type.◦ Interview may consist of closed-ended questions

and interviewee must choose from the options provided. Ex) Would you like to receive news from CNN? Yes No Don’t know

Semi-structured◦ Combination of structured and unstructured

Ex) Would you listen to the news from CNN? If yes, why? If no, why not?

The Types of one-on-one The Types of one-on-one InterviewsInterviews

Unstructured (or open-ended)◦ Questions or topics are open-ended, interviewee is

free to answer in any manner, and topics do not have to be covered in any particular order. Ex) Suggest ways of collecting the interview data.

Structured◦ Most controlled type.◦ Interview may consist of closed-ended questions

and interviewee must choose from the options provided. Ex) Would you like to receive news from CNN? Yes No Don’t know

Semi-structured◦ Combination of structured and unstructured

Ex) Would you listen to the news from CNN? If yes, why? If no, why not?

The Types of one-on-one The Types of one-on-one InterviewsInterviews

Unstructured (or open-ended)◦ Questions or topics are open-ended, interviewee is

free to answer in any manner, and topics do not have to be covered in any particular order. Ex) Suggest ways of collecting the interview data.

Structured◦ Most controlled type.◦ Interview may consist of closed-ended questions

and interviewee must choose from the options provided. Ex) Would you like to receive news from CNN? Yes No Don’t know

Semi-structured◦ Combination of structured and unstructured

Ex) Would you listen to the news from CNN? If yes, why? If no, why not?

The Types of one-on-one The Types of one-on-one InterviewsInterviews

Unstructured (or open-ended)◦ Questions or topics are open-ended, interviewee is

free to answer in any manner, and topics do not have to be covered in any particular order. Ex) Suggest ways of collecting the interview data.

Structured◦ Most controlled type.◦ Interview may consist of closed-ended questions

and interviewee must choose from the options provided. Ex) Would you like to receive news from CNN? Yes No Don’t know

Semi-structured◦ Combination of structured and unstructured

Ex) Would you listen to the news from CNN? If yes, why? If no, why not?

Guidelines for Developing Guidelines for Developing interview questions (Things to interview questions (Things to avoid)avoid) Long questions

◦ They are difficult to remember. Compound sentences

◦ Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”

◦ Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)

Using jargon◦ The interviewee may not understand.

Leading questions◦ Ex) Wrong: “Most of our users prefer the new look and feel of our site

over the old one. How do you feel?”◦ Right: “How do you feel about the visual appearance of this website?”

Unconscious biases◦ Ex) Wrong: “Safety experts recommend using a travel agent instead

of booking your travel online. Do you feel safe using travel websites?”◦ Right: “Do you feel that booking travel online is more or less

confidential than booking through a travel agent?”

Guidelines for Developing Guidelines for Developing interview questions (Things to interview questions (Things to avoid)avoid) Long questions

◦ They are difficult to remember. Compound sentences

◦ Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”

◦ Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)

Using jargon◦ The interviewee may not understand.

Leading questions◦ Ex) Wrong: “Most of our users prefer the new look and feel of our site

over the old one. How do you feel?”◦ Right: “How do you feel about the visual appearance of this website?”

Unconscious biases◦ Ex) Wrong: “Safety experts recommend using a travel agent instead

of booking your travel online. Do you feel safe using travel websites?”◦ Right: “Do you feel that booking travel online is more or less

confidential than booking through a travel agent?”

Guidelines for Developing Guidelines for Developing interview questions (Things to interview questions (Things to avoid)avoid) Long questions

◦ They are difficult to remember. Compound sentences

◦ Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”

◦ Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)

Using jargon◦ The interviewee may not understand.

Leading questions◦ Ex) Wrong: “Most of our users prefer the new look and feel of our site

over the old one. How do you feel?”◦ Right: “How do you feel about the visual appearance of this website?”

Unconscious biases◦ Ex) Wrong: “Safety experts recommend using a travel agent instead

of booking your travel online. Do you feel safe using travel websites?”◦ Right: “Do you feel that booking travel online is more or less

confidential than booking through a travel agent?”

Guidelines for Developing Guidelines for Developing interview questions (Things to interview questions (Things to avoid)avoid) Long questions

◦ They are difficult to remember. Compound sentences

◦ Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”

◦ Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)

Using jargon◦ The interviewee may not understand.

Leading questions◦ Ex) Wrong: “Most of our users prefer the new look and feel of our site

over the old one. How do you feel?”◦ Right: “How do you feel about the visual appearance of this

website?”

Unconscious biases◦ Ex) Wrong: “Safety experts recommend using a travel agent instead

of booking your travel online. Do you feel safe using travel websites?”◦ Right: “Do you feel that booking travel online is more or less

confidential than booking through a travel agent?”

Guidelines for Developing Guidelines for Developing interview questions (Things to interview questions (Things to avoid)avoid) Long questions

◦ They are difficult to remember. Compound sentences

◦ Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”

◦ Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)

Using jargon◦ The interviewee may not understand.

Leading questions◦ Ex) Wrong: “Most of our users prefer the new look and feel of our site

over the old one. How do you feel?”◦ Right: “How do you feel about the visual appearance of this website?”

Unconscious biases◦ Ex) Wrong: “Safety experts recommend using a travel agent instead

of booking your travel online. Do you feel safe using travel websites?”◦ Right: “Do you feel that booking travel online is more or less

confidential than booking through a travel agent?”

Guidelines for Developing Guidelines for Developing interview questions (Things to interview questions (Things to avoid)avoid) Long questions

◦ They are difficult to remember. Compound sentences

◦ Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?”

◦ Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record)

Using jargon◦ The interviewee may not understand.

Leading questions◦ Ex) Wrong: “Most of our users prefer the new look and feel of our site

over the old one. How do you feel?”◦ Right: “How do you feel about the visual appearance of this

website?”

Unconscious biases◦ Ex) Wrong: “Safety experts recommend using a travel agent instead

of booking your travel online. Do you feel safe using travel websites?”◦ Right: “Do you feel that booking travel online is more or less

confidential than booking through a travel agent?”

Timeline for an InterviewTimeline for an Interview

Group InterviewsGroup Interviews ‘focus group’: one form of group interview

Normally 3 to 10 people are involved

Participants are selected to provide a representative sample of typical users; ◦ they share certain characteristics

Benefit is that it allows diverse or sensitive issues that would otherwise be missed

QuestionnairesQuestionnairesQuestions can be closed or open (similar to

interview)◦ Closed questions are easier to analyze

can be administered to large populations◦ Paper, email and the web used for dissemination

Electronic questionnaires◦ Data goes into a database and is easy to analyze

Online questionnaires◦ Sampling can be a problem when the size of a

population is unknown

Advice for Designing a Advice for Designing a QuestionnaireQuestionnaire

Make questions clear and specific When possible, ask closed questions and offer a range of

answers Consider including a “no-opinion” option General questions should precede specific ones Avoid complex multiple questions The range of answers is appropriate and does not overlap

◦ Ex) 15-20, 20-25 (X) 15-19, 20-25 (O)

The ordering of scales should be intuitive and consistent◦ Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high

agreement

Avoid jargon Prove clear instructions on how to complete the

questionnaire Keep the questionnaire as compact as possible

Advice for Designing a Advice for Designing a QuestionnaireQuestionnaire

Make questions clear and specific When possible, ask closed questions and offer a range of

answers Consider including a “no-opinion” option General questions should precede specific ones Avoid complex multiple questions The range of answers is appropriate and does not overlap

◦ Ex) 15-20, 20-25 (X) 15-19, 20-25 (O)

The ordering of scales should be intuitive and consistent◦ Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high

agreement

Avoid jargon Prove clear instructions on how to complete the

questionnaire Keep the questionnaire as compact as possible

Advice for Designing a Advice for Designing a QuestionnaireQuestionnaire

Make questions clear and specific When possible, ask closed questions and offer a range of

answers Consider including a “no-opinion” option General questions should precede specific ones Avoid complex multiple questions The range of answers is appropriate and does not overlap

◦ Ex) 15-20, 20-25 (X) 15-19, 20-25 (O)

The ordering of scales should be intuitive and consistent◦ Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high

agreement

Avoid jargon Prove clear instructions on how to complete the

questionnaire Keep the questionnaire as compact as possible

Advice for Designing a Advice for Designing a QuestionnaireQuestionnaire

Make questions clear and specific When possible, ask closed questions and offer a range of

answers Consider including a “no-opinion” option General questions should precede specific ones Avoid complex multiple questions The range of answers is appropriate and does not overlap

◦ Ex) 15-20, 20-25 (X) 15-19, 20-25 (O)

The ordering of scales should be intuitive and consistent◦ Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high

agreement

Avoid jargon Prove clear instructions on how to complete the

questionnaire Keep the questionnaire as compact as possible

Advice for Designing a Advice for Designing a QuestionnaireQuestionnaire

Make questions clear and specific When possible, ask closed questions and offer a range of

answers Consider including a “no-opinion” option General questions should precede specific ones Avoid complex multiple questions The range of answers is appropriate and does not overlap

◦ Ex) 15-20, 20-25 (X) 15-19, 20-25 (O)

The ordering of scales should be intuitive and consistent◦ Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high

agreement

Avoid jargon Prove clear instructions on how to complete the

questionnaire Keep the questionnaire as compact as possible

Advice for Designing a Advice for Designing a QuestionnaireQuestionnaire

Make questions clear and specific When possible, ask closed questions and offer a range of

answers Consider including a “no-opinion” option General questions should precede specific ones Avoid complex multiple questions The range of answers is appropriate and does not overlap

◦ Ex) 15-20, 20-25 (X) 15-19, 20-25 (O)

The ordering of scales should be intuitive and consistent◦ Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high

agreement

Avoid jargon Prove clear instructions on how to complete the

questionnaire Keep the questionnaire as compact as possible

Question and Response Question and Response FormatFormat

Different types of questions require different types of responses.

Check boxes and Ranges◦ Ex) under 21 21-65 over 65

Likert Scales◦ Ex) The use of color is excellent: (where 1

represents strongly agree and 5 represents strongly disagree)

◦ 1 2 3 4 5

Semantic differential scales◦ Ex) Exciting Boring

What to Do with the DataWhat to Do with the Data Identify any trends or patterns

Using a spreadsheet like Excel can help in initial analysis

Often simple statistics are needed

Bar charts can be used to display data graphically

More advanced statistics can be used◦ show whether there is a relationship between question

and responses

ASKING EXPERTSASKING EXPERTS

Why Asking Experts?Why Asking Experts?Sometimes users are not easily

accessible.Involving users is too expensiveEvaluating a interface design takes

so long time.

Skillful experts can capture many of the usability problems by themselves.

We can employ just few experts.

Two Different Technologies Two Different Technologies for Asking Expertsfor Asking Experts

Inspections◦Heuristic Evaluation

Walkthrough◦Cognitive Walkthrough◦Pluralistic Walkthrough

Heuristic EvaluationHeuristic Evaluation

Heuristic EvaluationHeuristic EvaluationA discount method for quick,

cheap, and easy evaluation of the user interface.

Developed by Jacob Nielsen in the early 1990s based on heuristics ◦distilled from an empirical analysis of

249 usability problems

Heuristic Evaluation -Heuristic Evaluation -10 10 HeuristicsHeuristicsVisibility of system statusMatch between system and the real worldUser control and freedomConsistency and standardsHelp users recognize, diagnose, and recover

from errorsError preventionRecognition rather than recallFlexibility and efficiency of useAesthetic and minimalist designHelp and documentation

“Are users kept informed about what is going on?”“Is appropriate feed back provided within reasonable time about a user’s action?”

Heuristic EvaluationHeuristic EvaluationRevised for current technology◦HOMERUN for the Web : These heuristics

are more useful for evaluating commercial websites. High-quality content Often updated Minimal download time Ease of use Relevant to users’ needs Unique to the online medium Netcentric corporate culture

Heuristics still needed for new technologies◦mobile devices, wearable interfaces, virtual

worlds, etc.

How many Experts?How many Experts?Nielsen found that about 5

evaluations found 75% of the problems

Above that you get more, but at decreasing efficiency

Three Stages for Doing Three Stages for Doing Heuristic EvaluationHeuristic EvaluationBriefing session to tell experts what

to doEvaluation period of 1-2 hours in

which:◦Each expert works separately◦Take one pass to get a feel for the

product◦Take a second pass to focus on specific

featuresDebriefing session in which experts

work together to prioritize problems

Advantages and ProblemsAdvantages and ProblemsBest experts have knowledge of

application domain & usersFew ethical & practical issues to

considerCan be difficult & expensive to find

expertsBiggest problems◦important problems may get missed◦many trivial problems are often

identified

WalkthroughsWalkthroughs

Cognitive Walkthroughs

Pluralistic Walkthroughs

Cognitive WalkthroughsCognitive Walkthroughs

“Cognitive walkthroughs involve simulating a user’s problem-solving process at each step in the human-computer dialog, checking to see if the user’s goals and memory for actions can be assumed to lead to the next correct action.”

(Nielsen and Mack, 1994)

Four Steps of Cognitive Four Steps of Cognitive WalkthroughsWalkthroughs

Analysis TasksEvaluate each taskCompile the record of critical

informationSuggest revised design

Steps involved in Cognitive Steps involved in Cognitive Walkthroughs – First StepWalkthroughs – First StepThe characteristics of typical users

are identified and documented. Sample tasks are developed that

focus on the aspects of the design to be evaluated.

A description or prototype of the interface is also produced, along with a clear sequence of the actions needed for the users to complete the task.

Steps involved in Cognitive Steps involved in Cognitive Walkthroughs – Second StepWalkthroughs – Second StepDesigners and one or more

expert evaluators then come together to do the analysis

The evaluators walk through the action sequences for each task, placing it within the context of a typical scenario.

Steps involved in Cognitive Steps involved in Cognitive Walkthroughs – Second StepWalkthroughs – Second StepWhile experts are evaluating

each task, they try to answer the following questions◦Will the correct action be sufficiently

evident to the user?◦Will the user notice that the correct

action is available?◦Will the user associate and interpret

the response from the action correctly?

Steps involved in Cognitive Steps involved in Cognitive Walkthroughs – Third StepWalkthroughs – Third StepAs the walkthrough is being

done, a record of critical information is compiled in which:◦The assumptions about what would

cause problems and why are recorded.

◦Notes about side issues and design changes are made.

◦A summary of the results is compiled.

Steps involved in Cognitive Steps involved in Cognitive Walkthroughs – Fourth StepWalkthroughs – Fourth StepThe design is then revised to fix

the problems presented.

Pluralistic WalkthroughsPluralistic Walkthroughs

What is Pluralistic What is Pluralistic Walkthrough?Walkthrough?“Another type of walkthrough in

which users, developers and usability experts work together to step through a scenario, discussing usability issues associated with dialog elements involved in the scenario steps”

(Nielsen and Mack, 1994)

Steps of Doing Steps of Doing Pluralistic WalkthroughsPluralistic WalkthroughsScenarios are developed in the

form of a series of hard-copy screens

The scenarios are presented to the panel of evaluators.

The panelists discuss the actions that they suggested for that round of the review.

Then the panel moves on to the next round of screens.

What is benefit of What is benefit of Pluralistic Walkthrough?Pluralistic Walkthrough?Pluralistic walkthroughs have the

advantage of providing a diverse range of skills and perspectives.

The result of pluralistic walkthroughs includes a strong focus on users’ tasks.

The approach also lends itself well to participatory design practices by involving a multidisciplinary team in which users play a key role.

Thank you~Thank you~

Any question?Any question?