Observation, Interviews, and Questionnaires

a.k.a. How to watch and talk to your users

Agenda

Questions? Reminder: part 3 due NEXT WEEK Bring your prototype to class Highly recommended: read on Heuristic

Evaluation and Cognitive Walkthrough, be prepared for them

Observation Interview Questionnaire Evaluation plan discussion

Observing Users

Not as easy as you think

One of the best ways to gather feedback about your interface

Watch, listen and learn as a person interacts with your system

Usually what occurs during a “usability test”

Location

Observations may beIn lab - Maybe a specially built

usability lab• Easier to control• Can have user complete set of tasks

In field• Watch their everyday actions• More realistic• Harder to control other factors

UsabilityLab

http://www.surgeworks.com/services/observation_room2.htm

Large viewing area in this one-way mirror which includes an angled sheet of glass the improves light capture and prevents sound transmission between rooms.

Doors for participant room and observation rooms are located such that participants are unaware of observers movements in and out of the observation room.

Observation

Direct In same room Can be intrusive Users aware of your

presence Only see it one

time, relies on good note-taking

May use 1-way mirror to reduce intrusiveness

IndirectVideo recordingSoftware loggingReduces intrusiveness, but doesn’t eliminate itGives archival record, but can spend a lot of time reviewing it

Engaging Users

In simple observation, you see actions but not what is going on in their head

Qualitative techniques Think-aloud - very helpful Post-hoc verbal protocol - review video Critical incident logging - positive & negative Structured interviews - good questions

• “What did you like best/least?”• “How would you change..?”

Think Aloud

User describes verbally what s/he is thinking and doing

• What they believe is happening• Why they take an action• What they are trying to do

Very widely used, useful technique Better understand user’s thought processes

Potential problems: Can be awkward for participant Thinking aloud can modify way user

performs task

Cooperative approach

Another technique: Co-discovery learning (Constructive iteration) Join pairs of participants to work together Use think aloud Perhaps have one person be semi-expert

(coach) and one be novice More natural (like conversation) so removes

some awkwardness of individual think aloud Variant: let coach be from design team

(cooperative evaluation)

Alternative

What if thinking aloud during session will be too disruptive?

Can use post-event protocolUser performs session, then watches

video afterwards and describes what s/he was thinking

Sometimes difficult to recallOpens up door of interpretation

What if a user gets stuck?

Determine in advance when and how you will offer help

Use cooperative approaches: “What are you trying to do..?” “What made you think..?” “How would you like to perform..?” “What would make this easier to accomplish..?” Maybe offer hints

Inputs

Need operational prototype could use Wizard of Oz or other simulation

Need tasks and descriptions Reflect real tasks Avoid choosing only tasks your design best

supports Minimize necessary background knowledge Pay attention to time and training required

Data

Task based How do users approach the problem What problems do users have Need not be exhaustive, look for interesting

cases Performance based

Frequency and timing of actions, errors, task completion, etc.

Analyzing data can be very time consuming!

Capturing a Session

1. Paper & pencilIs definitely cheap and easyCan be slowMay miss things

Time 10:00 10:03 10:08 10:22

Task 1 Task 2 Task 3 …

Se

Se

Capturing a Session

2. Recording (audio and/or video)Good for think-aloudHard to tie to interfaceMultiple cameras may be neededGood, rich record of sessionCan be intrusiveCan be painful to transcribe and

analyze

Capturing a Session

3. Software loggingModify software to log user actionsCan give time-stamped key press or

mouse eventTwo problems:

• Too low-level, want higher level events• Massive amount of data, need analysis

tools

Example: Heather’s study

Software: MeetingViewer interface fully functional

Criteria – learnability, efficiency, see what aspects of interface get used, what might be missing

Resources – subjects were students in a research group, just me as evaluator, plenty of time

Wanted completely authentic experience

Heather’s evaluation

Task: answer questions from a recorded meeting, use my software as desired

Think-aloud Video taped, software logs Also had post questionnaire Wrote my own code for log analysis Watched video and matched behavior to

software logs

Example materials

Example logs2303761098721869683|hrichter|1098722080134|MV|START|5662303761098721869683|hrichter|1098722122205|MV|QUESTION|false|false|false|false|false|false| 2303761098721869683|hrichter|1098724978982|MV|TAB|AGENDA2303761098721869683|hrichter|1098724981146|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098724985161|MV|SLIDECHANGE|52303761098721869683|hrichter|1098724986904|MV|SEEK|PRESENTATION-A|566|604189|02303761098721869683|hrichter|1098724996257|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098724998791|MV|SEEK|PRESENTATION-A|566|604189|6041892303761098721869683|hrichter|1098725002506|MV|TAB|AGENDA2303761098721869683|hrichter|1098725003848|MV|SEEK|AGENDA|566|149613|6041892303761098721869683|hrichter|1098725005981|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725007133|MV|SLIDECHANGE|32303761098721869683|hrichter|1098725009326|MV|SEEK|PRESENTATION|566|315796|1496132303761098721869683|hrichter|1098725011569|MV|PLAY|566|3157962303761098721869683|hrichter|1098725039850|MV|TAB|AV2303761098721869683|hrichter|1098725054241|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725056053|MV|SLIDECHANGE|22303761098721869683|hrichter|1098725057365|MV|SEEK|PRESENTATION|566|271191|3157962303761098721869683|hrichter|1098725064986|MV|TAB|AV2303761098721869683|hrichter|1098725083373|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725084534|MV|TAB|AGENDA2303761098721869683|hrichter|1098725085255|MV|TAB|PRESENTATION2303761098721869683|hrichter|1098725088690|MV|TAB|AV2303761098721869683|hrichter|1098725130500|MV|TAB|AGENDA2303761098721869683|hrichter|1098725139643|MV|TAB|AV2303761098721869683|hrichter|1098726430039|MV|STOP|566|2711912303761098721869683|hrichter|1098726432482|MV|END

Data analysis

Basic data compiled: Time to answer a question (or give up) Number of clicks on each type of item Number of times audio played Length of audio played User’s stated difficulty with task User’s suggestions for improvements

More complicated: Overall patterns of behavior in using the

interface User strategies for finding information

Data representation example

Data presentation

Time spent answering question Time

playing Timeline Artifact Meeting # Subject # Q1 Q2 Q3 Q4 audio seeks seeks

1 1 10:10 12:16 5:33 25:40 40 9 2 2 1:44 3:19 3:00 4:59 13:57 44 0 2 3 4:20 2:54 5:18 0 10:22 35 3 3 4 0:45 2:43 2:36 0:59 10:41 2 3 3 5 2:23 0 0 2:59 3:11 8 2 4 6 6:13 7:53 2:53 12:16 14:36 9 7 5 7 4:16 3:14 0:27 4:14 21 0 5 8 8:01 4:41 1:33 9:51 2 3 6 9 4:45 0 0:59 5:59 12:27 53 3 6 10 3:22 0 1:20 2:00 6:56 15 5 6 11 0 2:40 1:33 2:12 6:49 19 10 7 3 3:04 1:35 5:52 2:36 14:13 10 11 7 6 0 1:03 0 0:29 0:00 0 0 7 12 NA NA NA NA 23:15 98 5 8 3 2:00 0:00 1:04 1:13 1:20 8 0 8 13 2:36 2:13 2:41 2:21 9:03 81 0 8 14 1:15 4:20 0:00 2:36 3:28 15 3 9 3 0 5:19 0 2:24 8:26 50 4 9 6 7:57 0:52 2:13 2:30 12:17 48 1 9 12 0 5:00 0 0 3:11 33 3 9 14 7:52 0:49 1:22 0:19 9:32 56 0 9 15 0 7:42 0 0:53 6:51 36 0 9 16 1:22 7:04 2:24 1:07 10:31 38 3 9 17 5:56 6:56 0:56 1:03 11:50 160 4

10 3 15:07 11:34 4:26 31:08 97 14 10 6 6:57 5:10 2:37 19:54 91 4 10 12 6:06 0 3:38 10:14 45 14 10 13 5:53 5:15 3:06 12:25 125 5 10 14 3:59 5:14 3:50 7:27 13 16 10 15 9:04 5:03 1:54 13:17 31 14 10 17 6:40 7:32 5:06 17:46 94 18 10 18 0 6:57 7:24 10:13 15 14

Average Over all questions: 4:04 11:05 43.5 5.6 St. Dev. 2:58 6:45 39.2 5.4

2 4 6

AgendaAction ItemPresentation

Timeline

0

40

20

2 4 62 4 6

AgendaAction ItemPresentation

Timeline

0

40

20

AgendaAction Item

PresentationTimeline

160

40

80

120

0

0 2 4 6 8 10 12 14 16 18 20

Time (minutes) in session

Loc

atio

n (m

in.)

in m

eeti

ng

reco

rd.

AgendaAction Item

PresentationTimeline

160

40

80

120

0

0 2 4 6 8 10 12 14 16 18 20

Time (minutes) in session

Loc

atio

n (m

in.)

in m

eeti

ng

reco

rd.

Some usability conclusions

Need fast forward and reverse buttons (minor impact)

Audio too slow to load (minor impact) Target labels are confusing, need

something different that shows dynamics (medium impact)

Need more labeling on timeline (medium impact)

Need different place for notes vs. presentations (major impact)

Interviews & Questionnaires

Subjective view of participants Quantitative – very structured

Questionnaires• often quantitative, but not entirely

Structured Interviews• Strict set of questions, deviation would

compromise study Qualitative – less or no structure

Semi-structured interviews• Some deviation encouraged

Unstructured interviews• i.e. the ethnographic interview• Little guide, very explorative

Interviews

Potentially lots of detail can vary questions as needed Inexpensive Time consuming to perform and

analyze Some interpretation required Subject to interviewer biases

Questionnaires

Expensive to create …but cheap to administer Easier to get quantifiable results Can gather info from many more

people Protects participant identity Only as good as the questions asked

Structured Interviews

More similar to questionnaires Require a lot of training for any hope

at inter-interviewer reliability But that means that they tend to give

much more repeatable results

Unstructured Interviews

Have a plan, but keep interview open to different directions

Get participant to open up and express themselves in their terms and at own pace

Create interpretations with usersBe sure to use their terminology

Take lots of time, but learn a lot as well

Semi-Structured Interviews

Predetermine data of interest - know why you are asking questions - don’t waste time

Plan for effective question types• How do you perform task x?• Why do you perform task x?• Under what conditions do you perform task x?• What do you do before you perform…?• What information do you need to…?• Whom do you need to communicate with to …?• What do you use to…?• What happens after you…?

See Gordon & Gill, 1992; Graesser, Lang, & Elofson, 1987

Asking Questions

Understand your goals Consider the ordering of the questions Avoid complex/long/multiple questions Avoid jargon; talk in participant’s

language Be careful of stereotypes, biases

Clarity is important

Questions must be clear, succinct, and unambiguous

How much time have you spent reading news on the Web recently? Some A lot Every day Rarely Etc.

None

0 to 5 hours

6 to 10 hours

11 to 20 hours

More than 20 hours

Avoid question bias

Leading questions unnecessarily force certain answers.Do you think parking on campus can be

made easier?

What is your overall impression of…1.Superb

2.Excellent

3.Great

4.Not so Great

Be aware of connotations

Do you agree with the NFL owner’s decision to oppose the referee’s pay request?

Do you agree with the NFL owner’s decision in regards to the referee’s pay demand?

Do you agree with the NFL owner’s decision in regards to the referee’s suggested pay?

Leading questions

People want to do well, give you what you are looking for

Be aware of your own expectations before creating questions and while interviewing

Use value neutral terms

What do you like about this system?

Vs.

Tell me what you thought about this system.

Avoid hypotheticals

Avoid gathering information on uninformed opinions

Subjects should not be asked to consider something they’ve never thought about (or know or understand)

Would a device aimed to make cooking easier help you?

Handle personal info carefully Ask questions subjects would not

mind answering honestly.What is your age?What is your waist size?

If subjects are uncomfortable, you will lose their trust

Ask only what you really need to know

What’s wrong with this picture?

How much easier is it to use this email client than Outlook?

I see you choose to use your keyboard shortcuts more than the mouse. Is that faster for you?

Your choice of red is different than any other user we saw. Why did you do that?

Planning your interview:

Introduction Warmup Main session Cool-off Closing

Record everything exactly in your participants’ languages

(don’t forget to test your recording equipment)

The warmup or “grand tour” question The first question helps set the tone for the

interview Familiarize the participant to talking Encourage the participant that their true

opinion does matter Question should be

Easy to answer But not answered easily

• More than just a “yes” or “no” response Examples:

Tell me about the work you do? What made you buy the computer?

Prompts

“Nudge” a participant in a direction, or to get additional response Silent: remain silent until they say more Echo: repeat back and then ask “then what

happens” etc. Make agreeing sounds: you say “uh huh”

and the other person continues Tell Me More: could you tell me more about

that? Clarifying: summarize and ask for

confirmation or clarification, often leads to new discussion

Contents of a survey

General/Background infoDemographic dataAlso functions as as a “warm up”Correlate responses between groups

Objective questions

Open-ended/subjective

Background examples

Demographic data:Age, genderTask expertise

• i.e. Have you ever worked in a restaurant?

MotivationFrequency of use

• How often do you…Education/literacy

• What training have you had in …?

Closed Format

Advantages Clarify alternatives Easily quantifiable Eliminate useless

answer

Disadvantages Must cover whole

range All should be equally

likely Don’t get interesting,

“different” reactions

Restricting set of choicesQuantifiable

Many forms of response

Dichotomous Multiple Choice Multiple Response Rank/Match Likert Rating

Questionnaire Styles

LaTeX

FrameMaker

WordPerfect

Word

Rank from1 - Very helpful2 - Ambivalent3 - Not helpful0 - Unused

___ Tutorial___ On-line help___ Documentation

Which word processingsystems do you use?

Likert-type scale

Typical scale uses 5, 7 or 9 choices Above that is hard to discern Doing an odd number gives the neutral

choice in the middle You may not want to give a neutral

option

Characters on screen were:

hard to read easy to read 1 2 3 4 5 6 7

What’s wrong with this picture?

2. What is your age? _______________3. How long have you used the internet?<1 year1-3 years3-5 years>5 years4. How do you get information about courses?EmailWeb siteFlyersRegistration bookletAdvisorOther students

5. How useful is the Internet in getting information about courses?______________________________________________________________________________________________________________________

On line questionnaires

Email or internet Change checkboxes into dropdowns,

etc Take advantage of the technology –

check input Ensure its as accessible as paper

(browser and email client compatibility) Ensure confidentiality – how is this

different from paper?

Free Web Survey Tools

Zoomeranghttp://www.zoomerang.com

Survey Monkeyhttp://www.surveymonkey.com

phpESPhttp://phpesp.sourceforge.netOpen Source surveys using PHP.

Analyzing your quantitative data

“Code” open ended responses or interview questions to make quantitative Categorize all responses

Look for trends in the data Count, average, tabulate Make charts, etc Run statistical analysis Use lo-fi methods (post-its, affinity diagrams,

etc)

Analyzing qualitative data

Find interesting cases, responses Look for patterns of responses

Use post-its, affinity diagrams, etc. Look for any useful suggestions,

improvements, explanations that help you improve your design

Gather illustrative quotes from users that demonstrate your conclusions

Evaluation discussion

Someone else should be able to pick up your plan and execute it.

Be as SPECIFIC as possibleWhat criteria are important?What tasks EXACTLY?What data? How will you record?What questions will you ask?