COMP5047 Pervasive Computing: 2012COMP5047 Pervasive Computing: 2012

Think-aloud usability experiments or concurrent verbal accounts

Judy Kay

CHAI: Computer human adapted interaction research group

School of Information Technologies

Overview

• Empirical methods

• Think-aloud

• Benefits

• Disadvantages

• Naturalised think-aloud

Postconditions for this week

• Describe the uses of Think Alouds• Describe the processes for conducting one• Describe advantages and limitations

• Ability to use Think-aloud as relevant and for your project

• Justify the use of Think Aloud in the overall testing of a pervasive computing application

• Be ready to do Think Aloud evaluations for your part of your project

Think aloud protocols

• Ask user to “think aloud” as they use the interface

• Often used with video, audio taping• Otherwise MUST make notes

• Helps observer interpret what is going on• Gives qualitative data mainly• 3-5 users may be enough (Nielsen) for

each stage of refining the prototype

Case study

• Test usability of teacher’s data projection facilities in this classroom

• What is intended functionality?• Formulate some concrete tasks – write

these as instructions– eg Suppose you currently have the projector

screen down but you want to use the whiteboard. So you want make the screen go up.

Case study

• Work in your groups to identify 3 tasks relevant to testing the teacher’s use of projection facilities in this room

Case study

• Call for volunteers to be users for this trial• Have not used these facilities before

• Have used them in other places in the uni

• Call for volunteers to conduct the trial• All will make helpful notes

Classification of think-aloud

• formative versus summative• predictive v empirical• laboratory v naturalistic• qualitative v quantitative

• For the project, you goal will be?

Design cycle

• User Centred Design• Define criteria for success• Define concrete tasks users should be able

to do - use these in evaluation• Prototype construction• Usability study

– Decide just what data to collect– Test design of experiment for timing (trial it)– Recruit users– Run study

• Goto top

Recruiting users

• How representative are they?– similarity to intended user population

– Age

– Gender

– experience in area

– interest/motivation

– computer literacy

• What effect does user population have for conclusions?

Stages of running an evaluation

1. Preparation

2. Introduction

3. The test

4. Questionnaire/interview

5. Debriefing

6. Analysis, reflection, summarising, reporting, conclusions for action

Steps 1- 5 done for each user test, as run

Step 6 is applied mainly after several users

Preparation

• Set up machine, room, environment• Check all of them• Check user instructions• Do a mental run-through

Be sure not to waste user's time because of your lack of preparation!

Introduction

• Welcome user, explain purpose of test– make clear system tested not user

– confidentiality

– anonymity of reporting

– opt out at any time

– what is recorded

• Invite any other questions to here– explain procedure

– if appropriate, do demo

– invite questions

The test

• User works through experiment....– recording

– ensure user feels supported

– show pleasure at problems identified

– critical to help user if stuck

• Questionnaire/interview– open and closed

What data should you collect?

• Observe – direct/indirect

– take notes

– video/audio/software monitor

– software logs for timing

• Questionnaire: – open

– closed

Debriefing

• Thank user• Remind them of usefulness of results• Pause to make sure all data collected• All notes written• May ask user to confirm details collected

Pitfalls

• Defining the right concrete tasks– Test all key aspects

– Multiple tasks for same aspects

• Instructions to the users– Do NOT lead the user

– Take particular care not to use words that are identical to terms on the interface

Benefits of think aloud

• “show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations”

(Nielsen, Usability Engineering, Academic press 1993, p195)

• Cheap• Slows users down

– studies show users may work faster with fewer errors due to care on critical elements

Problems of think aloud

• Not directly quantitative

• Add cognitive load to users

• User's “theories” must be interpreted with care

• Slows users down

• Users are aware they are being observed so behave accordingly

Facilitating think aloud

• What are you thinking now?• What do you think that message means?

(only after the user has noticed the message and is clearly spending time on it)

• don't help user except withHow do you think you can do it?• if user appears surprised, Is that what you expected to happen?

Naturalised think-aloud

• Multi-user interaction– Two (or more) users work on task

– Conversation is natural

– Observer collects dialogue

Problems with observing users

• Hawthorn effect

• People rationalise (Telling more than we can know)

• Qualitative data

Class activity - foreshadowed

• Formulate three concrete tasks for testing the console of the lecture theatre

• Write these out ready for use in an experiment

• After next part of the seminar: each group will conduct one practice Think-Aloud with the rest of the class writing feedback on how this was done.

Summary

• Relatively inexpensive

• Can identify major flaws

• And may indicate causes of user problems

• May give access to user's mental model

• Alters activity => meaningfulness