Extended Cognitive Walkthrough

Judy Kay

CHAI: Computer human adapted interaction research group

School of Information Technologies

2

Overview

• Predictive method • (with caveat Grigoreanu et al)

• Cognitive Walkthrough

• Benefits

• Disadvantages

(About 50,000 results from Google)

3

Postconditions for this class

• Describe the uses of Cognitive Walkthrough• Describe the processes for conducting Cognitive

Walkthrough analyses• Describe advantages and limitations

• Ability to perform a Cognitive Walkthrough study • Justify the use of Cognitive Walkthrough in the

overall testing of a pervasive computing application

4

• Focus on learnability• Appropriate for novice or casual users

• cf Think-Aloud?

• Does not focus on speed• cf Think-Aloud?

• Sequence is not known prior to inspection• Assesses user success and recovery from errors• Conducted by experts

• cf Think-Aloud?

5

Model of Exploratory Learning

• User has task– 0. will user understand this sub-task is needed?

• Explores system for useful elements– 1. will correct action be obvious? eg button visible

– 2. will user understand instructions? eg user understands the label on the button

• Selects one to try• User interprets system response

– 3. will user know if progress has been made?

Extended Cognitive Walkthrough

Takes account of mental model…..

7

• User has task– 0. will user understand this sub-task is needed

(given their mental model)

• Explores system for useful elements– 1. will correct action be obvious?

– (given their mental model)

– 2. will user understand instructions?

– (given their mental model)

• Selects one to try• User interprets system response

– 3. will user know if progress has been made?

– (given their mental model)

8

Example with basic CW

• Design a cash-operated machine for quick, easy purchase of train tickets by passengers, without training

• Abstract user goals:• Buy a ticket to the required destination

• Determine whether I can afford to buy the ticket to a particular destination

Adapted from Newman and Lamming,

Interactive System Design, 1995)

9

Example with basic CW

• Design a cash-operated machine for quick, easy purchase of train tickets by passengers, without training

• Designer goal breakdown to subgoals:• Determine fare to pay

– indicate destination

– indicate one-way or return

• Dispense ticket– pay money

– get ticket and change

Adapted from Newman and Lamming,

Interactive System Design, 1995)

10

The interface – lo-fi prototype

Concrete user task

Class activity: define 3 concrete tasks

12

The interface – lo-fi prototype

Concrete user task

Buy a one-way ticket to Town Hall

14

The task:

Buy a one-way ticket to Town Hall

15

How designer wants it to work – method 1 where use does not use keypad• Click destination

• ie Town Hall

• Click journey type• ie one way

16

• 0. Task?– Buy one-way ticket to Town Hall

• 1. Is correct action obvious?– Two possibilities:

• Destination

• One way / return

– Will user know both must

be set?

17

How to Help the User?

• Indicate extra information needed• Indicate steps 1 and 2 can be done in either order• Give some feedback on effect of each select action• Reorganise layout so that Steps 1 and 2 are

followed by the fare display

We alter our lo-fi prototypeand check this

19

• 2. Will user understand instructions?

• 3. Will user interpret machine action correctly?

20

• 2. Will user understand instructions?– Yes – due to labels and instructions

• 3. Will user interpret machine action correctly?– Yes (buttons light up, new state appears)

21

22

• 1. Correct action obvious?– Yes

• 2. Will user understand instructions?– Yes

• 3. Will user interpret machine action correctly?– Yes

23

Paying

• 1. Correct action obvious?

• 2. Will user understand instructions?

• 3. Will user interpret machine action correctly?

What are the cases to consider for payment (in cash)

Consider case where user has exact change

25

Paying

Designer intends user to • Pay in money and click • Click “4. Press for ticket”• User then lifts flap to collect the ticket

26

Paying – exact change case

• 1. Correct action obvious?– Yes

• 2. Will user understand instructions?– Yes

• 3. Will user interpret machine action correctly?– Unclear (no feedback on money accepted so far)

We alter our lo-fi prototypeand check this

28

Revised design gives feedback on amount paid so far

Consider other cases eg user realises they have insufficient money

30

Buying a Ticket: Insufficient Money?

• 1. Correct action obvious?

31

Summary of Flaws (so far)

• Confusion about need for steps 1 and 2• No feedback on amount deposited• No means to get money back• So far …...

32

Goals and Tasks

In this example:

Goal: buy a ticket

Sub-goals: (determine fare) and pay

User tasks: concrete cases used in CW

33

Extended cognitive walkthrough

• Adding user's mental model

• What does user believe?• How do you find this out?• What did we assume about the user’s mental

model?• What differences are there in the MM for:

• A novice user

• An expert user

Class activity:

List aspects of the user’s MM that would be relevant to the train ticket interface

What are the implications of some likely cases:

user familiar with existing interfaceuser familiar with a different bus ticket interface

Class activity on Extended CW:

37

Interface is info3315 website

Task

• INFO3315 student has just started semester and wants to determine the deadline for the first assessed work

• Define the relevant mental model

38

Mental Model

• Class web site is at

http://www.ug.it.usyd.edu.au/~info3315

• There is a lecture and a lab each week for most classes

• There is assessed practical work for most classes

• There are fixed deadlines for such work

39

Task sequence

• Go to class web site is at

http://www.ug.it.usyd.edu.au/~info3315

• [This is the home page]

• See the heading “Labs and Deadlines”

• Scan down to “Deadline: Prototype demos”

• This is listed under 19 Sep, Week 8

40

• User has task– 0. will user understand this sub-task is needed

(given their mental model)

• Explores system for useful elements– 1. will correct action be obvious?

– (given their mental model)

– 2. will user understand instructions?

– (given their mental model)

• Selects one to try• User interprets system response

– 3. will user know if progress has been made?

– (given their mental model)

41

• User has task– 0. will user understand this sub-task is needed – Yes

• Explores system for useful elements– 1. will correct action be obvious? – Perhaps not

– may go to “assessment page” as mental model primes them for the word “Assessment” and that page does not have dates

– may miss the “Deadline” in the heading– 2. will user understand instructions?– Perhaps not

– may well see Sep 19 (lecture date) and think that is the deadline

• Selects one to try• User interprets system response

– 3. will user know if progress has been made? – In this simple task the user does not need to take action

(so we can fit all this on a slide)

42

Potential GOTCHAs

• Need to have a prototype that is complete enough for a walkthough of an interesting concrete case

• But this is a technique for early in design process, where designer is ready to change it

• You then need to define:• the users, and their relevant MMs• a good set of tasks• the correct steps (intended by designer)

• Evaluator must imagine people’s thoughts on first using this interface• Keep referring to the mental model• Carefully assess vocabulary/text in terms of mental model

• Repeat process over the tasks, and for each important class of mental model

43

Returning to the reading…. A very impure CW that call ICW

• Additional forms of CW

• Expert evaluator leads the process

• Works with development team reps

• Agile process

• Some Validation (cf lab trial of SSCW part)

• Two stages:• Earlier is SSCW – “Simplified Streamlined”

• Later involves users in SPW – “Simplified Pluralistic”

44

Summary of uses

• Relatively inexpensive in our very, very lightweight approach

• Desk check– No users

– Better with expert evaluators

• Generally applicable

• Novice, casual, intermittent users

• Focus on learnability

45

Summary of usefulness

• Really useful technique, even for designer

• Better if done by • Outsider

• Expert

• But students and non-expert evaluators still can gain from using it

• Part of early usability evaluation because• Low cost

• No users needed