jesper kjeldskov & jan stage department of computer science aalborg university denmark new...

Jesper Kjeldskov & Jan StageDepartment of Computer Science

Aalborg UniversityDenmark

New Techniques for Usability Evaluation of Mobile Systems

2Kjeldskov & Stage, New Techniques for Usability Evaluation of Mobile Systems

Background

Mobile technologies and systems… PDAs, wearables, mobile phones, tablet computers

… challenge usability testing methods Users are physically mobile during use Use involve activities in physical surroundings Use context can be difficult to recreate realistically

This paper explore new techniques for usability testing mobile systems in laboratory settings


Two Evaluation Approaches

Field experiments Realistic use context Difficult to control Complicated data collection Complex and time consuming Safety and ethical issues

Laboratory experiments Experimental control

High quality data collection

Lack of realism


Laboratory vs. Field

Most usability evaluations of mobile systems are currently conducted in laboratory settings

A recent literature study revealed that… 41% of mobile HCI research involve evaluation 71% of this is done in laboratory settings

It is a widely adopted point of view that mobile systems require field evaluations, but… It is difficult to conduct field evaluations The added value of testing in the field is unknown Additional problems come at a high cost (time & effort)


Overall Research Question

How can new techniques for usability tests of mobile systems increase realism of use in a laboratory setting?

… while facilitating systematic data collection in a controlled environment

Previous studies: Specialized use contexts This study: Physical mobility during use


Two Experiments

Two experiments comparing techniques for lab- and field-based usability testing of mobile systems were conducted

Experiments explored different techniques requiring…1. Different levels of physical movement2. Divided cognitive attention

Example application: use of Short Message Service (SMS) on PDAs and mobile phones

Narrow focus on interaction rather than broad focus on use


Five Laboratory Techniques…

Attention needed to navigate

None Conscious

BodyMotion

None 1. Sitting at a table or standing

n/a

Constant 2. Walking on a treadmill with

constant speed

4. Walking at constant speed on a

changing track

Varying 3. Walking on a treadmill with varying speed

5. Walking at varying speed on a

changing track

The experiment: measuring the relative strengths and weaknesses with reference to field testing


The Laboratory Experiments

5 conditions (6 test subjects per condition) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)


The Field Experiment

1 condition: walking in a pedestrian street (6 test subjects) Number of usability problems Performance (task completion time) Subjective workload (NASA TLX)


Findings (1)

More problems found on average when seated at a table Statistical significance Lab techniques with physical movement comparable to

field evaluation

Mean number of usability problems identified by each technique

1. Sitting at a table

2. Walking on a treadmill with constant speed

3. Walking on a treadmill with varying speed

4. Walking at constant speed on a changing track

5. Walking at varying speed on a changing track

6. walking in a pedestrian street


Findings (2)

Techniques Total

Lab 1 Lab 2 Lab 3 Lab 4 Lab 5 Field

Critical 4 4 3 4 3 3 4

Serious 11 11 9 9 9 8 17

Cosmetic 19 8 8 8 6 12 32

Total 34 23 20 21 18 23 53

No technique identified all problems Most problems found when seated at table (34) Comparable numbers of critical problems found (3-4)

Number of identified usability problems categorized by severity


Findings (2)

Techniques Total


Critical 4 4 3 4 3 3 4

Serious 11 11 9 9 9 8 17

Cosmetic 19 8 8 8 6 12 32

Total 34 23 20 21 18 23 53

More than double the number of cosmetic problems were found while seated compared to the other lab techniques



Findings (2)

Techniques Total


Critical 4 4 3 4 3 3 4

Serious 11 11 9 9 9 8 17

Cosmetic 19 8 8 8 6 12 32

Total 34 23 20 21 18 23 53

Only 3 out of 4 critical usability problems were identified on basis of the field technique



Findings (3)

Techniques


Mental demands 29 75 204 126 185 148

Physical demands 92 117 112 118 127 194

Effort 52 163 106 228 178 186

Overall workload 27 35 48 55 48 54

Sitting at a table (lab 1) required significantly less mental activity compared to all other techniques but lab 2

Overall, sitting or walking at constant speed is experienced significantly less demanding than any other technique

Subjective experience of workload with the different techniques


Discussion (1)

Seating test subjects at a table… superior? Supported the identification of most usability problems Difference mostly accounted for by cosmetic problems

Less workload facilitated more thinking-aloud Relevance of cosmetic problems can be questioned

Increasing workload in lab Helped approximating the field condition but resulted in

fewer problems identified compared to when being seated

Better focus missing vital problems??


Discussion (2)

Usability problems and mobility Physical motion in the lab triggered unique interaction

problems also found in the field Added value in relation to e.g. layout and button sizes

Data collection in the field All field tests were recorded with

a camcorder Difficult to capture good images

of screen The “bodyguard” effect Changing the role of the test monitors? Mounting small cameras on test subject and device?


Future work

Cost/benefit of different techniques and settings Time and effort spent per problem found

When should lab or field tests ideally be applied?

How can field test techniques be improved? Improving data collection? Enforcing more experimental control? Doing something completely different?

jesper kjeldskov & jan stage department of computer science aalborg university denmark new...

Documents

kjeldskov stage

use slide

use use

systems pdas

different techniques

mobile phones narrow

mobile hci research

field testing slide