1

Soleh U. AyubiSHRS, University of Pittsburgh

Feb 24, 2012

Mobile Apps

USABILITY STUDY in Small Population

2

Outline of Presentation

1Intro to Usability Study

2Sample Size

3Mobile Usability Study

4Resources

3

Usability Study Introduction

Definition, goals, and methods

1

4

The goal is to identify any usability problems, collect quantitative data on participants' performance (e.g., time on task, error rates), and determine participant's satisfaction with the product.

Usability testing is a technique to evaluate a product by testing it with representative users. In the test, these users will try to complete typical tasks while observers watch, listen and take notes.

Usability Study Introduction

Usability.gov

5

Performance vs. Subjective MeasuresWe measure both performance and subjective (preference) metrics.

Testing the System, NOT the Users!Help them understand that they are helping us test the prototype or Web site.

Usability Study Introduction

Find the Best SolutionMost projects have to deal with constraints of time, budget, and resources. Balancing all those is one of the major challenges of most projects.

6

2 Sample Size

History, timelines, and the magic of five

7

History

Cambrian Explosion

Dot-Com Boom RebelionPre-

CambrianClarification

1981: Chapanis and colleagues suggest five to six users reveals most of the problems in a usability test1982: Jim Lewis propose a more precise estimate of a sample size of 5-6

2000: Nielsen publishes the widely cited article: "Why you only need to test with five users", which summarizes the past decade's research. Its graph comes to be known as the "parabola of optimism."

Skepticism builds over the magic number five2001: Jared Spool & Will Schroeder show that serious problems were still being discovered even after dozens of users

2006: In a paper based on the panel at UPA, Carl Turner, Jim Lewis and Jakob Nielsen review the criticisms of the sample sizes formulas and show how it can and should be legitimately used.

Usability Study Sample Size

1990: Robert Virzi proposed:-Additional subjects are less likely to reveal new information-The first 4-5 users find 80% of all problems-Severe problems are more likely to be detected by the first few users.

Measuringusability.com

8

9

10

11

Important Claims with empirical evidences

• Most usability problems are detected with the first three to five subjects.

• Running additional subjects during the same test is unlikely to reveal new information.

• Severe usability problems are detected by the first few participants (3-4)

12

Virzi (1992), Nielsen and Landauer (1993), Lewis (1994), Turner, Lewis, and Nielsen (2006)

13

The magic of five!

N(1-(1-L)n)L = 0.31

Useit.com

The real goal of usability study is not just document the weaknesses…

It’s to improve the design!

Choose SEVERAL REPETITIVE SMALL study

Instead of once SINGLE BIG study

15Turner, Lewis, Nielsen (2006)

Usability Results depend on:

• Properties of the system and interface.• Stage of the usability lifecycle.• Type and quality of the methodology used• Match between the test and the context of

real world usage• Representativeness of the test participants• Skill of evaluator

16

Turner, Lewis, and Nielsen (2006)

17

3 Mobile Usability Study

Objectives, methods, and tools

18

• Usability for mobile apps is not the same for desktop apps: input method, screen size, computing power, and connection speed.

Mobile Usability Study Introduction

• Navigation is paramount important

19

Mobile Usability Study Introduction

• Page Layout: the overall effectiveness of the page layout.

• Color Schemes: This can include an evaluation of the use of colors in the background, text, links, icons, button and other aspect of your mobile app.

• Findability: This will help you determine your application’s level of accessibility. Mobile users have far less patience than web users, so these items need to be placed in the best spot possible.

20

Mobile Usability Study Location

• Seems to be sufficient when studying user interface and navigation issues

• Do not simulate the context where mobile phones are used (Johnson 1998) and lack the desired ecological validity.

• Data collection techniques such as think aloud, video recording or observations being difficult in the field.

• It may require extra effort from test users and the tester.

Laboratory Field

21

Mobile Usability Study Procedures

• Usually using a think aloud protocol based on K. A. Ericsson and H. A. Simon (1980, 1984).

• Usually using scale (questionnaire) and open-ended question (depth-interview).

• Scale– Dumas and Redish (1993) use four point scale (the

most to the least) severity problem– Kallio et al. (2004) use three categories of severity

(high/critical, medium/serious, low/minor)

22

Mobile Usability Study Scale Definition

• Critical: If we don't fix this, users will not be able to complete the scenario.

• Serious: Many users will be frustrated if we don't fix this; they may give up.

• Minor: Users are annoyed, but this does not keep them from completing the scenario.

23

Mobile Phone Usability Questionnaire (MPUQ)

Young Sam Ryu, Tonya L. Smith-Jackson. Reliability and Validity of the Mobile Phone Usability Questionnaire (MPUQ). Journal of Usability Studies (2006)

24

Mobile Usability Study Data Collection & Analysis

Quantitative Data• Example: success rates, task

time, error rates, and satisfaction questionnaire ratings.

• Calculation and analysis:– percentage of participants

who succeeded or not at each task

– average time to complete tasks

– frequency of specific problems

Qualitative Data• Example: observations

about pathways participants took, problems experienced, comments, and answers to open-ended questions

• Look for patterns and trends, and add a description of each of the problems.

25

Mobile Usability Study Quantitative Report

Tsai et al., 2007

26

Tsai et al., 2007

27

Usability Study Resources

Web sites, papers, and tools

1 4Resources

Usability Study Resources

• Usability.gov• User Vision UK• UseIt.com - Jacob Nielsen Website • Nielsen Norman Group• MeasuringUsability.com• Nokia Usability Testing• Journal of Usability Studies• Usability Sciences• ACM Transaction on Human-Computer Interaction

28

29

Done! ? But wait… Is there any question?

Thanks!