• Stewart Hutchison (shutchison@missouriwestern.edu)

• Santhosh Kasula (skasula@missouriwestern.edu)

Missouri Western State University

PSY 600

Usability Testing Report for One Bus Away,

a web-based bus routing application

http://onebusaway.org/

Date of Report: 10/18/2014

Date of Test: 10/07 – 10/08

Location of Test: MWSU Psychology Lab

Prepared for: OneBusAway Dev Team

Prepared by: Stewart Hutchison

Santhosh Kasula

Table of Contents

Executive Summary

Participants

Tasks & Procedures

Results

Usability Videos (4:55)

Major Findings

Conclusion

Executive Summary

The goal was to determine:

1. How do undergraduate students navigate a novel

online map system?

2. How do satisfaction ratings compare to experience

and frequency of use?

Participants

Four undergraduate students were recruited from MWSU:

User Age Gender

1 35 Female

2 20 Male

3 23 Female

4 22 Female

Tasks & Procedures

Each participant was given:

• Demographic questionnaire

• Tasks to complete with a laptop computer (recorded with Camtasia)

• Satisfaction survey

Tasks & Procedures

Demographic Questionnaire:

1. What is your age?

2. What is your gender?

3. Have you used an online map to plan a route for travel?

4. If so, how often do you plan trips with online maps?

Tasks & Procedures

Tasks:

1. Find the web application.

2. Locate the CAMLS building.

3. Identify bus route #30.

4. When will the next bus arrive at CAMLS?

5. Where is this bus currently?

Satisfaction Survey (SUS, Jeff Sauro 2011):

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

[http://www.measuringu.com/sus.php]

Tasks & Procedures

Methodology

• Sessions were recorded with Camtasia

• Videos were reviewed and measured:

Efficiency (Time on task)

Effectiveness (Errors and assists)

Satisfaction (SUS survey)

Results

(Time on task, seconds)

User Task #1

Find Application

Task #2

Locate CAMLS

Task #3

Identify route 30

Task #4

When next bus…?

Task #5

Where is the bus…?

#1 2 31 49 N/A N/A

#2 32 33 255 380 437

#3 82 185 170 285 402

#4 31 30 225 305 250

Task # Time on Task

(seconds)

Errors Assists

1. Find Application x̅ = 36.75, s = 33.22 6 1

2. Locate CAMLS x̅ = 69.75, s = 76.84 8 1

3. Identify Route #30 x̅ = 174.75, s = 90.92 13 0

4. When next bus…? x̅ = 329.33, s = 96.67 21 1

5. Where bus currently…? x̅ = 363.0, s = 99.41 23 3

Results (Time on task, Errors, Assists)

Results (Demographics and Satisfaction)

User Used online maps? How Often? Satisfaction?

(SUS)

#1 Yes Weekly 5

#2 No N/A 5

#3 Yes Monthly 25

#4 Yes Weekly 47.5

Satisfaction Survey (SUS, Jeff Sauro 2011)

[http://www.measuringu.com/sus.php]

Satisfaction Survey (SUS, Jeff Sauro 2011)

[http://www.measuringu.com/sus.php]

Task 1 – Find the web application

3/4 users made errors…

Task 3 – Identify Route #30

Feedback and Google Map comparisons…

Task 4 – When will the next #30 bus arrive?

Getting a little “flustered”…

Task 5 – Where is the bus currently?

Nope… 4/4 users were unable or required assistance

Major Findings

• 3/4 were unable to correctly locate the web application

• All users compared the system to Google Maps features

• None of the users were able to locate the bus on their own

Major Findings

Positive feedback:

Major Findings

Highest satisfaction level (x̅ = 2.25, s = 1.41)

was reported for survey question #5:

“I found the various functions in this system were well integrated”

Conclusion

Each successive task was met with more difficulty, as

both average time and errors increased.

Although based in a different location, these

undergraduate students are potential users who

could transfer to your area.

What We Learned…

Ask the user questions before offering help (don’t be the nice guy)

Define concepts before reviewing videos (such as “error” and “assist”)

Recruit participants and collect data as early as possible

Questions?