unmoderated, online usability testing for web (brandon kopp & bill mockovak)
DESCRIPTION
Given at UXPA-DC's User Focus Conference, Oct. 19, 2012TRANSCRIPT
![Page 1: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/1.jpg)
The Use of an Unmoderated, Online
Usability Testing Service to Test a Website
Brandon Kopp & Bill Mockovak
Research PsychologistOffice of Survey Methods ResearchUXPA-DC User Focus Conference
19 October 2012
![Page 2: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/2.jpg)
2
TryMyUI
• Provides panel of participants that are presented with user defined tasks
• You receive a video screenshot of the participant’s computer with an audio voiceover of their comments
![Page 3: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/3.jpg)
Many Alternatives
OpenHallway.com
UserTesting.com
Loop11.com
UTest.com
Userzoom.com
OptimalWorkshop.com
![Page 4: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/4.jpg)
The BLS.gov Dropdown Usability Test
• During earlier usability testing, several users had problems with immediately activating dropdown menus
• Users would accidentally click menu links and get lost
![Page 5: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/5.jpg)
Prototypes
• Test design alternatives for mega-dropdowns
Prototype Type of Menu Control Display Width List of Subject Areas
A Click to show/close Fixed, 3 columns Partial list of content, with ‘view all’ option
B Hover, no delay Fixed, 1 column Compact
C Hover, with delay Fixed, 3 columns Current content
D Hover, no delay Fixed, 3 columns Current content
![Page 6: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/6.jpg)
Prototype A
• Need to click on tab to open
• Click on X or Arrow to close
• Abbreviated list is initially presented
• User clicks View All to see entire list
![Page 7: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/7.jpg)
Prototype B
• Dragging mouse over menu will open it
• Menu closes when you move mouse off it (no clicking is necessary)
• Higher-level categories are displayed
![Page 8: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/8.jpg)
Prototype C
• Current, ‘full’ list of topics
• Menu drops after delay, with mouseover
• Moving off menu makes it disappear quickly
![Page 9: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/9.jpg)
Prototype D
• Control Condition• Current menu on
BLS.gov• Menu drops
immediately (no delay) with mouseover
• Moving off menu makes it disappear quickly
![Page 10: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/10.jpg)
Method
• Define Participant Criteria
• Participants complete up to 10 tasks while providing verbal feedback
Gender: Any Age: 18-55
Country: U.S. Income: Any
Education: Any Employment: Any
Computer Experience: Beginner-Intermediate or Expert
Prototype# of
ParticipantsA 11
B 14
C 10
D 10
![Page 11: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/11.jpg)
Tasks
1. Find a publication called the Occupational Outlook Quarterly. This is an online magazine about jobs and careers.
![Page 12: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/12.jpg)
Tasks
1. Occupational Outlook Quarterly2. Mass Layoffs3. International Unemployment4. Green Jobs5. Survey of Occupational Illness and Injury6. Strikes and Lockouts7. Students and Teachers8. NY State Wages
![Page 13: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/13.jpg)
14
What the Participant Sees
Task Instructions
Browser / Webpage View
Recording Time
![Page 14: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/14.jpg)
What The Experimenter Sees
Video of Participant’s Screen
Task Index
![Page 15: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/15.jpg)
Example Videos
Feedback (A; Task 8)
Feedback (D; Task 6)
![Page 16: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/16.jpg)
Task Success and Time To Complete
A B C D0
1
2
3
4
5
6
7
8
6.26.6
7.4
6.7
# of
Tas
ks C
ompl
eted
(out
of 8
)
A B C D0
20
40
60
80
100
120
96.9
83.887.8
77.8
Tim
e to
Com
plet
e Ta
sk (i
n se
c.)
# of Tasks Successfully Completed Time to Complete Task
![Page 17: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/17.jpg)
First Click
Task Home Subject Areas
Databases & Tools
Economic Release Publications Elsewhere
on page
1 0 3 3 0 32 7
2 3 17 16 7 0 2
3 3 26 9 6 0 1
4 2 33 4 2 0 2
5 1 34 6 1 1 0
6 3 38 3 1 0 0
7 4 25 2 1 4 7
8 1 25 9 1 0 6
![Page 18: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/18.jpg)
Method
• Define Participant Criteria
• Participants complete up to 10 tasks while providing verbal feedback
• Following testing, participants write responses to 4 open-ended questions
Gender: Any Age: 18-55
Country: U.S. Income: Any
Education: Any Employment: Any
Computer Experience: Beginner-Intermediate or Expert
Prototype# of
ParticipantsA 11
B 14
C 10
D 10
![Page 19: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/19.jpg)
Open-Ended Questions
Easy/Very Easy Neither Easy Nor Difficult Difficult/Very Difficult0
10
20
30
40
50
60
70
80
47
20
33
61
0
39
67
17 17
69
8
23
A B C D
% o
f Par
ticip
ants
Sel
ectin
g Ra
ting
• How easy were these tasks to complete? Were they very easy, easy, neither easy nor difficult, difficult, or very difficult? How easy do you think these tasks would be for an average American citizen using this website?
![Page 20: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/20.jpg)
Feedback/Recommendations• Final Recommendation:
Prototype C
• Several participants were surprised that the Home tab had a menu
• No items under Workplace Injuries subcategory on Subject Areas menu
![Page 21: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/21.jpg)
Feedback/Recommendations
• Subject Areas menu extended off the screen on some participants’ browsers requiring scrolling
![Page 22: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/22.jpg)
Advantages
Method CostTraditional, In-Lab $40 per participantTryMyUI.com $35 per participantUserTesting.com $39 per participantUTest.com --OpenHallway.com $49-$199 per monthLoop11.com $350 per project
• Cost ($)
![Page 23: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/23.jpg)
Advantages
• Cost ($)• Cost (Time)
Task In-Lab WebRequesting participants 20 minutes total;
explaining criteria to recruiter 30 minutes total;
specifying test groups and criteria
Screening participants 10 minutes per participant 0 minutes; done by TryMyUI
Scheduling participants 15 minutes per participant 0 minutes; study done at participant convenience
Preparing for interviews 10 minutes per participant 60 minutes total; setting up web survey and tasks
Total (for 45 interviews)
26.6 hours 1.5 hours
Data collection period 3 weeks; based on interviewer schedule
1 – 2 days;Depends on participant criteria
![Page 24: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/24.jpg)
Advantages
• Cost ($)• Cost (Time)• Short data collection period• Participants that can be selected based on criteria• Videos can be shared• Participants are skilled/trained – At thinking aloud– At dealing with problems– At evaluating websites
• Can get replacements for unusable cases
![Page 25: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/25.jpg)
Disadvantages
• No follow-up• Cannot correct navigation errors• Limited to 20 minutes*• Task completion time, success, and other quantitative
methods have to be captured manually*• No rating scales*• Panel participants may have selection biases that
make them different from a ‘typical’ user
* May be different on other testing services
![Page 26: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/26.jpg)
Discussion
OpenHallway.com
UserTesting.com
Loop11.com
UTest.com
TryMyUI.com
• What testing service have you used?• What are the advantages/disadvantages of
that service?
Userzoom.com
OptimalWorkshop.com
![Page 27: Unmoderated, Online Usability Testing for Web (Brandon Kopp & BIll Mockovak)](https://reader033.vdocuments.net/reader033/viewer/2022061118/545c12aeaf7959b9098b466a/html5/thumbnails/27.jpg)
Contact Information
Brandon KoppResearch Psychologist
Office of Survey Methods Researchwww.bls.gov/osmr