using automated testing tools to empower your user research
TRANSCRIPT
4149 Pennsylvania, Suite 202Kansas City, MO 64111
Automated User Testing
Presented by
Chrys SullivanFounder and CEO, Useagility
An educational program sponsored and supported by User Experience Professionals
Association.
Using Automated Testing Tools to Empower Your
User Research
Join now at www.uxpa.org for member-only pricing
UXPA Seattle Conference 2016May 31 – June 3uxpa2016.org
Rapidly Test Usability & Measure UXUserZoom is the All-in-One UX Research Platform for the Data-Driven
Enterprise
• Chat box is available if you have any questions
• Look out for a quick poll question
• There will be time for Q&A at the end
• We will be recording the webinar for future viewing
• All attendees will receive a copy of the slides/recording
• Continue the discussion using #UZWebinar
Quick Housekeeping
Let’s understand each other’s background, roles and experience levelsPolls
About Useagility
6
Twitter: @Useagility
www.Useagility.com
Today we’ll learn:
7
How unmoderated testing works
and how you can use it in your organization.
01
Guidelines for selecting the right testing methodology (moderated vs.
unmoderated) based on your project needs and goals.
02
Steps for setting up successful unmoderated
studies.
03
Advice for recruitment, screening and use of online
panels.
05
Tips for avoiding the most common pitfalls in
automated UX testing.
04
Using automated testing with agile design and
development sprints to accommodate tight timelines and satisfy usability needs.
06
Evolution of Usability Testing Technology
8
Lab-based testing - Moderated testing- In lab setting
Remote, facilitated testing- Moderated testing - Human facilitator and
participant interact via web-conference
Automated, un-facilitated testing- Unmoderated studies conducted online via web-based applications • No human facilitator• Tasks are pre-set by researcher,
delivered through testing interface
Benefits of Unmoderated
9
Easier recruitment and scheduling No worries about no-shows &
replacements
Automated incentive payments
Simultaneously test larger number of participants (Quantitative)
Less expensive per-study costs than moderated testing
Get results more quickly
Built-in metrics (time on task, heat maps, rating scale, SUS, etc)
Our Favorite Ways to Use Unmoderated User Testing
10
Help Teams “Fail Fast” Testing design wireframes and
prototypes (Axure, Invision) Quickly weed out approaches that
don’t work
Resolve internal disagreement Let users decide which design works
best
Support Expert Review / Heuristics Evaluations
Validate suspected usability issues
Lean UX Research with Agile Quickly test prototypes as part of agile
development cycles
Prioritize backlog Measure usability impact for dev prioritization
A/B testing
Click-through / conversion testing
Limitations & Pitfalls of Unmoderated
11
User panels and “professional testers”
Stakeholders can’t observe live during the study
Can’t revise tasks once the study has been launched
Can’t adjust approach as sessions progress to optimize depth of learning
Lacks ability to probe unexpected insights
No real-time support for users who get significantly off-track
IT’S NOT MODERATED
Selecting right methodology
12
Quantitative or Qualitative?
Moderated or Unmoderated?
Selecting the right methodology has the greatest impact on quality and effectiveness of the study.
Define Goals
13
Hard to get business stakeholders to define their goals. Ask them questions:
• What is the primary purpose of the study?
• What do you need to know coming out of this study?
• What will you do with the study findings?
Define additional details
14
After goals are defined, understand other key components:
• What is the scope of study? (Big, small, how focused)
• What are we testing with? (Prototype, wireframes, live site)
• Who are the target users? • What tasks and scenarios will help
achieve study goals?
Which methodology is best?
15
Will you need moderated follow-up and probing questions to be successful?
Are you at early concept and UX strategy phase?
Do you need to go deep on attitudinal findings, mental model and user preferences?
Is it the first time you have tested a design? First time testing with a new user type?
Will it take longer than 20 – 30 minutes to complete tasks?
If yes = Moderated
Which methodology is best?
16
Is the scope narrowly focused?
Can testing tasks be accomplished in 30 min or less?
Is testing of task success and usability a primary objective vs. diving deep on user attitudes and preferences?
Is there minimal reliance on attitudinal feedback that requires follow up probing?
Is this follow-on testing after moderated?
If Yes = Unmoderated
Creating successful test plans for unmoderated, automated usability testing has unique challenges.
Key Steps - Unmoderated Automated Usability Testing
PARTIPANTS TASKS FOLLOW-UP QUESTINS ANALYSIS
Get right participants Design study and tasks effectively
Use follow-ups appropriately
Analyze finings accurately
Participants
18
Many ways to recruit for unmoderated: 011. Email existing user base
2. Social media
3. Site intercept
4. Panel provide by UX testing vendor
*In all cases, screening for representative sample is very important*
PITFALL #1: Panels
19
PITFALL#1: “Professional Testers” and “Imposters”
SOLUTION: Over-screen
0101
1. Use 2x the screener questions as normal
2. As first task in study, ask a detailed question to verify user is who they say
3. Over-recruit so you can throw out participants you feel may not be valid
Writing Tasks
20
Crafting study plan & writing tasks 01
1. Remember, you won’t be there to help or clarify
2. Avoid leading verbiage that may cause bias in task success
Instead of: View your current grades in Algebra
Use: Imagine you want to know your current grade in Algebra
3. Ensure prototypes enable users to complete tasks
4. Anticipate how to proceed if they fail a task
5. Determine what metrics or data will be meaningful given goals and context
Use follow-up questions
21
011. Use follow-up questions to check for understanding
Example: If the task was to locate their weighted-average grade in Algebra class, ask what their “weighted-average” means.
2. Use follow-up questions to probe for deeper insights on why a user behaved as they did
Example: If the task required multiple steps in a process, ask “were there parts of that process you were unsure about or found confusing, tell me more about why you felt that way.”
PITFALL #2 – Tasks skipped, misunderstood, leading
22
PITFALL #2 – skipped, misunderstood and leading tasks
SOLUTION – Conduct pilot session
0102
1. Conduct one session as pilot
2. Watch results
3. Adjust tasks and flow, add follow up questions to understand “why” if necessary
Analysis
23
Analyzing unmoderated testing 01
1. Watch all sessions in full • (run at 1.5 speed)
2. Use integrated tools to tag video clips and make notes
Analysis - Getting Lazy!
24
PITFALL #3 – Relying too heavily on metrics
SOLUTION: Use data appropriately and in context
0103
1. Question self-reported task success rates
2. Balance stated “preference” for design with analysis of user behavior
3. Give thought to which metrics are applicable and relevant to your study based on context
• Time-on-task, heat maps, conversion, click-path
25
Wrapping It Up
Now You Know
26
What unmoderated testing is and how to use it
01
Importance of selecting right methodology: moderated or unmoderated
02
Key steps to setting up and running successful unmoderated testing
03
Common pitfalls in unmoderated testing and ways to avoid them
04
Thank You!
27
Thank you for joining us today! Useagility contact info:
www.Useagility.com
Twitter: @Useagility
Need more info?Get answers about UserZoom. We are here to help. Talk with one of our knowledgeable staff.
Contact us
UserZoom10 Almaden Boulevard, Suite 250San Jose, CA 95113
Toll free number: (866) 599 1550
www.linkedin.com/company/userzoom@UserZoomFollow us