quick cheap insightful: usability testing in the wild
DESCRIPTION
How to gather data to made design decisions on almost anywhere almost any time for very little money.TRANSCRIPT
The quick, the cheap, and the insightful
Usability testing in the wild
Dana ChisnellSan Francisco STC November 2008
1
What is a usability test?
2
3
http://www.sigchi.org/chi97/proceedings/overview/tst.htm
4
5
Quick, cheap, insightful
✤ Minimal steps to focus on
✤ Where the value comes from in doing usability tests
✤ Where it may be risky to go minimalist
✤ How to think about the trade-offs
✤ What’s essential
✤ What might be nice to have
6
Minimal steps
Classic, with everything
7
✤ Chapter 5. Develop a test plan
✤ Chapter 6. Choose a testing environment
✤ Chapter 7. Find and select participants
✤ Chapter 8. Prepare test materials
✤ Chapter 9. Conduct the sessions
✤ Chapter 10. Debrief with participants and observers
✤ Chapter 11. Analyze data and observations
✤ Chapter 12. Create findings and recommendations
8
Photo courtesy of Tom Tullis
9
Classic, with everything
Can take a lot of time: weeks or months.
What if you have to do something quick? and cheap?
10
You don’t have to do it by the book.
11
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations
12
What’s essential?
Develop a test plan
Choose a testing environment
Find and select participants
Prepare test materials
Conduct the sessions
Debrief with participants and observers
Analyze data and observations
Create findings and recommendations
13
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers
14
What’s essential?
Develop a test plan
Find participants
Conduct the sessions
Debrief with observers
15
and decide together
Sit next to someone. Watch them do stuff.
16
Story:
17
California.
18
1 day
19
a ballot and instructions
20
a ballot and instructions
21
random participants
22
no helpers
23
results due: today
24
What to do?
25
Improvise: Ground rules
26
✤ Say “yes.”
✤ Trust your team and your participants
✤ Don’t let the team or stakeholders block
✤ Work to the top of your collective intelligence
27
How we got value
✤ Used the ballots and instructions at hand
✤ Focused on one thing at a time
✤ Had participants generate and collect data
✤ Drafted observers and debriefed
28
More examples
29
Cafe
30
Company lobby
31
Trade show
http://www.flickr.com/photos/dans180/72408664/
32
Listen in on
customer service
33
Transit seat
mates
34
Remote testing
Builds a storyShows organizing strategies
Email trails
Shows deliberate returns to sites
Bookmarks & favorites
Gives a recent snapshot of activity
Browser history
35
Frames the conversations users are having
User forums & wikis
Proxies for buyers, end users
Sales reps
View on persistent, difficult problems
Customer service logs
36
37
Value
Supporting usable design
Each phase includes input from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and objectives
38
Supporting usable design
Each phase includes input from users
Multidisciplinary teams
Enlightened management
Willingness to learn as you go
Defined usability goals and objectives
39
Where’s the value in testing?
70% watching someone use the design
20% working with the team to prepare to test
8% discussing what happened
40
41
Getting value in the wild
✤ Use what’s at hand
✤ Narrow the scope of the test
✤ Focus on
✤ what can make the most difference to the most users
✤ what can be implemented easily with the resources available
When is testing in the wild not valuable?
42
✤ If you need
✤ summative data
✤ benchmarks
✤ answers to hard problems
43
Risks
Risks of testing in the wild
✤ Participant sample may be too small, biased
✤ Inconsistent approach may net inconsistent data
44
What do you lose?
✤ Quantitative data
✤ Rigor
✤ Relatively unbiased sample, maybe
45
http://drb.lifestreamcenter.net/Lessons/process_maps/
46
Trade offs
What do you need?
✤ To inform a design:
47
Testing in the wild Classic usability testingQualitative data Quantitative dataOpportunity PlanningFits the schedule Don’t know how to fit UT into
the scheduleJust in time After the fact$ $$$$Something Maybe nothing
✤ Qualitative data
✤ Opportunity
✤ Fitting into a schedule
✤ Timeliness
48
Where’s the ROI?
49
So far.You don’t have to do it by the book
Value of usability testing
+ observing users
+ working with the team
Tradeoff: having some data over having no data
50
Coming up.What’s the bare minimum
Steps for testing in the wild
What to add if you have resources
51
Essentials
What you need
✤ Someone who will try the design
✤ Somewhere to test
✤ Something to study
52
Something(Activity)
Some place(Context)
Somebody(Human)
Follow these steps:
53
1. Plan, minimally
2. Get the team on board
3. Design the test, minimally
4. Recruit participants
5. Conduct sessions
6. Debrief and decide
Plan, minimally✤ What
✤ Why
✤ Who
✤ When
✤ Where
54
Plan, minimally: Example
✤ What: near-final design
✤ Why: inform user training and support
✤ Who: inexperienced customers
✤ When: the end of the week
✤ Where: trade show at user group meeting
55
Minimalist plan
56
Goals and objectives
Participant characteristics
Description of method
List of tasks
Find out whether information about admission is easy to find and use
People who have college-bound kids
Sit-by parents attending a high school basketball game, each trying three university sites with the same scavenger hunt task
Find out how and when applications are dueDetermine whether there are fees for applyingLearn when acceptances will be sent
Get the team on board✤ Visualize the desired user
experience
✤ Share the intellectual property of observing users
57
Design the test, minimally✤ Why are you testing?
✤ What questions are you trying to answer?
✤ What constrains the design?
✤ What are you going to do with what you find out?
58
Recruit participants✤ Convenience sample
✤ Requirements, not demographics
59
Sources of participants
60
✤ Staff not on the design project
✤ Friends and family
✤ Personal, professional networks
✤ Online social networks
✤ Community organizations
✤ Online classifieds
✤ Association, society, user group, union contacts
✤ Temp agencies
Conduct sessions✤ Rehearse (P1)
✤ Interview-based tasks (or based on previous field work)
✤ Explore in short, focused sessions
✤ Iterate test design and product design
61
Session outline
62
Greet the participant
Explain the study, your role, and their role
Interview (maybe)
Do tasks from interview
Debrief with participant
Debrief with observers
Moderating, not training
63
✤ Impartial, unbiased observing
✤ No teaching!
✤ Listen and watch
✤ Ask open-ended questions: Why? How? What?
Debrief and decide✤ Write up issues on sticky notes
and sort them into priority lists
✤ Ask for top 10 items -- base on data and observations, not opinion
✤ Take a vote for priorities
64
65
Nice to have
Add these ingredients
66
✤ Screened, scheduled participants
✤ Official paperwork
✤ Recordings
✤ More observers
Participants
67
What Why
appropriate experience
scheduled ahead of time
greater confidence in data
easier to get data
Paperwork
68
What Why
script
consent forms
recording waivers
pre-test questionnaire
post-test questionnaire
ensure consistent instructions and moderating
official acknowledgment for taking part
permission for recordings
experience, knowledge, value
feedback on tasks, UI
Recordings
69
What Why
audio
video or photos
Morae, Camtasia, other
transcripts, verbal protocol analysis
stories, double-check, highlights
digital data, automated collection
More observers
✤ Better-informed design recommendations
70
What Why
developers
management, execs
technology boundaries
business priorities
71
Nutshell
http://www.flickr.com/photos/stoan/145497333/
Take away:
72
✤ Value: observing, planning
✤ Plan: 4 steps
✤ Recruit: behavior
✤ Moderating: listen, observe
73
Quick.
You can compensate for some shortcomings, and just test more
Cheap.
Low risk, little money
Insightful.
Value comes from getting as close as possible to what real people are doing with your design
Where to learn more
74
Dana’s blog: http://usabilitytestinghowto.blogspot.com/
Download templates, examples, and links to other resources from www.wiley.com/go/usabilitytesting
