a practical guide to measuring user experience
DESCRIPTION
Measuring the success (or failure) of an experience can be a daunting and confusing endeavor. In this presentation Richard shares 11 guidelines to help you quantitatively measure your user experience. Richard provides techniques and tips for each phase and illustrates their use with real examples from his team’s work at Vanguard. To conclude, he describes some of the cultural and change management challenges involved when an organization uses data to inform design decisions.TRANSCRIPT
1
A Practical Guide to Measuring User ExperienceRichard Dalton, @mauvyrusset, #measuringux
http://www.flickr.com/photos/kwl/5282931021
2
#1 To learn from failure, you need to know if you’re failing and why – measurement can tell you that
@mauvyrusset #measuringux
3
THERE IS NOTHING FUNNY ABOUT R.O.I.
4
5
#2 Evaluate your experience against something you care about – is it meeting it’s objectives?
@mauvyrusset #measuringux
6
7
#3 The objectives of your experience will likely differ from those of the person sitting next to you
@mauvyrusset #measuringux
Tasks + EmotionsUsers do things to try to meet their goals
… and the business wants users to do things so it can meet its goals
Goals
Projects
Are realized through
Are enabled & encouraged by
Are created & changed by
Capabilities
8
Users have goals
… and the business has goals
Projects create new capabilities
Projects change existing capabilities
Capabilities help users to do the tasks they want to do
… and encourage users to do the tasks the business wants them to do
TasksUsers do things to try to meet their goals
… and the business wants users to do things so it can meet its goals + Emotions
Users also have feelings about their goals & tasks
… and the business wants to encourage certain feelings
9
#4 Measure how well tasks are satisfied by capabilities, not projects – otherwise you have no baseline
@mauvyrusset #measuringux
User-driven tasks
Business-driven tasksCapabilities
Follow Vanguard’s investing principles
Learn why Vanguard is great
Bring assets to Vanguard
Use Vanguard's products & services
Self-provision on the web
Spread the word about Vanguard
Trust Vanguard
Find an investment company
Act on my investments
Help my heirs be successful at Vanguard
Make good investment decisions
Monitor my investments
Stay current on news and commentary
Deal with taxes
Help other people be successful investors
Web
Phone
Paper
Mobile devices
Radio/TV
90 tasks groupedinto 8 categories
45 tasks groupedinto 7 categories
635 capabilities and counting …
10
Find an investment company
11http://filmfanatic.org/reviews/wp-content/uploads/2008/01/anfscd-parrot.png
Item profile(web)
Ready to buy
Tell others
Research an item
Trust us
Close the sale
Cross sell
Spread the word
User-driven tasks
Business-driven tasksCapabilities
12
Compare the item to others like it
Get information on the item
Find out how much the item costs
Buy the itemFind out shipping costs and times
Find out how to pay
Tell a friend about the item
See if other people like the itemBuy the item
See related items
Believe that the site is safe and secure
Print details about the item
Save the item to look at later
See related items
Tell other people about the item
13
Compare the item to others like it
Get information on the item
Find out how much the item costs
Buy the item
Find out shipping costs and times
Find out how to pay
Tell a friend about the item
See if other people like the item
Buy the item
See related items
Believe that the site is safe and secure
Print details about the item
Save the item to look at later
See related items
Tell other people about the item
2
3
4
5
6
1
11
7
8
9
10
12
Emotional considerations
High-level design & content approach
Success criteria – “what is good?”
Shoppers are commonly fearful or unsure that they have chosen the best product for their needs and want to “comparison shop” the price and/or features of several products.
Show recently viewed items and provide access to a “compare to similar items” tool.
The ratio of users looking at recently viewed items via the “compare to similar items” tool vs. pogo-sticking back to the gallery page should be 20:1 or better.
14
15
Measure Success Criteria
Measures vs. Success Criteria
http://3.bp.blogspot.com/_iUXX3rF1Axo/TC0fGhmlV0I/AAAAAAAAAMw/atKJuTja0AM/s1600/butterfly+growth+chart2.jpghttp://www.clarklings.com/uploaded_images/IMG_1100-732092.JPG
16
S1.01 Get information on the item
S2.01 Buy the item
P1.01 Buy the item
S1.03 Compare the item to others like it
S1.02 Find out how much the item costs
S1.04 See if other people like the item
S2.03 Find out how to pay
S2.02 Find out shipping costs and times
S3.01 Tell a friend about the item
Research an item
Ready to buy
Tell others
P2.01 See related itemsP3.01 Believe that the site is safe and secure
P4.02 Tell other people about the item
OutcomeDriversCro
ss se
ll
Trust
us
Sprea
d the w
ord
Ishikawa (fishbone) diagramhttp://en.wikipedia.org/wiki/Ishikawa_diagram
17
#5 Measuring outcomes can tell you if a capability is failing. Measuring drivers can tell you why
@mauvyrusset #measuringux
18
A. General effectiveness in satisfying the task“I’ve done it” or “Now I understand”
1. Completion or conversion rates2. Did future user behavior change as a
result of this interaction?
B. Findability of a known item“I know where I’m going, don’t get in my way”
Speed to find the first task
C. Were the user expectations met?“Oh, that wasn’t what I wanted, let me go back …”
Link bounce rate
D. Client satisfaction“Huh, that information was useless”
1. “Was this useful/helpful” surveys2. “Loss of sale” surveys
E. Was enough information provided?“Don’t leave me wanting more info”
Usage of a second source of information
F. Are users travelling the paths we expected them to?“I can’t find the link, I’ll go back to the homepage first”
Relative usage of one path vs. another to identical destination
What’s being evaluated? Measure
19
Success Criteria
Enduring Temporary
Measure a single solution against a predefined criteria
Compare two or more solutions against one another using A/B testing
Use for on-going monitoring
Criteria set by past user behavior or future expectations.
Use for point-in-time improvement
Point-in-time “winners” identified by test results.
ExampleTask: Get information on the item
Loss of sale surveys should show that problems with item information account for less than 5% of lost sales
ExampleTask: Print details about the item
A/B test two versions of the printer friendly entry point (link vs. button)
20
#6 Ask; how would the user behave if we nailed the design? How would they behave if we screwed it up?
@mauvyrusset #measuringux
21
22
Sometimes people aren’t ready to listen
23
#7 Be open about the uses & limitations of data & involve people early – it helps gain buy-in
@mauvyrusset #measuringux
24
Beware good intentions
25
#8 Avoid misleading measures - temptation to use the data is too strong. Ask “what if the result is X?”
@mauvyrusset #measuringux
26
Ostridge
http://fc05.deviantart.net/fs36/f/2008/285/4/f/You_make_kitty_scared_by_GreenLabRat.jpg
The only thing we have to fear is fear itself
27
#9 Be unbiased. Don’t be afraid to measure things that might contradict your own opinion
@mauvyrusset #measuringux
28http://img395.imageshack.us/img395/3899/hal90001600en6.jpg
I’m sorry Dave. I’m afraid I can’t do that
29
#10 Don’t lose your perspective about how data fits into your decision-making process
@mauvyrusset #measuringux
30http://www.flickr.com/photos/tammra/279392432/
But we have over 635 capabilities!
31
#11 Start small. Pick a capability, identify objectives, define measures and watch what happens
@mauvyrusset #measuringux