introducing: the lab notebook - a tool for managing a lean ux process
TRANSCRIPT
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Introducing: The Lab NotebookA hypothesis-driven approach for
delivering value faster
#UXLabNotebookJulie Casanave @jcasanaveKerry Holeman @uxkerry
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
What’s this all about?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Project completion can easily be mistaken for project success.
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
A successful launch delivers measurable customer value. If users don’t adopt the solutionwe need to evaluate and make a decision.
What was designed & delivered
How it’s used
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Science!
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
What’s the most important unanswered question?
What’s the smallest thing we can make to answer that question?
-Steve Mulder, Senior Director of Audience Insights, NPR
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Create a Lab Report
Createthe
experiment
Decide what to do
next
Definethe question
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Create a Lab Report
Createthe
experiment
How will we learn it?
Decide what to do
next
What do we do with what we
learned?
Definethe question
What do we want to learn?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the QuestionWhat do we want to learn?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Problem Statement
WHAT ISSUE NEEDS SOLVING?
“I am [customer] trying to [accomplish goal].But [insert problem] because [insert cause].
This makes me feel [emotion].
The consequence for [my team/company] is [adverse business effect]”
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
I am a 60-year old trying to take a retirement withdrawal online, because I need to cover the cost of a
new car.
But I don’t feel confident doing it online because I don’t understand if I’m subject to penalties or tax
consequences.
This makes me feel anxious.
The consequence for Fidelity is costly customer service calls and poor customer perception.
How might we…
“I am [customer] trying to [accomplish goal].
But [insert problem] because [insert cause].
This makes me feel [emotion].
The consequence for [my team/company] is …
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise
WHAT ISSUE NEEDS SOLVING?
“I am [customer] trying to [accomplish goal].But [insert problem] because [insert cause].
This makes me feel [emotion].
The consequence for [my team/company] is [adverse business effect]”
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the ExperimentHow will we learn?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
How the issue will be addressed
What we’ll build to address it
How we’ll know it works
HYPOTHESIS(solution strategy)
TESTstrategy
MEASUREMENTstrategy
+ +
“[This change] will enable [who] to achieve [what],
leading to [value].”
“The simplest way to test this is to [test
strategy].”
“We will know we’ve succeeded when we
observe [user behavior/sentiment].
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
HYPOTHESIS(solution strategy)
TESTstrategy
MEASUREMENTstrategy
+ +
A Turbo-Tax-like withdrawal process, which guides people through a decision path, will help customers withdraw retirement funds with a higher degree of confidence, leading to fewer calls to the call center.
[This change] will enable [who] to achieve [what], leading to [value].”
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
HYPOTHESIS(solution strategy)
TESTstrategy
MEASUREMENTstrategy
+ +
The simplest way to test this is with a moderated usability session + paper prototype which takes the customer down one sample decision-making path to completion.
The simplest way to test this is to [test strategy].
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Define the Experiment
HYPOTHESIS(solution strategy)
TESTstrategy
MEASUREMENTstrategy
+ +
We will know we’ve succeeded when we observe customers:1. Successfully complete the process without indicating
they’d abandon.2. Indicate that their level of confidence about
completing the transaction is higher than it is today.
We will know we’ve succeeded when we observe [user behavior/sentiment].
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise: Hypothesis
HYPOTHESIS(solution strategy)
TESTstrategy
MEASUREMENTstrategy
+ +
“[This change] will enable [who] to achieve [what], leading to [value].”
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide on a Test StrategySE
NTI
MEN
T(W
hat
peop
le s
ay) BEH
AVIOR
(What people do)
QUANTITATIVE(Reveals the
“what”)
QUALITATIVE(Reveals the “why”)
Surveys
Moderated Usability Study
A/B Testing
Self/Un-ModeratedUsability Study Opinion Lab Verbatims
Focus Groups Interview
Call Center Notes
Social Media Activity
Web Analytics
Emotional Reaction Study
Card Sort Treejack Study First Click Study
Eye-Tracking Study
Pre-release Post-release
PRO TIPBe biased towards behavioral methods
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Think creatively about what type of experiment will give the results you need.Deployed code is only one kind of experiment.(and often the most expensive kind)
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise: Test Strategy
HYPOTHESIS(solution strategy)
TESTstrategy
MEASUREMENTstrategy
+ +
“The simplest way to test this is to [test strategy].”
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
Measures of attitudes, often collected via survey• Emotional reaction• Perceived level of effort• Sentiment rating
• Level of confidence• Volume of opinion lab comments
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
Level of user involvement•Bounce rate •Channel switching
• Page visits • Searches •Click-throughs
• Eye-tracking• Visitors
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
Gaining new users of a product or feature•Click rates•Click-throughs
•New accounts
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
The rate at which existing users are returning•NA for our program
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
Efficiency, effectiveness, and error rate
•Completion speed• Error rate
•Abandonment•Completion rate
• Pathing•Re-searches
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
• Emotional reaction• Perceived level of effort• Sentiment rating
• Bounce rate • Channel
switching
• Click rates• Click-throughs
• Not applicable
• Completion speed• Error rate
• Level of confidence• Volume of opinion lab comments
• Page visits • Searches • Click-throughs
• Abandonment• Completion rate
• Eye-tracking• Visitors
• New accounts
• Pathing• Re-searches
What metric matters most?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
• Emotional reaction• Perceived level of effort• Sentiment rating
• Bounce rate • Channel
switching
• Click rates• Click-throughs
• Not applicable
• Completion speed• Error rate
• Level of confidence• Volume of opinion lab comments
• Page visits • Searches • Click-throughs
• Abandonment• Completion rate
• Eye-tracking• Visitors
• New accounts
• Pathing• Re-searches
Does this experience give you ahigher or lower level of
confidence compared to today’s process?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide How to MeasureHAPPINESS
ENGAGEMENT
ADOPTION
RETENTION
TASK SUCCESS Source: Google Ventures, How to Choose the Right UX Metrics for Your Product, by Kerry Rodden
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
• ??
WHICH ONE MATTERS MOST?
WHAT’S YOUR SET OF METRICS?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Exercise: Measurement Strategy
HYPOTHESIS(solution strategy)
TESTstrategy
MEASUREMENTstrategy
+ +
“We will know we’ve succeeded when we observe [user behavior/sentiment].
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Decide what to do nextWhat do we do with what we learned?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
What We Learned Major findings, unanticipated discoveries
Pivot? Or Persevere? Or Stop?At what point was the conclusion reached?
Time Lapsed Since Launch
Conclusion
What will happen next?Next steps
DD Month YYYYLaunch DateDocumenting what was learned
Outcomes How did experiment do against HEART metrics?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
A disproven hypothesis is evidence.
It is not failure.“I have not failed.
I've just found 10,000 ways that won't work.”-Thomas A. Edison
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
And when that experiment is done?
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
LATHER, RINSE, REPEAT? !
? !
? !
? !
? !
? !
DESIGN RUN ITERATE
LAB NOTEBOOK
LAB REPORT
LAB REPORT LAB REPORT
LAB REPORT
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
The “grain of salt” slide
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
MITIGATE RISK• Deliver value faster• Reduce wasted time, effort,
money
FOCUSED EXPERIMENTS• Clear learning agenda• Simple test strategy• Well defined metrics
© 2016 Proprietary material; do not use without permission of author | #UXLabNotebook
Thank You.
Julie Casanave | @jcasanaveKerry Holeman | @UXKerry