designing effective experiments by grace ng
DESCRIPTION
Every product decision is an experiment. If we can reframe our thinking from launching campaigns and building features to challenging assumptions and running experiments, then we can increase our productivity and get better results with less effort. Grace will share a structured process for designing effective experiments, and talk about some of the common pitfalls and challenges teams face when running experiments in their companies. Using examples from her work helping product teams within large companies, Grace will reveal a few ways to overcome these challenges.TRANSCRIPT
Designing Effective Experiments
Grace Ng | @uxceo
Test your business idea in just 5 minutes
Photo Q&A
Textual search is limiting. How can I search a visual query?
@uxceo
What is this?
@uxceo
Minimum Viable Product! An experiment that uses the least amount of effort
to get the most amount of customer feedback
@uxceo
Minimum Viable Product! An experiment that uses the least amount of effort to test the
Riskiest Assumption against real customer feedback
@uxceo
Do people even have visual questions to post? "Riskiest Assumption: Have photo Qs to post.
Ask Question
Upload Photo
@uxceo
What kind of bird is this? (Is it even a bird??)
@uxceo
Will there be people who are able to answer the photo questions that are posted? ""Riskiest Assumption: Someone can answer within given time.
@uxceo
What kind of bird is this? (Is it even a bird??)
@uxceo
Will this scale? Can this be replicated?
@uxceo
@uxceo
@uxceo
@uxceo
LEARN MEASURE
BUILD
@uxceo
I Don’t Care...
@uxceo
@uxceo
$2M. 2 Years.
$30M
Mistakes
Effective Interviews
1. Tested the wrong assumptions
Mistakes
@uxceo
Effective Interviews
"
2. Asked the wrong questions
Mistakes
@uxceo
Effective Interviews
"
"
3. Didn’t define success
Mistakes
@uxceo
Effective Interviews"
"
"
4. Don’t need to build to test
Mistakes
@uxceo
@uxceo
Case Study@uxceo
Founders UX Help@uxceo
Boot- strapped startup
founders
@uxceo
Boot- strapped startup
founders
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
@uxceo
Boot- strapped startup
founders
No reliable source
Getting UX
feedback
@uxceo
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
@uxceo
1. Cost• Project Cost• Opportunity Cost"
2. Existing metrics• Business KPI• Specific user behavior
How to MEASURE success?
@uxceo [email protected]
How to measure success?
@uxceo
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
@uxceo
Exploration Interviews@uxceo
Effective Interviews
1. I’m doing some research around...
2. Have you ever [experienced problem]?
3. Tell me about that time, what happened?
4. What did you do to resolve it?
5. How satisfied were you with the outcome?
6. If unsatisfied, how else have you tried to solve it?
Effective Interviews
@uxceo
Collect Data
• Level of pain/frustration "
• Frequency of encounter with pain "
• Existing behavior to solve problem "
• Time/money spent searching for a solution
Collect Data
@uxceo
No reliable source
Interview: 6/10
9/12 Persevere
Friends. Classes. Pay for UX help
@uxceo
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
@uxceo
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Pay for virtual UX feedback
Pitch: 8/10
@uxceo
No reliable source
Interview: 6/10
9/12 Persevere
Friends. Classes. Pay for UX help
@uxceo
Pay for virtual UX feedback
Pitch: 8/10
10/10 Test
2nd side of market
Don’t mind virtual.
It’s cheap.
@uxceo
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Pay for virtual UX feedback
Pitch: 8/10
@uxceo
Getting clients for quick UX
consulting
UX Designers
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Pay for virtual UX feedback
Pitch: 8/10
@uxceo
Getting clients for quick UX
consulting
UX Designers
Virtual UX
feedback
Will consult virtually
Pitch: 8/10
Seriously?@uxceo
No reliable source
Interview: 6/10
9/12 Persevere
Friends. Classes. Pay for UX help
@uxceo
Pay for virtual UX feedback
Pitch: 8/10
10/10 Test
2nd side of market
Don’t mind virtual.
It’s cheap.
@uxceo
Will consult virtually
Pitch: 8/10
0/10 Pivot!
No reliable source
Interview: 6/10
9/12 Persevere
Friends. Classes. Pay for UX help
@uxceo
Pay for virtual UX feedback
Pitch: 8/10
10/10 Test
2nd side of market
Don’t mind virtual.
It’s cheap.
@uxceo
Will consult virtually
Pitch: 8/10
0/10 Pivot!
No! 5 yrs exp. Established
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Pay for virtual UX feedback
Pitch: 8/10
@uxceo
Getting clients for quick UX
consulting
UX Designers
Virtual UX
feedback
Will consult virtually
Pitch: 8/10
Transition to UX
designers
Getting clients for quick UX
consulting
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Pay for virtual UX feedback
Pitch: 8/10
@uxceo
Getting clients for quick UX
consulting
UX Designers
Virtual UX
feedback
Will consult virtually
Pitch: 8/10
Transition to UX
designers
Getting clients for quick UX
consulting
Virtual UX
feedback
Give quick online
feedback
Pitch: 60%
@uxceo @nycuxpa #ngnycuxpa
Google Ads@uxceo
No Backend@uxceo
@uxceo [email protected]
The 5 Minute Technique
Quick MVP@uxceo [email protected]@uxceo
Quick MVP@uxceo
3 Data Points@uxceo @nycuxpa #ngnycuxpa@uxceo
3 Data Points@uxceo
No reliable source
Interview: 6/10
9/12 Persevere
Friends. Classes. Pay for UX help
@uxceo
Pay for virtual UX feedback
Pitch: 8/10
10/10 Test
2nd side of market
Don’t mind virtual.
It’s cheap.
@uxceo
Will consult virtually
Pitch: 8/10
0/10 Pivot!
No! 5 yrs exp. Established
Give quick online
feedback
Pitch: 60%
37/184 emails
Persevere
Started referring friends
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Pay for virtual UX feedback
Pitch: 8/10
@uxceo
Getting clients for quick UX
consulting
UX Designers
Virtual UX
feedback
Will consult virtually
Pitch: 8/10
Transition to UX
designers
Getting clients for quick UX
consulting
Virtual UX
feedback
Give quick online
feedback
Pitch: 60%
Boot- strapped startup
founders
No reliable source
Interview: 6/10
Getting UX
feedback
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Pay for virtual UX feedback
Pitch: 8/10
@uxceo
Getting clients for quick UX
consulting
UX Designers
Virtual UX
feedback
Will consult virtually
Pitch: 8/10
Transition to UX
designers
Getting clients for quick UX
consulting
Virtual UX
feedback
Give quick online
feedback
Pitch: 60%
Online platform is sufficient
Concierge: 60%
satisfied
Boot- strapped startup
founders
Getting UX
feedback
Virtual UX
feedback
Concierge
Customer Submits Request UX Designer gets email and submits feedback
@uxceo [email protected]
Concierge@uxceo
Manually
Customer Submits Request UX Designer gets email and submits feedback
@uxceo [email protected]
Manually@uxceo
No reliable source
Interview: 6/10
9/12 Persevere
Friends. Classes. Pay for UX help
@uxceo
Pay for virtual UX feedback
Pitch: 8/10
10/10 Test
2nd side of market
Don’t mind virtual.
It’s cheap.
@uxceo
Will consult virtually
Pitch: 8/10
0/10 Pivot!
No! 5 yrs exp. Established
Give quick online
feedback
Pitch: 60%
37/184 emails
Persevere
Started referring friends
Online platform is sufficient
Concierge: 60%
satisfied
20% Iterate
Wanted more
function- ality
High PainLow Pain
Low Frequency
High Frequency
Focus@uxceo
Grace Ng | @uxceo
Test your business idea in just 5 minutes