"so you think you are a data driven pm" - moriya kassis and edik mitelman @producttank...
TRANSCRIPT
So you think you are a
Data-Driven PM…?
This is an interactive session. Your cellphones are actually welcome here.And here as in life - you should act both fast and smart.
Let’s warm you up. Q #1. Here it
comes:
What is a successful experiment?
A successful experiment leads to
A follow up experiments
Changes to the Product
Experiment building algorithm
An issue or pain-point for the user Problem
hypotheses
Solution experiments
Users Personas
Don’t know thatAren't able toAre distracted byAre forced toNeed to
[Fill in the blank]
..
....
Solution Hypothesis
Q #2. Here it
comes:
Solution Hypothesis Structure
For example:1. If the call-to-action button is red then the number of
people registering will go up because the users will see the button better
2. If we change the copy explaining the value of registering then the number of people registering will go up because they will understand the value they get
3. If we remove all but one call-to-action on the page then the number of people registering will go up because they are not distracted by multiple call-to-actions
Solution Hypotheses create a tunnel vision!That’s why we have Problem Hypotheses!
Shoppers want to see an activity feed of product purchases on their homepage
solution
Why?Because…shoppers aren't buying enough products
business need
Why?Because customers don't think the product recommendations are authentic.
customer need
How to chose a hypothesis?
1. Pick the riskiest!2. High risk -> High Reward3. If you are right on all of them, what will bring
the biggest impact?4. Don’t waste resources on proving what you
already know
Hypothesis Prioritization
Q #3. Here it
comes:
Experiment design - variablesMust be controlled!
Must be measured!
ExperimentDesign
Q #5. Here it
comes:
Experiment design - A/B testing⊸ Small number of variables⊸ Measures one KPI⊸ Small traffic is usually enough⊸ No insight on variables
interaction
Experiment design - Multivariate testing⊸ A lot of variables⊸ Measures different KPIs⊸ Shows full interaction between
variables⊸ Requires a lot of traffic⊸ Longer to develop and analyze
Q #6. Here it
comes:
Are you ready for The challenge?
Experiment design - Testing
Complements!Not opposites.
Experiment design - Variants
Small changes can optimize, but won’t make a leap
Go widely different in the problem space
Q #7. Here it
comes:
Last but not least
Segmentation of the market [Country, Time of day, Day of week, Visitor type
(new versus returning), Search keywords]Segmentation allows you to understand and optimise for different users. There is no point in getting a 30% conversion rate in the wrong market which masks the fact you’ve only got a 5% conversion rate in your target market.
The next step is to ask yourself two questions:● What do these results mean for development
prioritisation? And● Why did I get these results?
Thanks!Any questions?You can find us at [email protected] [email protected]