measure or die
DESCRIPTION
How Yammer uses data to make product decisionsTRANSCRIPT
![Page 1: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/1.jpg)
![Page 2: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/2.jpg)
Measure or Die
Neil McCarthyKen Pascual
![Page 3: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/3.jpg)
Justifying featuresExecution decisionsDetermine success
![Page 4: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/4.jpg)
HOW DATA HELPS US PRIORITIZEStep 1: Justifying Ideas
![Page 5: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/5.jpg)
![Page 6: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/6.jpg)
Previous signup flow experiment
![Page 7: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/7.jpg)
![Page 8: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/8.jpg)
![Page 9: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/9.jpg)
Clues from the past
![Page 10: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/10.jpg)
Brainstorming
![Page 11: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/11.jpg)
Idea!
![Page 12: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/12.jpg)
Does this align with our goals?
• Vision: Yammer is a primary communication tool that users use to get work done.
• Goal: Grow the number of users that use the product.
• Problem: We lose too many potential users in our signup flow.
![Page 13: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/13.jpg)
How important is this?
![Page 14: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/14.jpg)
![Page 15: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/15.jpg)
big drop!
![Page 16: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/16.jpg)
And HALF of our invites come from the signup flow!
![Page 17: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/17.jpg)
Conclusion: This seems important.
![Page 18: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/18.jpg)
Why justify?
![Page 19: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/19.jpg)
HOW DATA HELPS US DETERMINE THE SUCCESS OF THE PROJECT
Step 2: Determining Success
![Page 20: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/20.jpg)
What’s your hypothesis?
![Page 21: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/21.jpg)
What we came up with…
If we move the email confirmation step to the end of the signup flow,then more users will make it through the signup flow and more users will be retained.
![Page 22: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/22.jpg)
Secondary objective: Kill the Desktop app
![Page 23: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/23.jpg)
We need another hypothesis
![Page 24: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/24.jpg)
“If we remove the prompt to install the desktop app, then we will lose an acceptable amount of engagement”.
![Page 25: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/25.jpg)
A/B Test Results: Killing the Desktop app prompt
![Page 26: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/26.jpg)
A/B Test Results: Killing the Desktop app prompt
![Page 27: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/27.jpg)
Interpretations
• Hypothesis disproved (in a good way).• Users find a way to post.
![Page 28: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/28.jpg)
Results!
![Page 29: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/29.jpg)
A/B Test Results: Moving the email activation step
![Page 30: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/30.jpg)
A/B Test Results: Moving the email activation step
![Page 31: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/31.jpg)
Interpretations
• Users saw the steps in a different order, and dropped off in similar percentages, BUT– Less users saw the prompt to “check email”
because it was at the end.– Thus, less users activated via email.– Thus, less users had the opportunity to engage.
![Page 32: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/32.jpg)
Why we chose this example
![Page 33: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/33.jpg)
How do you do this?
• Gotta log data!• Choose your own metrics• Data driven vs. data informed• Build your processes around data• How much PMs should know about statistics.
![Page 34: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/34.jpg)
Q & A
![Page 35: Measure or Die](https://reader034.vdocuments.net/reader034/viewer/2022052315/540b711a8d7f72f36a8b4760/html5/thumbnails/35.jpg)
Thanks for coming!
• Neil McCarthy, @the_neil• Ken Pascual, @vorped• Adam Zadikoff (original PM for this project),
@zadikafka
• Test results from the first signup flow test: https://medium.com/yammer-product/13eeb6435a73
• These slides on Slideshare coming soon!