kissmetrics summer 2012 state of ab testing
DESCRIPTION
3. We’ll cover...1 State of A/B Testing2 Q&A3 Demo 4. Let’s start with the data! 5. WHO AREWe? 6. Who are we? 176 in total 60 45 30 15 0 Ecommerce SaaS Agency/Cons Publishing Mobile Other 7. What’s the size of our teams? 90 67.5 45 22.5 0 1-5 6-10 11-25 26-60 51-200 200+ 8. What testing tools does everyone use? 90 67.5 45 22.5 0 Visual Website GA Content Optimizely None Other Optimizer Experiments 9. How many A/B tests did we run last month? 150112.5 75 37.5 0 0 1-4 5-9 10-19 20+ 10. It’s easy to get ahead of our competitors 11. How do we generate ideas for testing? 60 45 30 15 0 Test Experience Best Practices Research Needs Randomly 12. Step 1: Get Qualitative 13. Step 2: Make a prediction 14. Step 3: Confirm with an A/B test 15. How KISSmetrics generates testing ideas 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test 16. OURResults 17. Do we regularly increase conversions? 80 60 40 20 0 Always Sometimes Rarely Never 18. The revenue we gain from A/B testing 70 52.5 35 17.5 0 $1001- $10,001- Over None $1-1,000 No Idea 10,000 100,000 $100,000 19. 2 ways to make money with A/B testing 20. Option 1: Understand your customers first 1 Get qualitative insights 2 Predict how to improve 3 Confirm prediction with an A/B test 21. You’ll find the BIG wins much faster 22. Option 2: Get relentless 23. How pros like Facebook and Twitter do it 1 Build an entire growth team 2 Back-to-back testing 3 Look for 100s of small improvements 24. How do we A/B test with small amounts of data? 25. Do tests with customer interviews 1 Talk to people, give different options 2 Look for trends 3 Build and see if actions match 26. Do we track results over the long term? 130 97.5 65 32.5 0 Yes No 27. The pitfalls of not tracking long term results 28. Variation B performs the best in this funnel 29. Variation A gives the most revenue 30. LET’S TRACK THE RETURN ON A/B Testing 31. First, make sure you can track lifetime value You’ll need customer analytics that can track multiple purchases from the same person. 32. Where do we find customer analytics? 33. Second, connect A/B tools to customer analytics OR 34. What about Google Analytics? Connects tests to goals, not ecommerce transactions.TRANSCRIPT
Lars LofgrenMarketing Analyst - August 2012 [email protected] - Confidential - Do not distribute
SUMMER 2012 STATE OF A/B TESTING
@larslofgren #KISSwebinar
We’ll cover...
State of A/B Testing1
Q&A2
Demo3
Let’s start with the data!
WHO AREWe?
Who are we? 176 in total
0
15
30
45
60
Ecommerce SaaS Agency/Cons Publishing Mobile Other
What’s the size of our teams?
0
22.5
45
67.5
90
1-5 6-10 11-25 26-60 51-200 200+
What testing tools does everyone use?
0
22.5
45
67.5
90
Visual WebsiteOptimizer
GA ContentExperiments
Optimizely None Other
How many A/B tests did we run last month?
0
37.5
75
112.5
150
0 1-4 5-9 10-19 20+
It’s easy to get ahead of our competitors
How do we generate ideas for testing?
0
15
30
45
60
TestRandomly Experience Best Practices Research Needs
Step 1: Get Qualitative
Step 2: Make a prediction
Step 3: Confirm with an A/B test
How KISSmetrics generates testing ideas
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
OURResults
Do we regularly increase conversions?
0
20
40
60
80
Always Sometimes Rarely Never
The revenue we gain from A/B testing
0
17.5
35
52.5
70
None $1-1,000$1001-10,000
$10,001-100,000
No IdeaOver
$100,000
2 ways to make money with A/B testing
Option 1: Understand your customers first
Get qualitative insights1
Predict how to improve2
Confirm prediction with an A/B test3
You’ll find the BIG wins much faster
Option 2: Get relentless
How pros like Facebook and Twitter do it
Build an entire growth team1
Back-to-back testing2
Look for 100s of small improvements3
How do we A/B test with small amounts of data?
Do tests with customer interviews
Talk to people, give different options1
Look for trends2
Build and see if actions match3
Do we track results over the long term?
0
32.5
65
97.5
130
Yes No
The pitfalls of not tracking long term results
Variation B performs the best in this funnel
Variation A gives the most revenue
LET’S TRACK THE RETURN ONA/B Testing
You’ll need customer analytics that can track multiple purchases from the
same person.
First, make sure you can track lifetime value
Where do we find customer analytics?
OR
Second, connect A/B tools to customer analytics
What about Google Analytics?
Connects tests to goals, not ecommerce transactions.
