unleash the power of a/b testing - sitecore summit
TRANSCRIPT
Unleash the power of AB- testing
Digital marketing Summit 2016
The Optimisation
Wheel
This is where most start“The most innovative
company in our industry”
“Amazing creativity is the core of everything
we do”
What’s missing?
We are data driven - Horay!
Add experiments! Done or?
What’s missing?
8
9
10
This way or that
A / B testing direct visitors at random to different versions of a page or a
component.
The experiment is performed on your regular site on your real visitors and they
are not aware that they are in an experiment.
11
The classic A/B -test
A B
12
We have a winner!
You measure the outcome against the experiment goals (for example sales) for the different variations.
When the test has reached statistical significance a winner can be
declared.
13
Överfralla eller Underfralla?
Are you a scientist or an accountant?
14
“Where the rubber hits the road”
A/B Testning is the point when our
optimization wheel hits the cruel reality.
Two things happen - both are important
Forward movement
Gravel spray
What to expectYou will see one of these three outcomes:
17
1. Winning 2.Losing 3.Indecisive
18
# Winning tests
Total # of testsWin rate =
19
What’s a realistic goal?
50%0% 100%
In-house Agencies
19
What’s a bad test?
The only bad test is the one you don’t learn anything from
20
Case -
Lifesum (ShapeUpClub)
The Original
22
Variation 1
23
Test 1
24
Results
25
Engagement
Conversions
Revenue
+96%
+ 12%
+ 20%
+21% -15% +11%
+141%
WTF?
“Don’t be afraid to prioritize”
26
Test 2
27
Results
28
Engagement
Conversions
Revenue
+ 153% (+96%)
+ 51% (+12%)
+ 67% (+ 20%)
+26% + 32% +121% +188%
Key takeaways
- The only failed test is the one you don’t learn from
- Best practices don’t always work - Use several data points and
segments
Case - Namnband
Conversions Revenue
A/B-test #1
-6 % -13 %
A/B-test #2
+16 % +15 %
-25 % -48 %
Conversions Revenue
Conversions Revenue
34
35
Case - Spotify Premium
Premium trial for US market High Dropoff rate
Original
You must leave your credit card details in order to start your 30 day free trial
WHY?
37
Input to the hypothesisWe made a short exit survey.
The main reason stated:
38
“I don’t want to leave my credit card details for something that is free”!
Hypothesis:Give the user “A reason why”
39
We only use this to verify your account, you won't be charged anything for your trial
We need this just in case you decide to stay Premium after your free month
We need this because our music deals only allow free trials for users that are credit card or PayPal holders
A
B
C
The result
40
Original
“We only use this to verify…”
Blame the record companies
“In case you decide to stay…”
41
I’m waiting for the test results
Or my retirement
Whatever comes first
Case - Software company
Promo 1
46
Original
47
Variation - By the book
48
The Results?
49
0%
Because visitors don’t make any decision on
this page!
Average E-Commerce Uplift per Template Type
50
Testing sitewide to reduce bounce rate
51
Test 1 - Functional tagline
52
Engagement: + 6%
Original
Variation
Test 2 - Authority badge
53
Engagement: ±0%
Variation
Original
Product page test
54
Case - Swedoffice - Stripped Checkout
55
Checkout test
56
Test # 2
44
Knverteringsgrad +6% Intäkt per besökare + 10%
57
57
Remove navigation
Change up sales
The result?
58
58
0!
Why?
A heatmap in visual website optimizer gave us the answer59
Test #2
60
Conversion rate +6% Revenue per visitor + 10%
The BEST global
speakers
We’ve sold out every
singer year + 600 Attendees
No sponsors on stage
Scandinavia’s #1 Conversion Event
conversionjam.se