the evolution of a/b testing...advertising and marketing • improved roi = good • wasted...
Post on 23-May-2020
1 Views
Preview:
TRANSCRIPT
THE EVOLUTION
OF A/ B TESTING
@Nicole_Mint iens
Data Analysis & Insights Manager @GetG5
41 Shades of B lue
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
$200 MILLION
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
The Evolut ion of A/ B Test ing
@ Nicole_Mintiens
Adver t ising and Market ing• Im proved ROI = Good
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Adver t ising and Market ing• Im proved ROI = Good• Wast ed Resources = Bad
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Adver t ising and Market ing• Im proved ROI = Good• Wast ed Resources = Bad
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Adver t ising and Market ing• Im proved ROI = Good• Wast ed Resources = Bad• Market ing = Com plex
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Adver t ising and Market ing• Im proved ROI = Good• Wast ed Resources = Bad• Market ing = Com plex• Fact s > Opinion
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHY SPEND TIME TESTING?• $92 t raf f ic / $1 opt im izat ion
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHY SPEND TIME TESTING?• $92 t raf f ic / $1 opt im izat ion• 87% run 1-5 t est s/m ont h
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHY SPEND TIME TESTING?• $92 t raf f ic / $1 opt im izat ion• 87% run 1-5 t est s/m ont h
– Best brands run hundreds per m ont h
Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING FACTS• Im proves conversion rat e
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING FACTS• Im proves conversion rat e• Enhances ef fect iveness
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING FACTS• Im proves conversion rat e• Enhances ef fect iveness• Opt im izes ROI
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING FACTS• Im proves conversion rat e• Enhances ef fect iveness• Opt im izes ROI
So… YOU should t est !
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
The Evolut ion of A/ B Test ing
@ Nicole_Mintiens
MISCONCEPTIONS• Hur t conversion rat e
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MISCONCEPTIONS• Hur t conversion rat e• Reduce ef fect iveness
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MISCONCEPTIONS• Hur t conversion rat e• Reduce ef fect iveness• Wast e resources
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MISCONCEPTIONS• Hur t conversion rat e• Reduce ef fect iveness• Wast e resources• Dam age ROI
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
TESTING IS NOT
OPTIMIZATION
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
TESTING IS NOT
OPTIMIZATION…but it plays a role.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
DON’T JUST TEST, OPTIMIZE.• Avoid m ist akes
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
DON’T JUST TEST, OPTIMIZE.• Avoid m ist akes• Increase ef fect iveness
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
DON’T JUST TEST, OPTIMIZE.• Avoid m ist akes• Increase ef fect iveness• Im prove ROI
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
I’LL BE YOUR GUIDE• 7+ years opt im izing m ult ifam ily
– PPC, SMM, SEO & CRO
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING BASICS• St at ist ical hypot hesis-based
exper im ent at ion bet ween var iat ions of a cont rol.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING BASICS• St at ist ical hypot hesis-based
exper im ent at ion bet ween var iat ions of a cont rol.
• Goal: – Isolat e dif ferences t hat
m axim ize an out com e • i .e. conversion
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
CONVERSION DEFINITIONAct ion YOU want t he user t o t ake.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B/ N TESTING
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B/ N TESTING
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B/ N TESTING
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
• 2000 – Google runs f ir st t est t o f ind opt im um num ber of SERP
result s.
A/ B TESTING HISTORY
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
• 2000 – Google runs f ir st t est t o f ind opt im um num ber of SERP
result s.
• 2008 – A/B t est s raise addit ional $60 m il l ion for Obam a cam paign.
A/ B TESTING HISTORY
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
• 2000 – Google runs f ir st t est t o f ind opt im um num ber of SERP
result s.
• 2008 – A/B t est s raise addit ional $60 m il l ion for Obam a cam paign.
• 2010– Obam a cam paign’s Direct or of Analyt ics founds Opt im izely.
A/ B TESTING HISTORY
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
• 2000 – Google runs f ir st t est t o f ind opt im um num ber of SERP
result s.
• 2008 – A/B t est s raise addit ional $60 m il l ion for Obam a cam paign.
• 2010– Obam a cam paign’s Direct or of Analyt ics founds Opt im izely.
• 2011 – Google has run over 7,000 A/B t est s
A/ B TESTING HISTORY
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MULTIFAMILY AVG CONVERSION RATE
Source: 2016 G5 Multifamily Google Analytics Data
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MULTIFAMILY AVG CONVERSION RATE
Source: 2016 G5 Multifamily Google Analytics Data
~2%The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ECOMMERCE AVG CONVERSION RATE
~3%The Evolut ion of A/ B Test ing
@Nicole_Mint iens
AMAZON PRIME’S CONVERSION RATE
Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
2366%Better than
average!
AMAZON PRIME’S CONVERSION RATE
Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
AMAZON’S TECHNOLOGY
Source: Builtwith.com
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
AMAZON’S TECHNOLOGY
Source: Builtwith.com
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
SOFTWARE ADVANCEMENTS• Market ing aut om at ion
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
SOFTWARE ADVANCEMENTS• Market ing aut om at ion • Easy t o inst all
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
SOFTWARE ADVANCEMENTS• Market ing aut om at ion • Easy t o inst all• WYSIWYG edit ors
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHAT A/ B TESTING CAN DO
• Validat e hypot heses
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHAT A/ B TESTING CAN DO
• Validat e hypot heses• Inform choices
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHAT A/ B TESTING CAN DO
• Validat e hypot heses• Inform choices • Insight for it erat ion
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHAT A/ B TESTING CAN’T DO• Det erm ine YOUR
–Problem
Thanks, Boss!
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHAT A/ B TESTING CAN’T DO• Det erm ine YOUR
–Problem –User ’s
needs/want sThanks, Boss!
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHAT A/ B TESTING CAN’T DO• Det erm ine YOUR
–Problem –User ’s
needs/want s• Im prove t raf f ic
–Qualit y
Thanks, Boss!
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
WHAT A/ B TESTING CAN’T DO• Det erm ine YOUR
–Problem –User ’s
needs/want s• Im prove t raf f ic
–Qualit y–Volum e
Thanks, Boss!
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
CAN DOES NOT EQUAL SHOULDTest ing w il l wast e your t im e w it hout :• Enough t raf f ic
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
CAN DOES NOT EQUAL SHOULDTest ing w il l wast e your t im e w it hout :• Enough t raf f ic• Enough research
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
CAN DOES NOT EQUAL SHOULDTest ing w il l wast e your t im e w it hout :• Enough t raf f ic• Enough research• Technical resources
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
THEN HOW DO YOU IMPROVE?!
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A PROCESSt o im prove per form anceinform ed by INSIGHTand def ined by GOALS.
CONVERSION RATE OPTIMIZATION(CRO)
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A PROCESSt o im prove per form anceinform ed by INSIGHTand def ined by GOALS.
GOALS
CONVERSION RATE OPTIMIZATION(CRO)
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost• Ret urn on invest m ent
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost• Ret urn on invest m ent• Cust om er underst anding
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost• Ret urn on invest m ent• Cust om er underst anding• Google’s percept ion
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
GOALS & TRACKING
Source: http://borsen.dk/opinion/blogs/view/17/4159/den_svaereste_marketingdisciplin.html
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
YOUR IDEAL RESIDENTS
Prospects
Valu
e
I on ly rent t he m ost af fordable apar t m ent s.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
YOUR IDEAL RESIDENTS
Prospects
Valu
e
I on ly rent t he m ost af fordable apar t m ent s.
I’l l rent near m y t ech coworkers regardless of pr ice.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
CUSTOMER RESEARCH
Source: https://blog.kissmetrics.com/wp-content/uploads/2015/02/NYU.lowres.ifiwere.jpg
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
“The EFFECTIVENESSof any lead generat ion cam paign
com es down t o your HOOK.”Mike King @IPullRankFounder iPullRankConsult ant t o For t une/ INC500 brands.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
VOICE OF CUSTOMER RESEARCH• What m akes your
proper t y unique?
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
VOICE OF CUSTOMER RESEARCH• What m akes your
proper t y unique?
• Why you and not t he com pet it ion?
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
VOICE OF CUSTOMER RESEARCH• What m akes your
proper t y unique?
• Why you and not t he com pet it ion?
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
UNIQUE VALUE PROPOSITION (UVP):
Above-t he-fold EXPERIENCEt hat com m unicat es WHATyou do and WHYit ’s UNIQUE.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Or iginal (A)
UVP: FIVE SECOND TEST
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Or iginal (A) Var iat ion (B)
UVP: FIVE SECOND TEST
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Or iginal (A) Var iat ion (B)
UVP: FIVE SECOND TEST
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
CUSTOMER SEGMENTATION
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
CUSTOMER SEGMENTATION
User Persona Creator: https://xtensio.com/user-persona/
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Slide source: http://www.slideshare.net/aschottmuller/content-marketing-onpage-optimization-holy-grail
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
LARGEST LEAKS
Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
LARGEST LEAKS
Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYZE ENGAGEMENT PATTERNSHeat m aps
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYZE ENGAGEMENT PATTERNSHeat m aps
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYZE PERCEPTIONS
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYSIS FRAMEWORKG5 D-LITE
Deliver Relevance - Mat ch t hen exceed user expect at ions.DLITE
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYSIS FRAMEWORKG5 D-LITE
Lower Fr ict ion - Reduce user anxiet y. Sim plify t heir process.
Deliver Relevance - Mat ch t hen exceed user expect at ions.DLITE
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYSIS FRAMEWORKG5 D-LITE
Im prove Clar it y - Ensure m essaging is clear , concise & valued.
Lower Fr ict ion - Reduce user anxiet y. Sim plify t heir process.
Deliver Relevance - Mat ch t hen exceed user expect at ions.DLITE
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYSIS FRAMEWORKG5 D-LITE
Im prove Clar it y - Ensure m essaging is clear , concise & valued.
Lower Fr ict ion - Reduce user anxiet y. Sim plify t heir process.
Deliver Relevance - Mat ch t hen exceed user expect at ions.
Tr igger Act ion - Mot ivat e conversion. Rem ove dist ract ion.
DLITE
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
ANALYSIS FRAMEWORKG5 D-LITE
Im prove Clar it y - Ensure m essaging is clear , concise & valued.
Lower Fr ict ion - Reduce user anxiet y. Sim plify t heir process.
Deliver Relevance - Mat ch t hen exceed user expect at ions.
Exper im ent - Learn, t est and opt im ize for your users.
Tr igger Act ion - Mot ivat e conversion. Rem ove dist ract ion.
DLITE
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
“A solid t est hypot hesis is an
INFORMED SOLUTIONt o a real problem - not an arbit rary guess. ”Michael Aagaard
@Cont ent VerveInt ernat ional Speaker and Senior CRO @Unbounce
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
HYPOTHESIS TEMPLATEIf [cause], t hen [ef fect ], because [rat ionale].
Source: https://help.optimizely.com/Ideate_and_Hypothesize/Design_a_hypothesis_that_drives_your_business_goals
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
TESTING
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
TESTING Wait … do you even have enough t raf f ic?
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Gives you CONFIDENCEt hat t he changes you m ake t o your websit e ACTUALLYhave a posit ive IMPACT.
https://www.optimizely.com/optimization-glossary/statistical-significance/
STATISTICAL SIGNIFICANCE
“”
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MULTIFAMILY AVG SESSIONS/ DAY
Source: 2016 G5 Multifamily Google Analytics Data
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MULTIFAMILY AVG SESSIONS/ DAY
Source: 2016 G5 Multifamily Google Analytics Data
66The Evolut ion of A/ B Test ing
@Nicole_Mint iens
TEST DURATION CALCULATOR
Source: https://www.convert.com/tools/ab-test-duration-calculator/
Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
TEST DURATION CALCULATOR
Source: https://www.convert.com/tools/ab-test-duration-calculator/
Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
LOW TRAFFIC OPTIMIZATION
Source: http://unbounce.com/landing-pages/i-dont-have-enough-traffic-to-ab-test-my-landing-pages/
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING TOOLS
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
A/ B TESTING TOOLS
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
BUILD VARIATIONSUse t echnical resources t o ensure proper set up and t rack ing
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
BUILD VARIATIONSUse t echnical resources t o ensure proper set up and t rack ing
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
LAUNCH TESTOr iginal
(A)Var iat ion
(B)
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
LAUNCH TESTOr iginal
(A)Var iat ion
(B)
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MEASURE RESULTS
Image source: https://www.optimizely.com/ab-testing/
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MEASURE RESULTS
Image source: https://www.optimizely.com/ab-testing/
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
DRAW INSIGHTS AND ITERATE
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
Your propert ies aren't 1-size f it s all and your st rategy shouldn't be eit her.
DRAW INSIGHTS AND ITERATE
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
MOST IMPORTANTLYResearch and
user feedback is
VITAL.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
REPEAT
GOALS
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
THERE IS NO
BEST COLOR
41 Shades of B lue
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
INDUSTRY BENCHMARKS
• Cont ext less dat a
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
INDUSTRY BENCHMARKS
• Cont ext less dat a• Weak conclusions
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
INDUSTRY BENCHMARKS
• Cont ext less dat a• Weak conclusions• Blind duplicat ion
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
INDUSTRY BENCHMARKS
• Cont ext less dat a• Weak conclusions• Blind duplicat ion • Tact ics over st rat egy
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
USE STRATEGY,NOT JUST TACTICS
• Avoid r isks
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
USE STRATEGY,NOT JUST TACTICS
• Avoid r isks• Use dat a ef fect ively
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
USE STRATEGY,NOT JUST TACTICS
• Avoid r isks• Use dat a ef fect ively• Discover solut ions t o
YOUR problem s
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Track ing
– Heat m apping
– Test Durat ion Calculat or
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Track ing
– Heat m apping
– Test Durat ion Calculat or
• Analysis Tools:G5 D-LITE CRO Fram ework
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Track ing
– Heat m apping
– Test Durat ion Calculat or
• Analysis Tools:G5 D-LITE CRO Fram ework
• Hypot heses:If [cause], t hen [ef fect ], because [rat ionale]
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
FREEMIUM (OR CLOSE) RESOURCES• Research:
– Personas
– Track ing
– Heat m apping
– Test Durat ion Calculat or
• Analysis Tools:G5 D-LITE CRO Fram ework
• Hypot heses:If [cause], t hen [ef fect ], because [rat ionale]
• Test ing:
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
“CRO IS LEARNING WHYYOUR USERSaren’t
conver t ing and FIXING IT.”The Evolut ion of A/ B Test ing
@Nicole_Mint iens
GET THESE SLIDES:
ht tps:/ / goo.gl/ WUtpe5
THANKS!
DON’T JUST TEST, OPTIMIZE.
The Evolut ion of A/ B Test ing
@Nicole_Mint iens
top related