the evolution of a/b testing...advertising and marketing • improved roi = good • wasted...

THE EVOLUTION

OF A/ B TESTING

@Nicole_Mint iens

Data Analysis & Insights Manager @GetG5

41 Shades of B lue

The Evolut ion of A/ B Test ing

@Nicole_Mint iens

$200 MILLION


@Nicole_Mint iens


@ Nicole_Mintiens

Adver t ising and Market ing• Im proved ROI = Good


@Nicole_Mint iens

Adver t ising and Market ing• Im proved ROI = Good• Wast ed Resources = Bad


@Nicole_Mint iens

Adver t ising and Market ing• Im proved ROI = Good• Wast ed Resources = Bad• Market ing = Com plex


@Nicole_Mint iens

Adver t ising and Market ing• Im proved ROI = Good• Wast ed Resources = Bad• Market ing = Com plex• Fact s > Opinion


@Nicole_Mint iens

WHY SPEND TIME TESTING?• $92 t raf f ic / $1 opt im izat ion

Source: Econsultancy & Redeye Conversion Rate Optimization Report 2014


@Nicole_Mint iens

WHY SPEND TIME TESTING?• $92 t raf f ic / $1 opt im izat ion• 87% run 1-5 t est s/m ont h



@Nicole_Mint iens

WHY SPEND TIME TESTING?• $92 t raf f ic / $1 opt im izat ion• 87% run 1-5 t est s/m ont h

– Best brands run hundreds per m ont h



@Nicole_Mint iens

A/ B TESTING FACTS• Im proves conversion rat e


@Nicole_Mint iens

A/ B TESTING FACTS• Im proves conversion rat e• Enhances ef fect iveness


@Nicole_Mint iens

A/ B TESTING FACTS• Im proves conversion rat e• Enhances ef fect iveness• Opt im izes ROI


@Nicole_Mint iens

A/ B TESTING FACTS• Im proves conversion rat e• Enhances ef fect iveness• Opt im izes ROI

So… YOU should t est !


@Nicole_Mint iens


@ Nicole_Mintiens

MISCONCEPTIONS• Hur t conversion rat e


@Nicole_Mint iens

MISCONCEPTIONS• Hur t conversion rat e• Reduce ef fect iveness


@Nicole_Mint iens

MISCONCEPTIONS• Hur t conversion rat e• Reduce ef fect iveness• Wast e resources


@Nicole_Mint iens

MISCONCEPTIONS• Hur t conversion rat e• Reduce ef fect iveness• Wast e resources• Dam age ROI


@Nicole_Mint iens

TESTING IS NOT

OPTIMIZATION


@Nicole_Mint iens

TESTING IS NOT

OPTIMIZATION…but it plays a role.


@Nicole_Mint iens

DON’T JUST TEST, OPTIMIZE.• Avoid m ist akes


@Nicole_Mint iens

DON’T JUST TEST, OPTIMIZE.• Avoid m ist akes• Increase ef fect iveness


@Nicole_Mint iens

DON’T JUST TEST, OPTIMIZE.• Avoid m ist akes• Increase ef fect iveness• Im prove ROI


@Nicole_Mint iens

I’LL BE YOUR GUIDE• 7+ years opt im izing m ult ifam ily

– PPC, SMM, SEO & CRO


@Nicole_Mint iens

A/ B TESTING BASICS• St at ist ical hypot hesis-based

exper im ent at ion bet ween var iat ions of a cont rol.


@Nicole_Mint iens

A/ B TESTING BASICS• St at ist ical hypot hesis-based

exper im ent at ion bet ween var iat ions of a cont rol.

• Goal: – Isolat e dif ferences t hat

m axim ize an out com e • i .e. conversion


@Nicole_Mint iens

CONVERSION DEFINITIONAct ion YOU want t he user t o t ake.


@Nicole_Mint iens

A/ B/ N TESTING


@Nicole_Mint iens

• 2000 – Google runs f ir st t est t o f ind opt im um num ber of SERP

result s.

A/ B TESTING HISTORY


@Nicole_Mint iens


result s.

• 2008 – A/B t est s raise addit ional $60 m il l ion for Obam a cam paign.



@Nicole_Mint iens


result s.


• 2010– Obam a cam paign’s Direct or of Analyt ics founds Opt im izely.



@Nicole_Mint iens


result s.


• 2010– Obam a cam paign’s Direct or of Analyt ics founds Opt im izely.

• 2011 – Google has run over 7,000 A/B t est s



@Nicole_Mint iens

MULTIFAMILY AVG CONVERSION RATE

Source: 2016 G5 Multifamily Google Analytics Data


@Nicole_Mint iens

MULTIFAMILY AVG CONVERSION RATE


~2%The Evolut ion of A/ B Test ing

@Nicole_Mint iens

ECOMMERCE AVG CONVERSION RATE

~3%The Evolut ion of A/ B Test ing

@Nicole_Mint iens

AMAZON PRIME’S CONVERSION RATE

Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time


@Nicole_Mint iens

https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time

2366%Better than

average!

AMAZON PRIME’S CONVERSION RATE

Source: https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time


@Nicole_Mint iens

https://www.internetretailer.com/2015/06/25/amazon-prime-members-convert-74-time

AMAZON’S TECHNOLOGY

Source: Builtwith.com


@Nicole_Mint iens

SOFTWARE ADVANCEMENTS• Market ing aut om at ion


@Nicole_Mint iens

SOFTWARE ADVANCEMENTS• Market ing aut om at ion • Easy t o inst all


@Nicole_Mint iens

SOFTWARE ADVANCEMENTS• Market ing aut om at ion • Easy t o inst all• WYSIWYG edit ors


@Nicole_Mint iens

WHAT A/ B TESTING CAN DO

• Validat e hypot heses


@Nicole_Mint iens


• Validat e hypot heses• Inform choices


@Nicole_Mint iens


• Validat e hypot heses• Inform choices • Insight for it erat ion


@Nicole_Mint iens

WHAT A/ B TESTING CAN’T DO• Det erm ine YOUR

–Problem

Thanks, Boss!


@Nicole_Mint iens


–Problem –User ’s

needs/want sThanks, Boss!


@Nicole_Mint iens



needs/want s• Im prove t raf f ic

–Qualit y

Thanks, Boss!


@Nicole_Mint iens



needs/want s• Im prove t raf f ic

–Qualit y–Volum e

Thanks, Boss!


@Nicole_Mint iens

CAN DOES NOT EQUAL SHOULDTest ing w il l wast e your t im e w it hout :• Enough t raf f ic


@Nicole_Mint iens

CAN DOES NOT EQUAL SHOULDTest ing w il l wast e your t im e w it hout :• Enough t raf f ic• Enough research


@Nicole_Mint iens

CAN DOES NOT EQUAL SHOULDTest ing w il l wast e your t im e w it hout :• Enough t raf f ic• Enough research• Technical resources


@Nicole_Mint iens

THEN HOW DO YOU IMPROVE?!


@Nicole_Mint iens

A PROCESSt o im prove per form anceinform ed by INSIGHTand def ined by GOALS.

CONVERSION RATE OPTIMIZATION(CRO)


@Nicole_Mint iens

A PROCESSt o im prove per form anceinform ed by INSIGHTand def ined by GOALS.

GOALS

CONVERSION RATE OPTIMIZATION(CRO)


@Nicole_Mint iens

ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost


@Nicole_Mint iens

ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost• Ret urn on invest m ent


@Nicole_Mint iens

ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost• Ret urn on invest m ent• Cust om er underst anding


@Nicole_Mint iens

ALWAYS ROOM FOR IMPROVEMENT• Cust om er acquisit ion cost• Ret urn on invest m ent• Cust om er underst anding• Google’s percept ion


@Nicole_Mint iens

GOALS & TRACKING

Source: http://borsen.dk/opinion/blogs/view/17/4159/den_svaereste_marketingdisciplin.html


@Nicole_Mint iens

YOUR IDEAL RESIDENTS

Prospects

Valu

e

I on ly rent t he m ost af fordable apar t m ent s.


@Nicole_Mint iens

YOUR IDEAL RESIDENTS

Prospects

Valu

e

I on ly rent t he m ost af fordable apar t m ent s.

I’l l rent near m y t ech coworkers regardless of pr ice.


@Nicole_Mint iens

CUSTOMER RESEARCH

Source: https://blog.kissmetrics.com/wp-content/uploads/2015/02/NYU.lowres.ifiwere.jpg


@Nicole_Mint iens

“The EFFECTIVENESSof any lead generat ion cam paign

com es down t o your HOOK.”Mike King @IPullRankFounder iPullRankConsult ant t o For t une/ INC500 brands.


@Nicole_Mint iens

VOICE OF CUSTOMER RESEARCH• What m akes your

proper t y unique?


@Nicole_Mint iens

VOICE OF CUSTOMER RESEARCH• What m akes your

proper t y unique?

• Why you and not t he com pet it ion?


@Nicole_Mint iens

UNIQUE VALUE PROPOSITION (UVP):

Above-t he-fold EXPERIENCEt hat com m unicat es WHATyou do and WHYit ’s UNIQUE.


@Nicole_Mint iens

Or iginal (A)

UVP: FIVE SECOND TEST


@Nicole_Mint iens

Or iginal (A) Var iat ion (B)

UVP: FIVE SECOND TEST


@Nicole_Mint iens

CUSTOMER SEGMENTATION


@Nicole_Mint iens

CUSTOMER SEGMENTATION

User Persona Creator: https://xtensio.com/user-persona/


@Nicole_Mint iens

https://xtensio.com/user-persona/

Slide source: http://www.slideshare.net/aschottmuller/content-marketing-onpage-optimization-holy-grail


@Nicole_Mint iens

LARGEST LEAKS

Image source: https://s-media-cache-ak0.pinimg.com/564x/7e/c9/f9/7ec9f92ab4e7819c76011e432e760ece.jpg


@Nicole_Mint iens

ANALYZE ENGAGEMENT PATTERNSHeat m aps


@Nicole_Mint iens

ANALYZE PERCEPTIONS


@Nicole_Mint iens

ANALYSIS FRAMEWORKG5 D-LITE

Deliver Relevance - Mat ch t hen exceed user expect at ions.DLITE


@Nicole_Mint iens


Lower Fr ict ion - Reduce user anxiet y. Sim plify t heir process.



@Nicole_Mint iens


Im prove Clar it y - Ensure m essaging is clear , concise & valued.




@Nicole_Mint iens




Deliver Relevance - Mat ch t hen exceed user expect at ions.

Tr igger Act ion - Mot ivat e conversion. Rem ove dist ract ion.

DLITE


@Nicole_Mint iens




Deliver Relevance - Mat ch t hen exceed user expect at ions.

Exper im ent - Learn, t est and opt im ize for your users.

Tr igger Act ion - Mot ivat e conversion. Rem ove dist ract ion.

DLITE


@Nicole_Mint iens

“A solid t est hypot hesis is an

INFORMED SOLUTIONt o a real problem - not an arbit rary guess. ”Michael Aagaard

@Cont ent VerveInt ernat ional Speaker and Senior CRO @Unbounce


@Nicole_Mint iens

HYPOTHESIS TEMPLATEIf [cause], t hen [ef fect ], because [rat ionale].

Source: https://help.optimizely.com/Ideate_and_Hypothesize/Design_a_hypothesis_that_drives_your_business_goals


@Nicole_Mint iens

TESTING


@Nicole_Mint iens

TESTING Wait … do you even have enough t raf f ic?


@Nicole_Mint iens

Gives you CONFIDENCEt hat t he changes you m ake t o your websit e ACTUALLYhave a posit ive IMPACT.

https://www.optimizely.com/optimization-glossary/statistical-significance/

STATISTICAL SIGNIFICANCE

“”


@Nicole_Mint iens

MULTIFAMILY AVG SESSIONS/ DAY



@Nicole_Mint iens

MULTIFAMILY AVG SESSIONS/ DAY


66The Evolut ion of A/ B Test ing

@Nicole_Mint iens

TEST DURATION CALCULATOR

Source: https://www.convert.com/tools/ab-test-duration-calculator/

Image source: http://www.vintag.es/2016/09/the-good-old-days-12-crazy-vintage-ads.html


@Nicole_Mint iens

LOW TRAFFIC OPTIMIZATION

Source: http://unbounce.com/landing-pages/i-dont-have-enough-traffic-to-ab-test-my-landing-pages/


@Nicole_Mint iens

A/ B TESTING TOOLS


@Nicole_Mint iens

BUILD VARIATIONSUse t echnical resources t o ensure proper set up and t rack ing


@Nicole_Mint iens

LAUNCH TESTOr iginal

(A)Var iat ion

(B)


@Nicole_Mint iens

MEASURE RESULTS

Image source: https://www.optimizely.com/ab-testing/


@Nicole_Mint iens

DRAW INSIGHTS AND ITERATE


@Nicole_Mint iens

Your propert ies aren't 1-size f it s all and your st rategy shouldn't be eit her.

DRAW INSIGHTS AND ITERATE


@Nicole_Mint iens

MOST IMPORTANTLYResearch and

user feedback is

VITAL.


@Nicole_Mint iens

REPEAT

GOALS


@Nicole_Mint iens

THERE IS NO

BEST COLOR

41 Shades of B lue


@Nicole_Mint iens

INDUSTRY BENCHMARKS

• Cont ext less dat a


@Nicole_Mint iens

INDUSTRY BENCHMARKS

• Cont ext less dat a• Weak conclusions


@Nicole_Mint iens

INDUSTRY BENCHMARKS

• Cont ext less dat a• Weak conclusions• Blind duplicat ion


@Nicole_Mint iens

INDUSTRY BENCHMARKS

• Cont ext less dat a• Weak conclusions• Blind duplicat ion • Tact ics over st rat egy


@Nicole_Mint iens

USE STRATEGY,NOT JUST TACTICS

• Avoid r isks


@Nicole_Mint iens


• Avoid r isks• Use dat a ef fect ively


@Nicole_Mint iens


• Avoid r isks• Use dat a ef fect ively• Discover solut ions t o

YOUR problem s


@Nicole_Mint iens

FREEMIUM (OR CLOSE) RESOURCES• Research:

– Personas

– Track ing

– Heat m apping

– Test Durat ion Calculat or


@Nicole_Mint iens

https://www.convert.com/tools/ab-test-duration-calculator/


https://www.google.com/analytics/analytics/#?modal_active=none


– Personas

– Track ing

– Heat m apping


• Analysis Tools:G5 D-LITE CRO Fram ework


@Nicole_Mint iens





– Personas

– Track ing

– Heat m apping



• Hypot heses:If [cause], t hen [ef fect ], because [rat ionale]


@Nicole_Mint iens





– Personas

– Track ing

– Heat m apping



• Hypot heses:If [cause], t hen [ef fect ], because [rat ionale]

• Test ing:


@Nicole_Mint iens

https://usabilityhub.com/

https://adwords.google.com




https://www.google.com/analytics/optimize/

“CRO IS LEARNING WHYYOUR USERSaren’t

conver t ing and FIXING IT.”The Evolut ion of A/ B Test ing

@Nicole_Mint iens

GET THESE SLIDES:

ht tps:/ / goo.gl/ WUtpe5

THANKS!

DON’T JUST TEST, OPTIMIZE.


@Nicole_Mint iens

the evolution of a/b testing...advertising and marketing • improved roi = good • wasted...

Documents