#conversion2015 amsterdam keynote ton wesseling

O N L I N E D I A L O G U E G u i d i n g y o u t o d i g i t a l m a t u r i t y

D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s # C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5

To n W e s s e l i n g

A / B - t e s t e n

E m a i l : t o n @ w e s s e l i n g . n e t !L i n k e d i n : t o n w e s s e l i n g !

T w i t t e r : @ t o n w !!

Ton Wessel ing !

onlinedialogue.com #Conversion2015 – Amsterdam – Ton Wesseling – April 14th 2015

Webanal isten.nl


Onl ine Dia logue


Test ing.agency


Emerce magazine: conversion column


Conversion Conferences wor ldwide


Conversion speakers


Where is Craig Sul l ivan?


Replacing Craig…


At least he quotes me


So..


More exper iments

“If you double the number of experiments you do per year

you’re going to double your inventiveness”

Jeff Bezos, CEO Amazon – 2004!


Like Amazon, Like Zalando G

erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


Exper iments per year

1000+


Why not?

Compare to the past?


Why can this lead to fa i lures?


A big weekly change!

26 out of 44 weeks: over 5% change!


Methods used to improve conversion


Methods companies plan to use


I t ’s just

HOT


Lack of resources


Test more


Importance of a structured approach


More exper iments


The negat ive effect trend

Time span!

Aver

age

effe

ct o

f opt

imza

tion!

at c

ompa

nies

app

lyin

g op

timiz

atio

n!M

oney spend on!O

ptimization w

orldwide!


I t ’s not a solut ion

It’s a method


I t ’s a way of working

It’s DNA


So i t ’s a method:

When

How What


Growth curve – how i t was teached


Optimizat ion phases - ROAR

Time span!

Con

vers

ions

per

mon

th!

Risk! + Optimization! + Automation! Re-think!

1.000 conversions!per month!

10.000 conversions per month!


Phase 1: br idge the gap – Risk


Fake adwords test ing

Do they click

on your idea?


Fake facebook test ing

Do they like your idea?


Fake landing page test ing

Do they subscribe to your idea?


Phase 1: br idge the gap – Risk


Get out of the office


Risk


ROAR - Opt imizat ion

Time span!

Con

vers

ions

per

mon

th!





Warning: low significance levels

ü  Big chance the winner is not a real winner


Warning: low power levels

ü  Big chance a real winner is not recognized


Example test set-up

ü  90% significance level

ü  100 conversions per variation

ü  Average conversion rate: 2%

ü  20 out of 100 ideas is a winner…

ü  …with an average uplift of 10%


Average: 75% significance & 40% power


So:

ü  100 tests will give 10 false positives (90% significance)

ü  40% of the 20 winners are recognized (40% power)

ü  8 real winners & 10 false positives

ü  NOT 18 improvements of 10%+


You need big numbers per var iat ion

Time span!

Con

vers

ions

per

mon

th!





With 1000 conversions + a month

20+ tests a year


Tests with impact, on average

1 out of 3

Simply truth: you’re just not always able to create a winner with enough impact!


So in th is case, you’ve impact

Every 6 weeks


Slower? Energy wi l l run out!


What to test? Business cases!

Delivery –test with in stock. 4 weeks

or 2 days?!


What to test? Business cases!

Service charge, do it or not? Include in price or not?!


What to test? Big design changes


Warning: choose what to analyze up front

ü  You just go and dig unitil you find something

(you will always find something that seems true BUT is not!)


Don’t stop the test once i t ’s s ignificant!


You just let i t run unt i l i t ’s s ignificant


You just let i t run unt i l i t ’s s ignificant

Your test has a fixed lenght!

It’s even a fixed amount of test population, but we love to test business cycles and/or weeks!


ABTestGuide.com/calc


ROAR – Opt imizat ion – moving up

Time span!

Con

vers

ions

per

mon

th!





Final ly you can test detai ls…

Is Multi Variatie Testing (which is also A/B-testing but with more varations) a good idea? Almost never!!


You should grow to

Customer Buying Reasons


Reasons behind the customer journey


Why is your user here?


How can you

help your user’s brain

to fullfil it’s needs?


Data & Psychology

Digital Data Persuasion Psychology

Analyze &

Experiment


Test ing takes t ime

Use it to LEARN


Lack of resources


Create a test schedule


FA C T & A C T – 7 s t e p s t o l e a r n

&

Tell Conclude Analyze

Test Create Analyze Find


Source: Psychology


Source: Ask


Source: L isten


Many sources out there to find & analyze


Test which dia logue has impact


Analyt ics


Heatmaps


Which locat ions are cr i t ical


FA C T & A C T – 7 s t e p s t o l e a r n

&

Tell Conclude Analyze

Test Create Analyze Find


Now you’re test ing

On your way to 4 tests a week

Max capacity with 1 teammember on each discipline: analyst, ux, developer, psychologyst!


Your winners are st i l l spare?

Less than 25%?

Please higher your significance level to 95% or more if it drops below this 25 number!!


T ip: Your test takes too long

ü  Cookie deletion (you will loose 10% in two weeks)

They will re-enter the the and pollute your samples!


T ip: Don’t confuse cross device users

ü Only test 1 device at a time!


T ip: Maybe you’re test ing broken stuff

ü  Always check browser compatibility!

ü Dont’ break dynamic stuff!


T ip: You forgot to leave out some vis i tors

ü  Only test on those who can be changed (if you test a sales promotion – leave out your current clients who just logon)


T ip: You’re not s lowing the control down

ü  Add the same code to the control


T ip: You’re not test ing on fresh people

ü  Start with control for a full purchase cycle (and then start sending traffic to your variations – setting the control to 0%)


T ip: You forgot to give them t ime to buy

ü  Stop getting fresh visitors after your test time (send new traffic to the control, but give you tested people time to finish converting)


ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!





You’re already th inking of more teams

And still have test capacity

*test capacity – the number of tests you can start per week based on your audience & conversions!


Algor i thms


You tested which locat ions are cr i t ical


Automation


Now you’ve moved to

Algorithm testing

Which internal and external influencers do have an impact on which segment of users?!


ROAR – Automation

Time span!

Con

vers

ions

per

mon

th!





CRO becomes user knowledge


More impact on discover


A way of l i fe


Customer knowledge center

Making money f rom convers ion improvements

Creat ing more and more cus tomer ins igh ts Which can be monet ized

on even a la rger sca le


Now, that’s

HOT


So you keep on groing G

erm

an E

com

mer

ce m

arke

t, th

ank

you

Andr

é M

orys


Done test ing? No!

TESTING NEVER STOPS

Okay, end of life, cash cow, no need for any further growth accelaration or user knowledge!


ROAR – when to start with what

Time span!

Con

vers

ions

per

mon

th!




O N L I N E D I A L O G U E G u i d i n g y o u t o d i g i t a l m a t u r i t y

D e m y t h e n , d e l e u g e n s e n d e i l l u s i e s # C o n v e r s i o n 2 0 1 5 – A m s t e r d a m – A p r i l 1 4 t h 2 0 1 5

To n W e s s e l i n g

A / B - t e s t e n

E m a i l : t o n @ w e s s e l i n g . n e t !L i n k e d i n : t o n w e s s e l i n g !

T w i t t e r : @ t o n w !!

Ton Wessel ing !

#conversion2015 amsterdam keynote ton wesseling

Business

i n e d

e d i n

b t e s t e n e

e u g e n s e n d e

t o n w e s s e

u s i e s

t w i t t e r

s t e r d