a/b testing - what your mother didn't tell you

P.O.P.  Copyright  2013,  All  Around  The  World  January  14,  2014  

A/B  TesAng      

What  your  mother  didn’t  tell  you  hDp://www.allaroundtheworld.fr/  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

What  is  A/B  TesAng?  

•  TesAng  customers,  not  code  •  Show  different  versions  •  Pick  the  best  one  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Who  Relies  On  A/B  TesAng?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Transform  Your  Company  

•  Test  your  customers,  not  your  code  •  Stop  guessing  what  works  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Should  your  paycheck  depend  on  my  hunch?  

Conversion  Funnel  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Conversion  Funnel  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

→  

A  Basic  Test  

•  Your  current  behavior  is  the  base  •  A  variant  is  a  new  version  (just  one  change)  •  You  can  have  one  or  more  variants  

•  But  we  assume  only  one  for  this  talk  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Which  Generates  More  Clicks?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Which  Generates  More  Clicks?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

10,017  Visits   9,983  Visits  

125  Conversions   132  Conversions  

Which  Generates  More  Clicks?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

9,950  Visits   10,550  Visits  

125  Conversions   155  Conversions  

The  Numbers  (95%  Confidence)  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

hDp://getdelve.com/2011/12/math-­‐behind-­‐ab-­‐tesAng-­‐for-­‐curious-­‐execuAves-­‐part-­‐1/  

Version   Visits   Conversion   Rate   Not  Random  

Red   10,017   125   1.25%   —  

Blue   9,983   132   1.32%   51%  

The  Numbers  (95%  Confidence)  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

hDp://getdelve.com/2011/12/math-­‐behind-­‐ab-­‐tesAng-­‐for-­‐curious-­‐execuAves-­‐part-­‐1/  

Version   Visits   Conversion   Rate   Not  Random  

Red   9,950   125   1.26%   —  

Blue   10,550   155   1.47%   94%  

Which  Sells  More  Product?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Which  Creates  More  Profit?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

What  Confidence  Level?  

Is  99%  beDer  than  90%?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

The  Cycle  

Create  Experiment  

Only  one  difference  per  version!  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Roll  the  Dice  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Track  Performance  

Sta:s:cs  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Choose  the  Best  

Pa:ence  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Create  Experiment  

Wash,  rinse,  repeat  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Choose  the  Best  

•  Ignore  the  first  day  •  Full-­‐business  cycle  •  What  if  you’re  wrong?  

•  False  posiAves  

hDp://www.einarsen.no/is-­‐your-­‐ab-­‐tesAng-­‐effort-­‐just-­‐chasing-­‐staAsAcal-­‐ghosts/  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

LimitaAons  of  A/B  TesAng  

•  Doesn’t  tell  you  why  •  Side  effects  •  Doesn’t  tell  you  loyalty  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Benefits  

•  Hyundai:  62%  increase  in  test  drive  requests  •  Artsy  Editor:  47%  increase  in  click-­‐throughs  •  Visual  Website  OpAmizer:  50%  signup  increase  •  AssessmentDay:  62%  increase  in  sales  •  SmileyCookie:  41%  increase  in  sales  •  BaseKit:  25%  increase  in  conversions  •  Mobal:  27%  increase  in  sales  •  You  Need  A  Budget:  85%  increase  in  downloads  

hDp://visualwebsiteopAmizer.com/case-­‐studies.php  

What  Are  They  TesAng?  

•  Add-­‐to-­‐cart  buDon  placement  •  Call  to  acAon  buDons  •  Using  popup  forms  •  Removing  content  •  Link  color  •  Adding  tesAmonials  •  NavigaAon  menus  •  Removing  signup  forms  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Gemng  Started  

•  Vendor  or  in-­‐house?  •  Go  for  quick  wins  •  Run  an  AA  test  •  Be  pa:ent!  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Problems  

Ego  January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Problems  

•  Forgemng  caching  •  Non-­‐random  selecAon  

•  Forgemng  selecAon  

•  TesAng  only  the  view  •  Performance  

•  Dead  code  •  Profit  versus  conversion  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Tips  

•  Most  experiments  fail  •  Don’t  roll  the  dice  prematurely  

•  A/B  test  bugs  •  Test  “stupid”  ideas  •  Don’t  require  rollouts!  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

SegmentaAon  

•  New  versus  returning  customers  •  Different  browsers  •  Accept-­‐Language  •  Time  of  day  

•  IP  country  •  Gender?  Age?  Something  else?  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

See  Also  

•  The  MulA-­‐Armed  Bandit  hDp://stevehanov.ca/blog/index.php?id=132  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

Conclusion  

•  Lots  of  money  lying  on  the  table  •  Egos  are  bad  •  Data  is  good  •  Customer  behavior  trumps  expert  opinion  

•  Customer  behavior  trumps  code  behavior  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/  

QuesAons?  

hDp://www.allaroundtheworld.fr/  

January  14,  2014   ©  2014  hDp://www.allaroundtheworld.fr/