best practices & lessons learned from over 223,628 a/b tests

Best Practices and Lessons Learned from

Dan SirokerCo-Founder & CEO

A/B Tests

Wednesday, June 5, 13



A/B Tests223,628


‘05‘06‘07‘08’09’10


Lessons Learned


1.

2. Less is more. Reduce choices.3. Words matter. Focus on your call

to action.

4. Seek the global maximum.

5. Fail fast.

6. Start today.

Lessons LearnedDefine quantifiable success metrics.


Lesson #1:

Define quantifiable success metrics.


Visitvisitors per day


Visitvisitors per day Email

sign up rate


Visitvisitors per day Email

sign up rate $

$ per recipient


Visit Email $sign up

rate$ per

recipientvisitors per day


Media

Button


Button:

1. Sign Up

2. Learn More

3. Join Us Now

4. Sign Up Now

Media:

1. Get Involved Image

2. Family Image

3. Change Image

4. Barack’s Video

5. Springfield Video

6. Sam’s Video

Splash page experimentVariations:


Button: “Sign Up”


Button: “Learn More”


Button: “Join Us Now”


Button: “Sign Up Now”


Button:

1. Sign Up

2. Learn More

3. Join Us Now

4. Sign Up Now

Media:


2. Family Image

3. Change Image

4. Barack’s Video


6. Sam’s Video

Variations:

Splash page experiment


Media: “Get Involved”


Media: “Family”


Media: “Change”


Media: “Barack’s Video”


Media: “Springfield Video”


Media: “Sam’s Video”


Button:

1. Sign Up

2. Learn More

3. Join Us Now

4. Sign Up Now

Media:


2. Family Image

3. Change Image

4. Barack’s Video


6. Sam’s Video

Variations:



Email Subscriptions Volunteers Amount

Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000



Lesson #1:

Define quantifiable success metrics.


Lesson #2:

Less is more. Reduce choices.


vs.


vs.

+15% conversion rate

+10% $ per visitor


vs.


vs.

+8% engagement


vs.


vs.+16.8% $ per visitor


v.


v.

+14.8% conversion rate


v.


Lesson #2:

Less is more. Reduce choices.


Lesson #3:

Words matter. Focus on your call to action.


Donation button experiment

Variations


NeverSigned Up

Signed Up, Never

Donated

Previously Donated

0.0% 0.0% 0.0%

+2.3% +27.8% +16.3%

-27.8% N/A N/A

+15.2% -24.6% +11.9%

+8.5% +2.9% +18.4%

Donation button experiment

Variations


vs.


vs.

+14.6% clicks


vs.


vs.+15.75% $ per visitor


Lesson #3:

Words matter. Focus on your call to action.


Lesson #4:

Seek the global maximum.


Source: http://blog.intercom.io/criticism-and-two-way-streets/


http://blog.intercom.io/criticism-and-two-way-streets/

http://blog.intercom.io/criticism-and-two-way-streets/

vs.


vs.

Comments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html


http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html


vs.

+0.5% retention+1.4% engagement

Comments: http://blog.netflix.com/2011/06/new-look-and-feel-for-netflix-website.html




vs.


vs.

+600% engagement


vs.


vs.

+49% conversion rate from 240 A/B tests


Lesson #4:

Seek the global maximum.


Lesson #5:

Fail fast.


vs.


vs.-92.3% video clicks


vs.


vs.

-10% conversions


vs.


vs.

fewer conversions


Lesson #5:

Fail fast.


Lesson #6:

Start today.


Source: http://techcrunch.com/2013/04/10/optimizely-series-a/


http://techcrunch.com/2013/04/10/optimizely-series-a/

http://techcrunch.com/2013/04/10/optimizely-series-a/

4,000+ Happy Customers!


Lesson #6:

Start today.


1. Define quantifiable success metrics.

2. Less is more. Reduce choices.

3. Words matter. Focus on your call to action.

4. Seek the global maximum.

5. Fail fast.

6. Start today.

Lessons Learned


The Future of Optimization



What’s the best experience for the average user?

One to Many




One to Many One to FewWhat’s the best experience

for segments of users?




One to Many One to FewWhat’s the best experience

for segments of users?

One to OneWhat’s the best experience

for each user?


One to Many


One to Few


One to One




A/B Tests223,628


Appendix


best practices & lessons learned from over 223,628 a/b tests

Technology