webshop wednesday #11: best practices and lessons learned from 100,000 a/b tests, martijn...

Best Practices and Lessons Learned from

100,000 A/B TestsA/B TestsMartijn Geerlings

100,000

Thursday, November 28, 13

Is de (klant) reis...

belangrijker dan de bestemming?


2009! 2010! 2012!2011! 2013!


Lessons Learned


Lesson #1:

Define success .


Lesson #1:

Define quantifiable success metrics.


Visitvisitors per day



Emailsign up

rate



Emailsign up

rate$

$ per recipient


Visit Email $sign up

rate $ per recipientvisitors per day


Media

Button


Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:

1.Get Involved Image

2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Splash page experimentVariations:


Button: “Sign Up”Thursday, November 28, 13

Button: “Learn More”Thursday, November 28, 13

Button: “Join Us Now”Thursday, November 28, 13

Button: “Sign Up Now”Thursday, November 28, 13

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:


2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment


Media: “Get Involved”Thursday, November 28, 13

Media: “Family”Thursday, November 28, 13

Media: “Change”Thursday, November 28, 13

Media: “Barack’s Video”Thursday, November 28, 13

Media: “Springfield Video”Thursday, November 28, 13

Media: “Sam’s Video”Thursday, November 28, 13

Button:

1.Sign Up

2.Learn More

3. Join Us Now

4.Sign Up Now

Media:


2.Family Image

3.Change Image

4.Barack’s Video

5.Springfield Video

6.Sam’s Video

Variations:Splash page experiment


Splash page experiment


Email Subscriptions Volunteers Amount Raised

Original: 7,120,000 712,000 $143,000,000

+40.6% +2,880,000 +288,000 +$57,000,000

New: 10,000,000 1,000,000 $200,000,000

Splash page experiment


Lesson #1:

Define quantifiable success metrics.


Lesson #2:

Less is more. Reduce choices.


vs.


vs.

+8% engagement


+43.8% Purchases


vs.


vs.

+14.8% conversion rate


Lesson #2:

Less is more. Reduce choices.


Lesson #3:

Words matter. Focus on your call to action.


Donation button experiment

Variations


Donation button experiment

VariationsNever

Signed UpSigned Up,

Never DonatedPreviously Donated

0.0% 0.0% 0.0%+2.3% +27.8% +16.3%-27.8% N/A N/A+15.2% -24.6% +11.9%+8.5% +2.9% +18.4%


vs.


vs.

+14.6% clicks


Lesson #3:

Words matter. Focus on your call to action.


Lesson #4:

Fail fast.


vs.


vs.

-10% conversions


vs.


vs.

-10% conversions


Lesson #4:

Fail fast.


Lesson #5:

Seek the Global Maximum.


+0.5% Retention +1.4% Engagement


Lesson #5:

Seek the Global Maximum.


1. Define quantifiable success metrics.2. Less is more. Reduce choices.3. Words matter. Focus on your call to action.4. Fail fast.5. Seek the Global Maximum.

Lessons Learned


One more thing...


6000+ happy testers

Start Today!


webshop wednesday #11: best practices and lessons learned from 100,000 a/b tests, martijn...

Self Improvement