The Power of A/B Split TestingPresentation By: Alex Foley

August 4, 2016

My pain point: “I’m not sure how to best open up my

presentation at #DOYOLive.”

HYPOTHESIS 1: ENTHUSIASTIC, CONFIDENT; WWE TYPE

My hypothesis: The crowd may need to be energized!

Pro:Crowd attention and energy

Con:Potentially make a fool of myself

FUN FACT…

HYPOTHESIS 2: RIDE THE POP CULTURE WAVEMy hypothesis: A Pokemon Go reference can’t fail, right?

Pro:Widely known and top of mind

Con:Some folks may already be sick of the Poke-mania

The Results?

VOTE ON MY NEXT TEST

• Justin Bieber

• Drill Instructor

• Trump

WHAT WE WILL LEARNWhat is it?

Why it’s important?

Best Practices

The New Wave of Website Development

Resources

How Marketing “Used to” Be Done

HOW MARKETING “USED TO” BE DONE

Source: isha.sadguru.org

87% of Marketers Consider Data Their Most Underutilized Asset.

-Teradata 2015 Global Data-Driven Marketing Survey

What is A/B Split Testing?

A/B Split Testing: testing a hypothesis by measuring the performance of one version (A) against another (B) in an equal sample size.

In reality – it is the scientific method

Why is this important for us as marketers?

It keeps you thinking, it keeps you sharp

It is the formula for continuous improvement

You don’t have to be right the first time

It can give you an edge on the competition

It directly impacts the bottom line

74% of companies who take a structured approach to conversion (including A/B testing) have improved sales

Source: Econsultancy and RedEye 2014

Modern Origins

TAGUCHI’S LOSS FUNCTION—A CAR PART EXAMPLE

Bad Car Part

Take Car to Dealer

Dealer Staffs

Accordingly

Time Wasted on Preventable Problem

Missing Work to

Deal with Repairs

YOU MIGHT SAY THIS IS…

THE BUTTERFREE EFFECT!

THE FOUNDATION OF TODAY’S LEADING COMPANIES

THE BASIC A/B TESTING METHODOLOGY

A/B Learnings

Define Goals and

Hypothesis

Test a variable

Analyze and

Conclude

Record Findings

The Golden Rule of A/B Testing:Only test 1 variable at a time!

WHAT SHOULD I BE TESTING?• Email subject lines• Email preview text• CTAs• Colors• Size• Copy• Lengths of copy• Types• Sales copy• Buy now vs. learn more vs. x• Headlines• Font• Font color• Bullets vs. no bullets• Amount of content • Amount of choices or CTAs

• Page templates• Layouts• Heavy images or video vs. text• Email personalization tokens• Pricing• Pricing copy• Form aesthetics• Images – size, positioning, type, #• Email sending times• Email senders• Persona-based content• Number of form fields• Form field wording

So…everything?

THE CONUNDRUM – WHERE TO START!?

• Start with your goals and identify primary influencers

• Start with your leading hypotheses

• Start with key assets that underperform

• Start big and go narrow

A/B Testing —the Basis for Growth Driven Design

TRADITIONAL WEB DESIGN

1.5 - 2 Years

1.5 - 2 Years

3 Month Redesign

Legacy Website

Project Budget$25,000 - $250,000

3 Month Redesign

WebsiteV01

WebsiteV02

Project Budget$25,000 - $250,000

Impa

ct

Time

05/03/2023

31

THE GDD DIFFERENCEIm

pact

1.5 - 2 Years

1.5 - 2 Years

3 Month Redesign

Launch Pad

Website

Project Budget$25,000 - $250,000

3 Month Redesign

Time

WebsiteV01

WebsiteV02

Project Budget$25,000 - $250,000

Gained Opportunity

Gained Opportunity

LOOK AT YOUR TOP-VIEWED PAGES

BRAINSTORM AND HYPOTHESIZE

A

B

UTILIZE HEATMAPS AND USER SESSION DATA

UTILIZE HEATMAPS AND USER SESSION DATA

A/B Testing Software

A/B TESTING TOOLS

A/B Testing Without Tools

It is an imperfect science, but it is quite doable!

MANUAL TESTING

CREATE A TRACKING SPREADSHEET

HOW LONG IS LONG ENOUGH?It depends on:• Overall traffic• Traffic to that asset• The scale of data for your business

Pro Tip

Test it!

Find your benchmarks.

USE A CONFIDENCE CALCULATOR

A Client Case Study

Our Hypothesis: 3D book covers will drive more clicks than flat book covers

Flat Image 3D Image

Success Metric: % of views to clicks

THE RESULTS MAY SURPRISE YOU

Slight difference in views to clicksBigger difference in clicks to submissions

An A/B Testing Case Study – Sim City

AN A/B TESTING CASE STUDY – SIM CITY

MULTIVARIATE TESTING VS. A/B TESTING

A/B = testing 1 variable

Multivariate testing = testing many variables and combinations of those

Ideal for high traffic websites

Source: optimizely.com

A MULTIVARIATE CASE STUDY – 2008 PRESIDENTIAL CAMPAIGN

Success metric: sign-up rates

24 total combinations: 4 button – 6 image multivariate test

Hypothesis: video media elements will perform better

Source: blog.optimizely.com

A MULTIVARIATE CASE STUDY – 2008 PRESIDENTIAL CAMPAIGN

The Results• Hypothesis disproved

• 40.6% improvement rate

• 2,880,000 additional emails

• $60 million in donations

Source: blog.optimizely.com

A REVIEW OF KEY CONCEPTS• A/B testing is not an option, it is a must to stay competitive

• Only test one factor at a time per hypothesis

• You must be patient and wait for a significant sample size before drawing conclusions

• You are never done with A/B testing – don’t settle

• Track the results! Set regular times to update and revisit your findings

• Make it part of your culture and daily thought process

RESOURCES• How to Test Your Landing Pages infographic

• The A/B Test Calculator and A/B Testing Guide from HubSpot

• The A/B Test Flow Chart – print this out and hang it up!

• Hotjar – a free heat mapping tool

• 71 A/B Testing Ideas

• 10 A/B Testing Tools

• Growthdrivendesign.com

• http://hubs.ly/H03W7hV0

Thank You!