A/B Testing 101with JR Hopwood

A B

Overview

Introduction

Need convincing?

Deciding on a test

Evaluate a test

Wrap up A BVs.

Introduction

About the presenter

JR Hopwood With over a decade of web analytics and website testing experience, JR works with iAPPS

website partners to assist them in getting the most out of their web applications.

The iAPPS Success Group (iSG) is a team of support and digital services specialists who help Bridgeline Digital customers succeed online.

Bridgeline Digital iAPPS Success Group

Need Convincing?

Need Convincing?

Do you like peanut butter?

Do you like jelly?

Do you like PB&J?

Need Convincing?

You just tested! Peanut Butter has always worked for you… But you wondered if Jelly could add some spice… And then, WHOA, what about adding them together???

A/B Testing works the same way We take something that may be ‘working’ today and see if we can improve on it Sometimes we’re right, sometimes we’re wrong

A/B Testing provides the numbers You no longer have to rely solely on opinions Testing analytics will show you precisely what works and what doesn’t

Need Convincing?

AB Testing allows you to keep what is working today and just test a subset of traffic

Current Page Test Page

75% 25%

All Visitors

Ways to create subsets:

PercentageTraffic Source

Site ActionEtc…

Need Convincing?

Now what if that translated to more conversions?

Current Page Test Page

75% 25%

All Visitors

5% conversion 10% conversion (+100%)

If each conversion was $1 and you had 1,000 visits you

made $12.50 more than you would have!

Need Convincing?

Conversely, what if that translated to less conversions?

Current Page Test Page

75% 25%

All Visitors

5% conversion 2.5% conversion (-50%)

If each conversion was $1 and you had 1,000 visits you only lost $6.25 over what you

would have made… you

tried.

Need Convincing?

Stop the bleeding! Luckily with AB Testing you can turn it off…

… just try and wait until there is some statistical significance Sometimes conversion numbers are skewed due to visitor behavior changes, so be patient Example: No one buys on the weekends

Statistical Significance

=Confidence

Need Convincing?

Improve ROI or minimize the losses Testing can only help conversion in the long run

Even a failed test can prove to you that you SHOULD NEVER do that variation again

Since you aren’t sending all of your traffic to the new variation the gains and losses have a lessor effect on your current bottom line

…but could provide a substantial change to your future one!

This gives you the opportunity to try new things without going ALL IN Just remember, you need enough traffic to gain Statistical Significance

Need Convincing?

Here’s an example scenario I see all the time… Page A converts at 25% and has for several months… Marketing team doesn’t want to touch it, because… CEO doesn’t want to lose revenue…

What is a marketer to do? Remind CEO that we aren’t changing the page for everyone

Do the analysis… show that even a 3% bump in conversion on that page translates into $n increase in revenue

Calculate the resource time of those involved (including agencies) Compare this against the potential gains… and you’ll get the green light

Even if your site is non-revenue driving still show conversion increases Show how these translate into real people If your site just pushes promotional brochure-ware, but for each brochure downloaded

your company makes $n in return… get the picture?

CEO’s like the bottom line Once you prove that testing can have an immediate impact on bottom line the tide will

turn and your CEO will start asking YOU when the next test is

Decide what to test

Decide what to test

Define a Goal

The How’s Gut / Common Sense Web Analytics Settle an argument… I’m not kidding

The What’s Design Functionality Content

Decide what to test

Don’t just test to test… Make sure you are staying true to your KPIs Whether you sell widgets or provide information you have something to test

We want to test because we want to… Increase revenue Increase conversion Decrease Bounce Rate Decrease Exits

Once your goal is defined the fun begins…

Decide What to Test | Gut / Common Sense

Why test a page you just feel doesn’t have the best usability? GUT…

Why NOT test a page that isn’t included in your funnel and that no one? COMMON SENSE…

It’s OK to use these as a basis for a test

Decide what to test | Gut / Common Sense

Some questions to consider:

Where is the largest fallout / drop off / abandonment?

What page presents the largest impact to conversion?

Does a page not match your brand?

Does a landing page not match the offer?

Decide What to Test | Web Analytics

Numbers don’t lie … often

Before making decisions based on your analytics please be sure it is implemented properly!

Metrics to consider Bounce Rates Exit Rates Funnel Abandonment

Decide What to Test | Web Analytics

Some questions to consider:

Are there pages that have a higher bounce rate than most?

Are there pages that have a higher exit rate than most?

Does a page in a funnel have an unusually high abandonment?

A little gut… These are great questions, but sometimes there are explanations…

Does an exit link to off site? Do you push visitors to call or email? Does the page have an iFrame, Flash or HTML 5 element?

Decide What to Test | Settle an argument

Marketing person A says one thing and Marketing person B says another Test it and prove which one does better!

Decide What to Test | Design

Obviously design changes are the low hanging fruit of website testing• Huge gains have been achieved by testing:

• Buttons• Imagery• Layout

Don’t go overboard…• The more changes you make to a page the less specific the test can be

• Example: If you wanted to test a form location on a landing page to see if the left or right side converted better (could be a great test)

• However, if at the same time you change the button color or copy…• What actually tested better? The layout change or the other elements??

Multivariate also has a time and a place• Multivariate testing allows for multiple changes to layout and sections on a web page at one

time• This can get tricky if you don’t have enough traffic OR if your tests are WILDLY different

• My recommendation would be to only move to multivariate after a few AB Tests so you get the hang of it and what to look for in the evaluation

Decide what to test | Functionality

There are so many ways to code a widget or functionality on a website• Think about how many Date Pickers you’ve seen out there…• Sometimes though there is a need to test these

• In our Date Picker example, is showing 2-months better than just showing 1?• What about a login, should you ask for user name and password at once or just user

name, then password on the next page?

Get layout right first…• Before you turn to functionality as the problem are you doing a good enough job explaining or

calling out on the page?

Decide What to Test | Content

It is amazing how the smallest content changes can make a difference• What would you click on more?

• “Begin Reservation”• “Continue”

• What content would you engage with more?• “You really need this product”• “We really think you should have this product”

Don’t couple with design…• As mentioned previously, if you change the layout or button colors you won’t be able to tell for

sure whether it was the copy or the other changes!• If you must do this at the same time, do a multivariate test so you can see how the variations

perform independently AND together

Evaluate a Test

Evaluate a test

Evaluate a Test

Testing platforms come with evaluation tools out of the box, use them!• The metric you’re looking for is Statistical Significance, Confidence or Probability to Beat

Original• What this shows is if the variation were to run 24/7/365 what is the chance it will outperform

the control, or original• Why this is necessary is that in the test the page actually ISN’T being shown to every

visitor• Statistical analysis crunches the numbers and determines the probability based on

various factors

Evaluate a Test

The higher the Significance the more confident you can be to move forward with that variation

• Obviously the more visitors that conduct the test and perform better on the competing variation the higher that significance will be

• Example: If you throw me 10 pitches and I hit 7, you can be somewhat sure I have about a 70% chance of hitting the next pitch. But if you throw me 100 pitches and I hit 70 you can be more confident in that 70%, since it has been confirmed over a longer period and by several pitches

DO NOT stop a test if Statistical Significance has not been reached• This only skews the data set and may not play out once you go to 100%

Things to Consider for your First Test

Don’t go overboard, but be bold• Start with elements that are easy to spot• Don’t let your first test simply be copy changes• The reason for this is so in your first test you’ll be able to see bigger variations in conversion

data and thus be able to learn about test confidence

Sometimes the end is better…• If you lose 10% of your visitors on the final step of your booking flow just think how quickly

your ROI could change if you lost only 5%... Or less• Sometimes small changes on those final pages are the difference between clicking “BILL ME”

and exiting• These visitors made it all the way to the final step… they didn’t leave because they are no

longer interested!

Have FUN!• Get other departments involved to secure buy in• “Hey design team, if you could redesign this page what would it look like?”• “Hey sales team, what would you change to make the site better?”

• Warning, this team is competitive!!!!• With A/B Testing the numbers don’t lie… a variation wins and variations lose

First Test Checklist

Have I defined my website goals?

Have I evaluated where a problem may exist?

Have I come up with a solid competing variation(s)?

Have I come up with a hypothesis on which variation will win?

Have I devised a segment strategy for who sees the test variations?

Is one variation outperforming another?

Do we have statistical significance?

Go with that variation

Repeat

Now it’s your turn to become the mad scientist!

Q & A

Thank You!

this webinar prepared bythe iSG Digital Services Team

at Bridgeline Digital

For more information contact iSG at:

strategysupport@blinedigital.com

Or call Seth Rich : (781) 404-5423