rapid a/b testing 10 principles (jared waxman)
Post on 21-Oct-2014
4.428 views
DESCRIPTION
Smart web and ecommerce companies companies are turning to Website Optimization (A/B and MVT Testing) as the next big area for growth. When done right, optimizing your website can be the most highly leveraged function within web marketing. Most companies fail at the first step - organization design - and never get their A/B testing program off the ground. Jared Waxman discusses how Intuit changed up the organization process to achieve one-day A/B testing sprints, and saw multi-million dollar payback.TRANSCRIPT
Jared Waxman
Fast & Furious: 10 Principles for
Rapid A/B Testing
Questions? Email [email protected]
Jared Waxman presents…
Presented at Internet Marketing Conference
Sept 16-17
Vancouver, CANADA
E X P E R I E N C E S A N D L E S S O N S W H I L E R U N N I N G A / B T E S T I N G F O R E C O M M E R C E S I T E S AT :
M a k e r s o f Q u i c k e n , T u r b o T a x , Q u i c k T a x a n d Q u i c k b o o k s
3
Intuit Small Business – Web Business
How much: On order of ~$100 M annual revenue transacted online
What: Desktop software, Services, Software-as-a-Service (subscription model)
Where: 45 Websites, 10 Shopping CartsTraffic: QuickBooks.com ~ 1 M monthly
unique cookies
Jared Waxman
3
4
QuickBooks.com – Various Timepoints
Jared Waxman
4
5
I’ve tested…
I’ve run 1 or MORE tests against each of these hypotheses:How much does a Free product cannibalize Paid
products?What’s the optimal way to position a Free product in
the lineup to maximize all units while minimizing paid cannibalization?
Does one Free offering cannibalize less than another Free offering?
What is the most effective “voice” used in describing the product?
What category of images sells the product the best?
Jared Waxman
5
6
…and tested…
Do videos sell better than static images?Do expanded versus collapsible product specs perform
better?Does having a special price for upgraders yield higher
overall sales than a flat pricing model? Does it improve customer sentiment?
Does exposing retail options significantly shrink direct online sales?
Which discount % level of promotional price discounts yields highest revenue?
Does adding cart CTA on home and line-up pages increase conversion or change pathing?
What is the optimal color, size, placement, and copy on CTAs?
Jared Waxman
6
7
…and tested…
Where is the optimal place in the purchase funnel for customer to configure their product?
Where is the optimal place in the purchase funnel to ask for credit card info?
How many pages in the checkout flow is optimal?What home page weighting of offerings moves us to
the desired platform mix while continuing to grow revenue?
How effective are certain product chooser tools, demo, or guides in moving conversion rates?
How effective is a certain cart abandonment tactic in moving conversion rates?
Jared Waxman
7
8
…and tested…
Where is the right step in the purchase funnel to cross-sell other offerings?
How many cross-sell offerings are optimal for a given funnel step?
Should cross-sells be passive or intrusive?How effective and efficient is proactive chat in
boosting over conversion?How many CTAs and/or featured offerings are
optimal on the home page?How many offerings is optimal within the line-
up?
Jared Waxman
8
9
…and tested…
Are bullet points or prose styles more effective on landing pages?
Are PPC ad group specific links on landing pages more effective than generic links?
What’s the likelihood of successful for a new product offering if added to the online store?
How would a New User vs Existing User focused site perform against overall online revenue?
Can lift conversion rates by guessing at New vs Existing and then sending them down different paths?
How effective would a simple User Contribution System be at engaging and converting prospects?
Jared Waxman
9
10
…and tested.
Does simplifying the home page by removing promotions on the homepage to other auxiliary Company sites/product offerings reduce the net traffic that gets sent?
Do menu titles perform better if they are job-oriented or product-oriented?
Do promos perform better as %s or as $s off?Do promos perform better as %/$ off vs Free
Shipping?Does a certain messaging approach consistently
outperform another regardless of specifics?
Jared Waxman
10
11
What has been the benefit of testing?
Half our growth in the online business comes from testing the pure site experience. The other half from acquisition programs.
Testing is highly leveraged growth – a handful of employees. Produces 3-digit ROI.
Acquisition programs requires serious cash and overall produces 2-digit ROI.
Jared Waxman
11
12
Today’s Focus: Organization Design
These gears must all work together.
Each deserves separate treatment.
Focus of this presentation
Jared Waxman
12
Jared Waxman 13
Fast & Furious
My Testing Team throughput in just 12 business days:• 5 tests (3 of which shown on next page) including 1
microsite• 11 Factors Tested• 26 Recipes In All
1 Big Winner: 20% increase on the success formulaAnnual impact ~ $10M
Plus…
1 Loser (but killed quickly)Some breakevens
14
Rapid Testing Team
Go Broad (8 recipes)
2/20/09
Then Narrow (8 Recipes)
3/4/09
And Broad, Again!
3/9/09
Formula: Index 1.29 Weekend, .91 Weekday
Formula: Index 1.20 WE/WD
Control
1
2
4
Jared Waxman
Formula: Index 0.50
3
Narrow Again (4 Recipes)
4/9/09
Product A: Index 2.00
Product A: Index 1.60
Jared Waxman 15
Top 10 “Go Fast” Testing Principles
10. Testing “Strike Force” with no outside dependencies
9. Exactly one “Approver” 8. One straight-forward
formula for success 7. Inform at go-live, then
again with results 6. Fully-baked ideas as the
starter, then testing team tightens it. After phase 1, iterate with true MVT and refine as appropriate.
5. Test ideas based in theory and customer data. Then go test, not talk.
4. Manage to deadline with quality. Reduce scope as needed!
3. Take in ideas from anywhere and everywhere. Don’t take in orders.
2. Open, flexible test calendar basing next tests on data from existing ones. No pre-slotted commitments
and finally…If you follow the prior 9 principles you’ll be able to deliver…
1. One-Day Test Sprints! From schedule-it to go-live in 8 hrs.
16
Process & Traps
Traps:
• The “Evil And”
• Management not aligned – competing goals
Traps:
• Ownership issues
• Re-tests, small tests
Traps:
• Debate
Traps:
• Perfection
Jared Waxman
16
Jared Waxman 17
IN THE ORG, YOU’LL ENCOUNTER…
Digging Deeper into Organization Design
18
HiPOs
Jared Waxman
18
19
HiPOs: Tips
1. Get Management to understand why testing success probability means it’s in their interest to have a healthy distance from individual test designs. It’s in your interest too.
2. Get Management engaged at the right level – setting hierarchy of goals and resourcing for highly leveraged testing.
Jared Waxman
19
Jared Waxman 20
Targets20
21
Targets: Tips
1. Set by Management, not testing team.2. Testing Team Lead responsible to push
back if needs more constraints.3. Put in place a hierarchy of success metrics.
Should be consistent across seasons, managers, businesses. Revenue per visitor (RPV) typically at top of hierarchy. Can assign each success metric it’s own team.
Jared Waxman
21
22
Design by Committee
Jared Waxman
23
Design By Committee: Tips
1. Hire a Test Lead whose business instinct and creative eye you know and trust.
Keep him/her in the kitchen on business strategy. Then let him/her be the one and only approver on what
goes live.
2. Ideas get thrown “over the wall” to the testing team.
3. Testing Teams set up as neutral, with no built-in biases towards specific businesses, ideas, or managers.
4. Idea submitter and/or related business unit have no official role during the execution phase.
Jared Waxman
23
24
Sacred Cows
Jared Waxman
25
Sacred Cows: Tips
1. Assign Executives to Goals Hierarchy. Assign Segment Marketers job of throwing ideas “over the wall”. Assign the Test Lead the job of selecting and executing.
2. Separate Segment Marketers from Testing.3. Get a clear list of constraints, a consistent success
metric, and then challenge everything.4. Outside viewpoint. Much easier to get people think
outside the box when they know some crazy-sounded idea worked somewhere else.
5. Test big. And if you can’t, test small. Remember you can also pull if it’s losing money.
6. Influence through your framing. Testing can be framed not as a challenge, but a way to quantify a “known” lever so it can be expanded or further optimized.
Jared Waxman
25
26
Stampedes
Jared Waxman
27
Stampedes: Tips
1. Cordon off your Testing Teams to focus your highly leveragable resources
2. Set clear, long-term goals for the Teams.3. Empower the Team Leads to decide
priorities, timing, workload.4. Leverage outside agencies to rapidly expand
capabilities and expertise as needed.
Jared Waxman
27
Jared Waxman
Fast & Furious: 10 Principles for
Rapid A/B Testing
Want to learn more? Email [email protected]