slcsem testing-jan2014
DESCRIPTION
Ryan Hutchings of Vacation Roost talks testing and optimization for websites in 2014TRANSCRIPT
![Page 1: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/1.jpg)
1
A Crazy Culture of Testing
![Page 2: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/2.jpg)
2
Plethora of Piñatas
1. Webinars: Marketing Experiments & Site Tuners
2. Landing Page Optimization: Optimizely
3. User Testing: Usertesting.com
4. Browser Extensions: Cookies tab in Firebug & Responsive design view in Web developers
5. Tag management – Brighttag.com, Google Tag Manager
Useful Tools
Landing Page vs. Conversion optimization
Shiny Objects
![Page 3: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/3.jpg)
3
Getting in the game: some webinars, look into crazyegg and Google Experiments
Get me some tools: Usertesting, Optimizely, more webinars, analytics based projects
Fuel on the Fire: statistical significance, full A/B, adopted methodology
Phase I
Phase II
Phase III
![Page 4: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/4.jpg)
4
Phase I & II
Landing Page optimizationOptimizely/Google ExperimentsVersion testing – compare time periodsGut based w/ mix of some research (webinars or competition
• Seasonality, external factors (economy, news, weather), time of day, marketing & advertising efforts
• STATISTICAL SIGNIFICANCE• MATHMATICAL METHODOLOGY
![Page 5: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/5.jpg)
5
The “Next Level”
Phase III
MethodologyRealistic mathematical understanding of site funnel metricsFull A/B testing platformMix of large scale A/B & LPO – dual trackFull tracking system & statistical baselines
![Page 6: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/6.jpg)
6
Methodology - Conversion Heuristic
C = probability of conversion
m = motivation of user (when)
v = clarity of the value proposition (why)
i = incentive to take action
f = friction elements of process
= anxiety about entering information/taking action
![Page 7: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/7.jpg)
7
427% Lift
![Page 8: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/8.jpg)
8
Math & the Funnel
1,000 * 40% bounce = 600
1,000 * 30% bounce = 700
3.5% conversion = 21 vs. 25
WIN?
1,000 * 40% bounce * 45% funnel entries@ 4% & 55%@1% = 13
1,000 * 30% bounce * 45% funnel entries@ 4% & 55%@1% = 15
![Page 9: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/9.jpg)
9
The Basic Funnel
Landing Pages
Search Results
Property
Cart
Billing
Purchase
3.5%
1.7%
40%
“rising tide” A/B
Single page LPO
![Page 10: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/10.jpg)
10
A/B & LPO – The Dual Track
A/B Tests
Internal system with server-side splitting,
custom vars
Full manual reporting toolset
LPO
Optimizley
Software determines
Headlines/CTA
PPC/SEO
Lead Forms
Images
Search Results
Cart
Product
![Page 11: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/11.jpg)
11
Determining the Priority & Process
Landing Pages
Search Results
Property
Cart
Billing
Purchase
“rising tide” A/B
Single page LPO
Book Now or Web Lead
Checkout
Book
10% - 30%
Optimizely
![Page 12: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/12.jpg)
Cart 1 Cart 2
Results• 32% Conversion rate increase
• 96% Level of confidence
• $800,000 Additional annual online sales
![Page 13: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/13.jpg)
![Page 14: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/14.jpg)
![Page 15: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/15.jpg)
Rigorous Testing
29.8% Lift
18.7% Lift
54% Total Lift
![Page 16: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/16.jpg)
Rigorous Testing – Sample of current tests
Site / Page Lift
Beaver Creek Lodging 29.8%
Breckenridge Lodging 427.4%
Park City Resort 128.9%
Vail Destination 69.1%
Mexico Destination 8.7%
Mexico Lodging Ideas 5.8%
Mexico Travel Ideas 5.6%
And many more…
![Page 17: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/17.jpg)
17
Tidbits from Testing
• Orange buttons for main CTA
• One main CTA button vs. multiple options
• People on pages
• Security/value icons work every time
• Consistent value propositions
• Fold doesn’t matter much
• Typically the vertical layout works
• Smaller “hero” images
![Page 18: Slcsem testing-jan2014](https://reader036.vdocuments.net/reader036/viewer/2022062319/55529524b4c9051f108b4d88/html5/thumbnails/18.jpg)
18
PowerPack Punch of Death
1. Marketing Experiments
• Webinars, research library, methodology
2. A/B Split Testing Capabilities
• Split testing across the whole site, server side traffic splitting, custom vars
3. Full Tracking based on vars
• Online & Offline conversions based on split
4. Optimizely
• Ability to deploy on the fly, integrate w/GA & crazyegg
5. GA Reporting
• Custom reports & custom segments
6. Statistical Tracking & “Funnel Math”
• Excel report and/or online tool (visualwebsiteoptimizer.com/ab-split-significance-calculator/ – based on 5% p-value, or 95% confidence level)