© 2016 Tealium Inc. All rights reserved. | 1
Building a Culture of Testing and Optimization
Joel Ring, HUB International
2000
2006
2015
2009
Joel RingDirector of Analytics, HUB International
15+ Years in digital analytics
Created and managed the Web Analytics department at bestbuy.com
Sr. Digital Marketing Consultant
working with some of the
biggest brands on the web
Created the Digital
Strategy & Solutions team helping clients
with their analytics and
digital marketing
Manage the Analytics team
focusing on digital
analytics, data sciences, and
testing & optimization
A lot has changed
2000
2006
2015
2009
The great thing about the
internet is you can measure
anything!
We’re ‘gonna fail-fast and
break things!Like Facebook!
The only way to know what will work best is to
test it!
With all the data generated through testing we will be able to understand our customers
like never before and learn
new ways to meet their
needs.
Every minute…• Facebook users share nearly 2.5 million pieces of
content.• Twitter users tweet nearly 300,000 times.• Instagram users post nearly 220,000 new photos.• YouTube users upload 72 hours of new video content.• Email users send over 200 million messages.• Amazon generates over $80,000 in online sales.
“Two-thirds of business are unable to extract value from their information.”
Source: 2015 Information Value Index from PwC and Iron Mountain
If we have access to all
this data, how do we create
an environment
that enables us to derive value
from that data?
Definition of an
effective test and
learn organization look like?
Implemented roles and procedures to create,
execute, and evaluate testsAND
are actively communicating results to drive change.
“The key to great analysisis not math,
it’s communication.”
Keys to creating a test and
learn organizatio
n
Sponsors Ideators
Executors Analyzers
The sponsor helps evangelize the testing organization and helps drive changes based on the results.
The ideators help develop testing ideas. These can be part of the testing team or stakeholders.
This is the group that executes the tests. This will generally encompass multiple departments and can include creative and technical resources.
The analyzers tabulate and communicate the results of the tests. They will be the team members who declare the test “winners”.
• Individuals can, and will, fall into multiple roles
• Include people from multiple departments
• Make sure to include stakeholders
• How many people in these roles?• Forrester estimates 5-15
How do I get the initial buy in?
Demonstrate value using existing
channels that already allow for testing
Email – Display Advertising – Paid Search
How do I get the initial buy in?
Conduct a kick off meeting
Conduct a kick off meeting
• Bring together people who will be either involved in or interested in the execution and effectiveness of your testing program
• Goal of meeting is to educate generate excitement through collaboration
• Kick off agendao Testing 101
• Types of test (A/B, multivariate, etc.)• What makes a good test• What is a testing plan
o Intro to our testing platformo Intro to our testing processo Ideation workshop
• Used the ideation ideas to start our testing prioritization
Develop a testing plan
Develop a testing plan
• Guides your testing prioritization, status and results
• What is in a testing plan?o List each test ideao Conversiono Status (Live, scheduled, complete, removed, on-hold)o Date launchedo Date completedo Results (winner)o Test type (A/B, A/B/C, multivariate)o Test location (site, email, SEM, display advertising)
• Keep in a central location accessible by the team
Test NameTest Cell Test Description Conversion Test Type Test Location Status
Launch Date
End Date Winner
Link to test variations Results
Control: Current quote first position Control Group
Test: The IRS is expecting “many filings” to be in error at some level. No one can just blow off the law, but there will be something of a pass on reporting penalties as long as it can be demonstrated that effort went into compliance. Walter Fagan
Test Group
Control: What’s dominating the Employee Benefits agenda in 2016?Get the answers to the big questions →
Control Group
Test A: What’s dominating the Employee Benefits agenda in 2016?Get expert insights and tools you can use in the year ahead →
Test Group A
Control: current template designTest: Test: plain text format that will appear personalized including 1. Dear [name] 2. Sincerely [regional EB president]
Control: no nurture emailTest: CM nurture series to those who completed ACA nurture Ems
Control: always onTest A: second panel of ACA siteTest B: new contact form to be designed for campaignControl: no button/SM buttons Control
Test: remove SM buttons and add contact us button that goes to contact a rep form TestControl: https://www.hubinternational.com/employee-benefits/contact-us/
Test: modal pop-upControl: Current copy Control
Test:Headline: Optimize your Employee Benefits in 2016
Box 1: ACA Reporting & Verification AuditsBox 2: Cost Management Strategies
Test
Q1 Campaign Site
Live 9-Feb TBD
Complete 4-Feb 1-Apr Control ResultsEB Always On HP Interstitial Link clicks A/B Q1 ACA Campaign Site
ACA Panel Jack Quote 1 Share button clicks A/B
TBD TBD
CM nurture to ACA list Increase lead pipeline
A/B email
Live 9-Feb TBD Results
CM nurture email #2 template Email CTR A/B On-Hold
Results
Contact Us button location Button clicks A/B Q1 ACA Campaign Site
ACA Ad words Landing page test Form Completion Rate
A/B/C Ad words
Live 7-Mar
Complete 22-Feb 11-Apr Test
TBD
Results
Contact Rep Form Design - sub pages
Form Completion Rate
A/B Always On Site Live 9-Mar
L1 page CTA Link clicks A/B Q1 L1 page Complete 9-Mar 6-Apr Test Results
Conduct regular testing/analytics meetings
Conduct regular testing/analytics meetings
• Include core team and stakeholders
• Use the testing plan as a guideline for the agendao Review tests in progress
• Determine any status changeso Review tests scheduled to launcho Discuss any new testso Reprioritize based on new tests (if necessary)o Present deep dives from previous meeting questions
• Use the meeting to generate new test and analysis ideas
Standardize the test process and embed it in business as usual
DK Cool Crayon
Test approved Build testAsset(s)
needed?Add to test
plan N
Create asset(s)
Y
Review test Launch test
Wait 7 days
DK Cool Crayon
Test approved Build testAsset(s)
needed? Review test
Create asset(s)
Launch testAdd to test plan
Y
N
Wait 7 days
Why wait 7 days?
• Early results show greater variance.
• Behavior often varies by day of week, want to ensure all days are represented.
DK Cool Crayon
Test approved Build testAsset(s)
needed? Review test
Create asset(s)
Launch testAdd to test plan
Y
N
Wait 7 days
Declare winner?
Y
Review meeting
N
Determine next steps
Update test plan End
Suspend test
Declare Winner
Continue testing
Update test plan
Update test plan
End
Revise test?
N
Y
Review results
Final thoughts
Start Slow
Start Slow
Use basic tests to establish your
process, then grow into complexity
Don’t panic if your tests don’t have significant results
You still learn from tests with no differences
Don’t panic if your tests don’t have significant results
Implemented roles and procedures to create,
execute, and evaluate testsAND
are actively communicating results to drive change.
Questions?
THANKYOU